ControlNet is a neural network architecture that adds spatial control to text-to-image diffusion models such as Stable Diffusion. A diffusion model makes an image by starting from random noise and removing noise step by step, guided by a text prompt. ControlNet adds a second input, a condition image such as edges, a depth map, a human pose or a segmentation map, so that the picture follows its layout. Lvmin Zhang, Anyi Rao and Maneesh Agrawala published it at ICCV 2023.
ControlNet copies the weights of the model's encoder blocks into a locked copy, which keeps the original model unchanged, and a trainable copy, which learns the new condition. Zero convolutions, layers whose weights start at zero, join the two, so training starts with no effect on the output and does not damage the original model. Training therefore works with small datasets of image pairs. The repository has pretrained models and demo apps for Canny edges, lines, scribbles, depth, normal maps, pose and segmentation.
The official code and models are free and open source under the Apache 2.0 license.