用于多条件扩散控制的 T2I 适配器

T2I-Adapter is a method for steering a pretrained text-to-image diffusion model with an extra picture-like input, called a condition map. A text prompt alone is vague about layout: it can say 'a cat on a sofa' but not where the cat sits or which way it faces. A condition map fills that gap. It can be a line sketch, a depth map (a grayscale image where brightness encodes distance from the camera), a semantic segmentation map (each pixel labeled with an object class), a human pose skeleton, a canny edge map, or a coarse grid of colors. The adapter reads the map and nudges the image generator so the output respects it, while the generator itself stays frozen.

The method was introduced in the paper 'T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models' by Chong Mou, Xintao Wang and colleagues, posted to arXiv on 16 February 2023. The code appeared in Tencent ARC Lab's GitHub repository a day earlier. The paper's central argument is that a large model like Stable Diffusion has already learned how to draw structure and color, so a control method does not need to teach new drawing skills. It only needs to learn an alignment between the external signal and the model's internal features, which a small network can do cheaply.

Here is how it works step by step. Stable Diffusion generates images in a compressed latent space using a UNet, a neural network with an encoder that shrinks the feature map through several resolutions and a decoder that expands it back. Generation starts from random noise and removes noise over many steps. The adapter takes the condition map at 512 by 512 pixels and uses pixel unshuffle, an operation that folds spatial blocks into channels, to bring it down to 64 by 64, matching the latent size. It then passes the result through four feature extraction blocks, each a convolution plus two residual blocks, with downsampling between them. This yields four feature maps at 64, 32, 16 and 8 pixels square, exactly the sizes of the UNet encoder's four stages. Each adapter feature is simply added to the matching encoder feature. The text prompt still enters through cross-attention as usual.

Training touches only the adapter. Each example is a triple of an image, its condition map and a caption. The image is encoded to latents, noise is added at a random timestep, and the frozen UNet, now receiving the adapter features, must predict that noise. The standard diffusion loss flows back only into the adapter's weights. The paper trained on Stable Diffusion 1.4 for 10 epochs with batch size 8 on four 32 GB V100 GPUs, finishing within three days. Sketch maps came from 164,000 COCO images run through an edge detector, segmentation used COCO-Stuff, and keypose, color and depth adapters used 600,000 image-text pairs from LAION-Aesthetics with poses extracted by MMPose and depth by MiDaS.

The main alternative is ControlNet, posted to arXiv six days earlier by Lvmin Zhang, Anyi Rao and Maneesh Agrawala. ControlNet makes a trainable copy of the UNet encoder and joins it to the locked original through zero-initialized convolutions. That copy is large: Hugging Face's September 2023 comparison lists a ControlNet for SDXL at 1,251 million parameters and 2.5 GB in fp16, against 79 million parameters and 158 MB for T2I-Adapter-SDXL, a reduction of about 94 percent. The same post notes that the adapter runs once for the whole denoising process, while ControlNet runs at every step, so the adapter's added inference cost is small. ControlNet's larger trainable branch gives it more capacity, and both families remain supported side by side in the diffusers library.

On the paper's own benchmark, the 5,000-image COCO validation set, Stable Diffusion with text alone scored an FID of 24.68, while text plus a segmentation adapter scored 16.78 and text plus a sketch adapter scored 17.36. FID, Frechet Inception Distance, measures how close the statistics of generated images are to real ones, and lower is better. CLIP score, which measures text-image agreement, rose slightly from 0.2648 to 0.2666 with the sketch adapter. These are the authors' own numbers on one dataset, and that table did not include ControlNet, so it does not settle which method controls better.

Three properties made the approach popular. Adapters are composable: several can be summed with per-adapter weights, for example depth plus pose or sketch plus color, with no extra training. They are generalizable: an adapter trained on Stable Diffusion 1.4 also worked on 1.5 and on community fine-tunes derived from the same base, such as Anything-V4.0. They are small: ablations in the paper showed an 18-million-parameter 'small' and a 5-million-parameter 'tiny' version still controlling sketches well.

The limitations are concrete. When several adapters are combined, the weights must be tuned by hand, which the authors name as the method's main limitation. Because the adapter has no timestep input, the authors found that guidance given late in sampling had little effect, so they sampled training timesteps from a cubic distribution that favors early, noisy steps. An adapter is tied to the base model family it was trained for, which is why separate adapters were released for SDXL. Output quality also depends on the condition map: Stability AI warned that Stable Doodle results vary with the drawing, the description and scene complexity. In practice, the SDXL adapters were trained on 3 million LAION-Aesthetics V2 pairs for 20,000 to 35,000 steps, and the TencentARC repository lists about 15 GB of GPU memory for inference.