用于图像编辑的 LoRA 滑块

A text-to-image diffusion model such as Stable Diffusion XL makes a picture by starting from random noise and removing it step by step, guided by a text prompt. Prompts are a blunt control. Asking for a slightly older person usually changes the face, the pose and the background along with the age, because small prompt changes can produce a very different image from the same random seed. A LoRA slider is a way to get a smooth, targeted dial instead: a small add-on to the model's weights that pushes one attribute up or down by an amount the user chooses, ideally with everything else held still.

The add-on is a LoRA, short for low-rank adaptation, a technique published by Edward Hu and colleagues in June 2021 for cheaply fine-tuning large language models. Instead of changing a full weight matrix W0, LoRA learns an update written as the product of two thin matrices, B times A, whose inner dimension, the rank, is small, such as 4. The original paper reported that, compared with full fine-tuning of GPT-3 175B, LoRA cuts trainable parameters by about 10,000 times and GPU memory by about 3 times, and adds no inference latency because the update can be merged into the original weights. The update is multiplied by a scale factor when it is applied, and that scale is the slider: 0 leaves the model unchanged, positive values apply the learned edit, and negative values apply it in reverse.

Concept Sliders, the method that made this idea well known, was posted to arXiv in November 2023 by Rohit Gandikota, Joanna Materzynska, Tingrui Zhou, Antonio Torralba and David Bau, of Northeastern University, MIT and an independent researcher, and published at ECCV 2024. Training a text slider needs only prompts: a target concept, such as person, and two opposing attributes, such as old and young. The frozen original model is asked to predict noise for each, and the method builds a training target that is the model's normal prediction for person, shifted toward old and away from young by a guidance strength. The LoRA is trained so that, given only the prompt person, the adapted model produces that shifted prediction. In effect, the slider learns to apply a classifier-free-guidance style push permanently, inside the weights.

Two additions make the edits cleaner. First, a single prompt pair can pick up unwanted attributes, so the method adds preservation concepts: when training an age slider, the authors repeat the objective over prompts naming different races, so the slider learns age without shifting ethnicity. The official code exposes this as an attributes option, for example male and female for an age slider. Second, the low rank itself acts as a constraint. The authors report that fine-tuning without the low-rank restriction reduced both edit precision and image quality, while low-rank training found a minimal subspace for the concept.

Sliders can also be defined from images. Given small sets of before and after pictures, the method trains the slider to reproduce the difference between paired images, which lets an artist define a direction by example, or lets the authors transfer a direction found in StyleGAN, an older face-generation model with well-studied editable directions, into Stable Diffusion. At generation time, the authors follow a technique called SDEdit: the first denoising steps run with the slider switched off, so the overall layout is set by the original model, and the slider is applied for the remaining steps, which keeps the edited image structurally close to the unedited one.

Compared with alternatives, sliders trade setup cost for convenience at use time. Prompt editing needs no training but changes too much. Image-editing methods such as Prompt-to-Prompt change cross-attention for one image and need a separate pass for each new edit; the paper reports that its sliders achieve comparable CLIP-score changes, a measure of how well the image matches the requested edit, with smaller LPIPS distance, a measure of perceptual change, from the original image. Once trained, a slider is a small file that works in a single generation pass, can be scaled continuously and can be combined with others.

The limitations are stated in the paper. Disentanglement is imperfect: the authors observed residual effects on other attributes, and sliders trained on a single prompt pair were sometimes entangled with unintended directions. Each slider is tied to the base model it was trained on, so a Stable Diffusion XL slider does not work on another model family, and the project's own repository describes its FLUX.1 support as experimental, with results that may not match SDXL. The fix-hands slider reduced but did not eliminate hand errors, from 62% to 22% in the study. Like any image-editing tool, sliders for faces and bodies can be used to alter images of real people, so the usual consent questions apply.

In practice, the code, data and trained sliders are public through the project site and GitHub repository, along with notebooks for Stable Diffusion 1.x and XL and an inference demo. Training a text slider requires writing a short configuration of target, positive, negative and neutral prompts with settings such as rank 4 and alpha 1, and a GPU that can run the base model.