LoRA ati Paramita-Ṣiṣe Tuning

Full fine-tuning updates a model's trainable base parameters, with optimizer state and checkpoints contributing to resource requirements. LoRA offers a different parameterization for selected weight updates. The original selected matrix stays frozen, while two trainable matrices form an additive low-rank update. Their inner dimension r controls the allowed update rank and parameter count; it is not a declaration that the whole pretrained matrix has rank r. For a matrix with d rows and k columns, factors with shapes d by r and r by k contain r times d plus k parameters. In a stated 4-by-4 example with rank one, the factors contain four plus four, or eight parameters, compared with sixteen entries in a full unconstrained update. This is an illustrative matrix calculation rather than a universal model-wide percentage. Reducing trainable parameters can reduce optimizer-state and task-checkpoint requirements, but the base computation and relevant activations still exist. A compatible update can also be folded into a dense base matrix before inference. The layer then uses the merged matrix instead of evaluating a separate adapter branch, with dtype and implementation compatibility still relevant. Different task adapters can share a base checkpoint, while serving behavior depends on the system used. QLoRA adds another set of choices: a frozen base stored at low precision, with gradients reaching trainable LoRA parameters through higher-precision computation. Its paper uses 4-bit NormalFloat storage and generally BFloat16 computation. Double quantization compresses the quantization constants themselves, addressing metadata overhead rather than meaning that every adapter gradient is quantized twice.