AI Glossary Term

What does “QLoRA” mean?

Definition

A fine-tuning technique that combines 4-bit weight quantization with LoRA adapters to reduce memory needs.

Browse the full AI GlossaryExplore all free AI guides

How QLoRA works

Diagram for QLoRA, layers from top to bottom: Base model weights stored in 4-bit; Small LoRA adapters in higher precision: trained; Fits fine-tuning on a single GPU.Base model weights stored in 4-bitSmall LoRA adapters in higher precision:trainedFits fine-tuning on a single GPU

Example

A researcher fine-tunes a multi-billion-parameter model on one consumer graphics card by compressing the base weights to 4-bit and training only small adapter layers.

Why it matters

It lowers the hardware barrier for customizing large models. Quality is usually close to standard LoRA at much lower memory.

Common questions

What does QLoRA combine?

4-bit quantization of the frozen base model plus LoRA adapters that are trained.

Who introduced it?

A 2023 paper from researchers at the University of Washington.

What is the trade-off?

Slightly slower and a small potential accuracy cost from quantization.

Related terms

LoRA (Low-Rank Adaptation)A parameter-efficient fine-tuning method that adds low-rank adapter matrices.QuantizationConverting model weights to lower precision formats such as 8-bit or 4-bit.Fine-TuningContinuing training on domain-specific data to adapt a pre-trained model to a specific task.Instruction TuningFine-tuning a model on instruction-response pairs to improve task following.Model QuantizationReducing numeric precision of model weights to decrease memory and inference cost.Parameter-Efficient Fine-Tuning (PEFT)Methods that adapt models by training a small subset of added parameters.

Learn more in our free guides

QLoRA and 4-Bit Fine-Tuning

Test yourself

QLoRA and 4-Bit Fine-Tuning Quiz

See also

DPO (Direct Preference Optimization)Flash AttentionTree of ThoughtsMulti-Head AttentionReActPositional EncodingAgentic LoopRoPE (Rotary Positional Embeddings)