AI Glossary Term

What does “LoRA (Low-Rank Adaptation)” mean?

Definition

A parameter-efficient fine-tuning method that adds low-rank adapter matrices.

Browse the full AI Glossary Explore all free AI guides

How LoRA (Low-Rank Adaptation) works

Example

Instead of updating all billions of weights in a language model, a team trains small adapter matrices for a customer-service style. The adapter file is small, and the same base model can swap different adapters for different clients.

Why it matters

LoRA makes fine-tuning possible on modest hardware and makes adapters easy to store and share.

Common questions

What does low-rank mean?

The weight changes are represented by two thin matrices multiplied together, which needs far fewer numbers than a full update.

Does LoRA match full fine-tuning?

Often close on many tasks, though full fine-tuning may do better on some.

Can adapters be combined?

Sometimes, with care. They can be merged into the base weights or swapped depending on the task.

Parameter-Efficient Fine-Tuning (PEFT)Methods that adapt models by training a small subset of added parameters.QLoRAA fine-tuning technique that combines 4-bit weight quantization with LoRA adapters to reduce memory needs.Fine-TuningContinuing training on domain-specific data to adapt a pre-trained model to a specific task.Instruction TuningFine-tuning a model on instruction-response pairs to improve task following.DPO (Direct Preference Optimization)A training method that fine-tunes models directly on preference pairs without needing a separate reward model.Label SmoothingA regularization method that softens hard labels to improve generalization.

Learn more in our free guides

AI Workflow Automation Barlow Twins and Redundancy Reduction Domain Adaptation Kubeflow and ML Pipeline Orchestration

Test yourself

AI Workflow Automation Quiz Barlow Twins and Redundancy Reduction Quiz Domain Adaptation Quiz