AI Glossary Term

What does “On-Device AI” mean?

Definition

AI inference performed locally on user hardware rather than in a remote cloud service.

Browse the full AI Glossary Explore all free AI guides

How On-Device AI works

Example

A phone transcribes a voice memo without internet by running a speech model on its own chip. The audio never leaves the device.

Why it matters

Local processing can improve privacy, speed, and reliability. Models must be small enough for the hardware, so they are less capable than the largest cloud models.

Common questions

Is it more private?

Often, since data need not leave the device, though apps may still send other data elsewhere.

How do models fit?

Through compression such as quantization and smaller architectures.

Does it drain the battery?

It uses power, but dedicated chips for AI make it more efficient.

Causal InferenceMethods for estimating cause-and-effect relationships rather than simple correlations.Semantic SearchSearch that matches meaning rather than exact keyword overlap, often using embeddings.InferenceThe runtime phase where a trained model generates predictions or outputs.Inference-Time ComputeThe amount of processing power consumed while producing each response.Intent ClassificationPredicting the user's purpose from a text query to route it correctly.Model QuantizationReducing numeric precision of model weights to decrease memory and inference cost.

Learn more in our free guides

FDA Regulation of AI Medical Devices Core ML for On-Device Apple Deployment ExecuTorch for On-Device PyTorch FDA Predetermined Change Control Plans for AI Devices

Test yourself

FDA Regulation of AI Medical Devices Quiz Core ML for On-Device Apple Deployment Quiz ExecuTorch for On-Device PyTorch Quiz