What is Mixture of Experts (MoE)?

A task where a model assigns an input to one or more predefined categories.

An attention pattern where each token attends only to a fixed-size window of nearby tokens to reduce compute.

A model component that dynamically focuses on relevant parts of an input when producing an output.

A reasoning style where an AI model decomposes a problem into intermediate steps.

A multimodal model architecture that learns shared representations between text and images.

The processing resources required to train and run models, often measured in FLOPS or GPU hours.