LLM Inference Routing and Load Balancing Quiz

Answers and explanations appear as you take the quiz.

1. Why is plain round-robin often a poor load-balancing strategy for LLM inference?
   • Requests vary hugely in cost and replicas hold sticky KV caches
   • It is illegal in most countries
   • Round-robin only works for databases
   • It requires a GPU to run
2. What is 'prefix-cache affinity' routing trying to achieve?
   • Sending each request to a random GPU
   • Routing a request to the replica that already cached its prompt prefix
   • Encrypting the prompt before sending
   • Compressing the model weights
3. In difficulty-based model routing, what typically happens to an easy query?
   • It is rejected
   • It is sent to the largest model available
   • It is sent to a small, cheaper model
   • It is queued for 24 hours
4. Which live signal is most useful for an LLM-aware load balancer?
   • The server's hostname length
   • The color of the rack
   • The time zone of the user only
   • Queue depth and KV-cache occupancy of each replica
5. What does a tool like LiteLLM provide in a multi-provider setup?
   • A single proxy that balances and fails over across many model providers
   • A new transformer architecture
   • A GPU cooling system
   • A replacement for the tokenizer
6. Why can one long-output request outweigh many short ones for balancing?
   • Long outputs use no resources
   • Each output token costs a full forward pass, so total token load matters more than request count
   • Short requests are always slower
   • The router cannot see output length at all
7. A serving system reports goodput rather than raw request count. What does goodput usually emphasize?
   • The total number of model parameters
   • Only the number of HTTP connections opened
   • Requests served while meeting the specified latency or service-level targets
   • The fraction of requests that use the largest model
8. A gateway first chooses a small or large model, then chooses a replica of that model. What are these two decisions?
   • Tokenization followed by decoding
   • Training followed by fine-tuning
   • Prefill followed by prompt construction
   • Model routing followed by replica load balancing