Guide-linked quiz · Hard Level
Multi-Model Serving and Model Multiplexing Quiz
Balance shared model utilization with routing, on-demand loading, caching, eviction, GPU memory limits and tenant isolation.
Question 1 of 8
What does model multiplexing commonly allow a shared serving worker to do?
Preview every question8 questions
Answers and explanations appear as you take the quiz.
- What does model multiplexing commonly allow a shared serving worker to do?
- Keep every tenant model permanently loaded before any request
- Choose a model only after ignoring the request identity
- Load or select a requested model or adapter when needed
- Train all tenant models as part of each inference request
- Why can a shared model cache increase latency for an infrequently used model?
- The cache changes the model output labels
- An evicted model may need to be loaded again on its next request
- Eviction guarantees a faster reload from device memory
- An evicted model remains resident but is marked inactive
- What should an API do with a client-provided model identifier?
- Treat the identifier as a filesystem path supplied by the client
- Use the identifier to bypass authorization checks
- Assume any artifact named by a client is compatible
- Resolve the identifier through a validated registry or allowlist
- Which resource can limit how many models stay resident on one GPU?
- GPU memory capacity limits resident weights and runtime state
- The number of names in the API schema sets device memory
- The training-set label format determines residency
- More model identifiers always mean more available GPU memory
- Why might a simple least-recently-used eviction policy perform poorly?
- It always retains the largest model regardless of request history
- It selects entries using GPU memory as their cache key
- It ignores model size and the cost or likelihood of reloading
- It prevents cold starts whenever a model is evicted
- Which metrics help diagnose model multiplexing behavior?
- Only the number of model names configured
- Cache hits, evictions, load duration, and cold-start rate
- Only training loss for each model
- Only the API route count
- What should be considered when comparing shared multiplexing with dedicated replicas?
- Compare only model-file counts across architectures
- Compare only inference framework names
- Compare only training accuracy
- Compare demand patterns, resource use, latency, and isolation needs
- Why account for in-flight requests before unloading a model?
- Active requests may still depend on the model weights or runtime state
- Removing a model entry automatically redirects active work
- Unloading updates every client schema before a response
- In-flight requests no longer use the model once eviction starts
Keep testing yourself
More quizzes picked for your level.