Seldon Core and Inference Graphs
Seldon Core is an open-source platform for deploying machine learning models on Kubernetes, with a standout feature: inference graphs.
Overview
Instead of serving one isolated model, it lets you chain models, routers, combiners, and transformers into a single directed graph that runs as one deployable service.
Deep Dive
Many real production use cases involve more than a single model call. You might preprocess input, route a request to one of several models, run an ensemble, and then post-process the result. Seldon Core expresses this as an inference graph defined in a SeldonDeployment (or, in the v2 architecture, via the Seldon Core Operator and MLServer). The graph is built from reusable component types: a Model serves predictions, a Transformer modifies inputs or outputs, a Router decides which child to call (enabling A/B tests and multi-armed bandits), and a Combiner aggregates outputs from multiple models for ensembling. Seldon supports many frameworks through prepackaged servers and custom Python wrappers, and it exposes rich metrics, distributed tracing, and payload logging out of the box for observability and explainability.
Technical Insight
An inference graph is a directed acyclic graph where each node is a microservice with a standard predict interface, and Seldon's orchestrator (the service orchestrator/executor) routes a request through the graph and merges responses. Because Routers can implement multi-armed bandit logic, traffic can shift adaptively toward better-performing models based on live reward signals. Seldon Core v2 decouples the graph from individual model servers using MLServer and the Open Inference Protocol, enabling multi-model serving and overcommit on shared hardware.
Strategic Impact
Cost and budget
Architecture decisions drive performance and operating cost for years.
Clearer decisions
Technical education helps teams choose the right stack, not just the newest one.
Quality control
Better engineering choices reduce reliability incidents in production.
The Future of Seldon Core and Inference Graphs
Seldon is moving toward modular, data-centric MLOps with Core v2's pipeline and dataflow design, plus tighter coupling with drift detection (Alibi Detect) and explainability (Alibi Explain). As LLMs and agentic systems become compound graphs of retrieval, models, and tools, the inference-graph abstraction maps naturally onto these workflows. Expect more emphasis on multi-model serving efficiency, streaming, and standardized observability so that complex, multi-step AI systems remain debuggable and governable in production.
Real-World Implementation
A lender chains a Transformer that one-hot encodes features into a model node, then a Transformer that formats the score, all as one SeldonDeployment.
A media company uses a Router node running a multi-armed bandit to dynamically send more traffic to whichever recommendation model is earning higher click reward.
A team ensembles three fraud models with a Combiner node that averages their scores before returning a single decision to the caller.
A regulated insurer attaches Seldon's payload logging and Alibi explainers to an inference graph so every prediction can be traced and explained for audits.
Risks & Guardrails
Optimizing one benchmark can hide broader system weaknesses.
Infrastructure and maintenance costs are often underestimated.
Security and observability gaps can grow as systems become more complex.
Implementation Roadmap
Define latency, quality, and cost targets before implementation.
Benchmark under realistic load and data conditions.
Instrument monitoring for errors, drift, and user impact.
Prepare rollback and incident response paths before scaling.
Keep Exploring
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Seldon Core and Inference Graphs quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Next guide
TensorRT and Inference Engines
Frequently asked questions
What is Seldon Core and Inference Graphs?
Seldon Core is an open-source platform for deploying machine learning models on Kubernetes, with a standout feature: inference graphs. Instead of serving one isolated model, it lets you chain models, routers, combiners, and transformers into a single directed graph that runs as one deployable service.
What is the defining feature that distinguishes Seldon Core from a single-model server?
Seldon Core lets you compose models, routers, combiners, and transformers into a single inference graph deployed as one service.
Which Seldon graph component decides which child node to send a request to, enabling A/B tests?
A Router directs requests to one of its children and can implement A/B tests or multi-armed bandits.
Which component type aggregates outputs from multiple models for ensembling?
A Combiner merges the responses of multiple child models into a single output, which is how ensembles are built.
What kind of graph structure does a Seldon inference graph form?
Seldon inference graphs are directed acyclic graphs where each node is a microservice with a standard predict interface.
In Seldon Core v2, which server and protocol enable multi-model serving across the graph?
Core v2 decouples the graph from model servers using MLServer and the Open Inference Protocol for multi-model serving.