Total Cost of Ownership for Self-Hosted LLMs

Self-hosting means downloading an open-weight model, such as a Llama, Mistral, Qwen or Gemma variant, and serving it on infrastructure you control. That can be servers you own or GPUs you rent in the cloud. The weights may be free to download, but running them is not. Total cost of ownership has several layers: - **Hardware.** Purchased GPUs are depreciated over a few years. Rented GPUs are billed by the hour whether they are busy or idle. Owned hardware also needs power, cooling, rack space and networking. - **Software and operations.** You need an inference server such as vLLM, SGLang or NVIDIA TensorRT-LLM. You also need load balancing, monitoring, logging, security hardening, and redundancy so one failed node does not take the service down. - **Staff.** This is often the largest hidden cost: engineers who deploy, tune, patch and upgrade the stack, plus someone on call. - **Upgrades.** New models arrive often. Each migration means re-evaluating, re-tuning and sometimes buying new hardware. Utilization decides the outcome. An API charges only for the tokens you use. A self-hosted GPU costs the same whether it serves one request or thousands. Effective cost per token is total monthly cost divided by tokens actually served. A lightly used cluster can cost far more per token than an API, while a steadily loaded one can cost much less. Three misconceptions are common. The first is that GPU rental price times hours equals the cost, which ignores staff and idle time. The second is that open weights mean free. The third is that self-hosting is automatically more private. A poorly secured server can be riskier than a provider with audited controls. There are good reasons to self-host: - control over your data - predictable costs at high, steady volume - customization through fine-tuning - freedom from a provider changing or retiring a model