Self Hosting LLMs: When It Pays Off and When It Wastes Money

What you need to know

• Self-hosting LLMs is a GPU infrastructure project, not a configuration change. It requires dedicated GPU hardware, an inference server, and operational capacity.
• The cost break-even vs commercial APIs is typically above $50,000 per month in equivalent commercial API spend for most SaaS products.
• Data residency compliance is the strongest justification for self-hosting regardless of cost comparison.
• Open-source 70B models are GPT-3.5-class for structured tasks. Frontier commercial model quality requires commercial APIs in 2026.
• Fine-tuning justifies the infrastructure investment when the application has high-volume narrow tasks and the team has ML engineering capability.

The core argument

The instinct to self-host LLMs usually comes from one of two places: wanting to avoid API costs, or wanting control over data. Both are legitimate motivations, but both are often addressed with the wrong solution. The cost motivation is undermined by the GPU infrastructure cost, which typically exceeds the API cost at the scale most SaaS products operate at. The data control motivation is legitimate, but for many products, contractual data processing agreements with commercial API providers may satisfy the requirement without the infrastructure overhead.

The economics become clearer with numbers. A product spending $5,000 per month on OpenAI API calls would spend roughly $2,200 per month on a single A100 instance, plus engineering time to manage the inference infrastructure, plus the quality degradation from using a smaller model. The comparison is not simply cost per token; it is total cost of ownership including infrastructure management time. For a product at $5,000 per month in API spend, the financial case for self-hosting is weak. For a product at $50,000 per month, it begins to make sense with the right workload.

The workloads where self-hosting makes practical sense in my experience: high-volume text classification and extraction where a fine-tuned smaller model outperforms a general model at lower latency; products with genuine data residency requirements that cannot be satisfied by commercial API data processing agreements; and teams that have specific fine-tuning use cases where the domain specialization produces measurable quality improvement that justifies the infrastructure investment.

Common mistakes

1. Choosing self-hosting to avoid the appearance of vendor dependency without modeling the actual costs. The GPU infrastructure creates its own vendor dependency (AWS, GCP, or on-premises hardware providers) while adding operational complexity. Before committing to self-hosting, model the full cost: GPU instance cost, inference server engineering time, monitoring and maintenance, and the performance gap versus the commercial model being replaced.

1. Starting with a 70B model when a 7B model would suffice. Many SaaS application use cases (classification, summarization of structured data, simple Q&A with provided context) are handled adequately by 7B and 13B models, which require dramatically less GPU hardware. Start with the smallest model that meets quality requirements, not the largest model available. The cost difference between hosting a 7B and a 70B model is roughly 8 to 10x in GPU memory requirements.

1. Not benchmarking the self-hosted model against the commercial baseline before committing to infrastructure. Run the application's actual use cases against the candidate self-hosted model before provisioning the infrastructure. If the quality is acceptable for the use case, proceed. If it is not, the infrastructure investment produces a worse product at potentially higher cost. Benchmark first, infrastructure second.

1. Underestimating inference latency for user-facing features. Self-hosted inference on moderately sized GPU hardware is slower than commercial API responses, which route to purpose-built inference clusters. For user-facing real-time features (chat, autocomplete, instant search augmentation), the latency difference matters. Measure time-to-first-token and token generation speed on the target hardware before designing user-facing features around self-hosted inference.

1. Not planning for model updates and version management. Commercial APIs update models with backward-compatible improvements. Self-hosted models require the team to decide when to update to a new model version, test the new version, redeploy, and manage the transition. This operational overhead is ongoing. Plan for model version management as a recurring task, not a one-time setup.

Where to start

1. Run the cost analysis for the current commercial API spend. Pull the last three months of API spend. Calculate what the equivalent GPU infrastructure would cost at cloud spot instance rates. Factor in 20 to 30 percent engineering time overhead for infrastructure management. If self-hosting is not clearly cheaper at current volume, the motivation for self-hosting should be quality, data residency, or fine-tuning, not cost.

1. Test the candidate model on the application's actual prompts before investing in infrastructure. Use a managed service (Replicate, Together.ai, Groq) to run the target open-source model against the application's prompts for a week. This produces quality comparison data and cost data (managed inference is more expensive than self-hosted but less expensive than frontier commercial APIs) without the infrastructure investment. If the quality is acceptable, the self-hosting investment is justified by the managed service cost reduction.

1. Start with Ollama on a single GPU instance for the first three months. Ollama provides the simplest path to running open-source models in production. Start with one GPU instance, one model, one use case. Learn the operational patterns (model loading, inference throughput, memory pressure) before scaling the infrastructure. This limits the downside of a self-hosting decision that turns out to be wrong.

Related reading

• RAG (Retrieval Augmented Generation) for SaaS: When It Helps and When It Does Not
• Serverless vs Containers vs Bare Metal: A Cost and Flexibility Map
• Self Hosting on Hetzner vs AWS: The Real Tradeoffs
• SaaS Pricing for Founders: A Cost to Build to Sell Framework