The Quiet Cost of AI Infrastructure: GPU Reserved Capacity

What you actually need to know

• GPU on-demand pricing works for prototypes. At production scale, it is a tax on teams that did not plan ahead.
• The discounts on one year reserved GPU capacity typically run forty to sixty percent off on-demand rates. That is not a rounding error. It is often the difference between a profitable and unprofitable AI feature.
• Lead times for GPU reservations at hyperscalers can be several weeks. Planning after the cost problem arrives means overpaying while you wait.
• Batching is the first optimization, not the last. Teams that batch inference requests well need fewer GPUs before they start negotiating reservations.
• Self hosting versus API is a unit economics question, not a preference question. The crossover point is real and calculable.

The core argument

The bill for running AI features at scale does not arrive all at once. It arrives gradually, through monthly invoices that grow faster than the user base, until someone looks at the cost per user and realizes the product economics are not working. By the time that conversation happens, the team has been overpaying for months. The fix is not complex. It is just a conversation that most teams have too late.

The core issue is that on-demand GPU pricing is designed for variable workloads. It is priced at a premium because it is flexible. Most AI inference workloads at any meaningful scale are not actually variable. They have a baseline load that runs every day, plus peaks that come from traffic spikes. That baseline is the portion that should be on a reservation. The peaks can stay on-demand.

I have seen this pattern on multiple client engagements. The team builds the AI feature, ships it, and runs on-demand instances because the early load is uncertain. A few months in, the load stabilizes. The on-demand cost is visible and painful. Somebody finally looks at the reservation pricing and discovers that committing to capacity they are already using reliably would cut the bill by roughly half. Then they discover the reservation takes a few weeks to activate, during which they keep paying the on-demand rate.

The right time to model the reservation is as soon as the load becomes predictable, not after the bill becomes alarming. The data you need to make the decision is your average inference throughput over thirty days. Run the math before the quarterly review forces you to.

The unit economics of AI inference

On-demand versus reserved: the actual numbers

A standard GPU instance for inference on a mid-size model runs roughly two to four dollars per hour on-demand across major providers. A one year reservation for the same instance class drops that to approximately one to two dollars per hour, depending on provider and region. At twenty four hours of continuous operation, that is forty eight to ninety six dollars per day on-demand versus twenty four to forty eight dollars on a reservation. Over twelve months, the savings on a single instance run between eight thousand and seventeen thousand dollars.

Most production AI features at scale are running more than one instance. The math scales linearly with the number of instances in your baseline load.

The batching multiplier

Before reserving anything, optimize your inference batching. Serving one request at a time on a GPU that could batch eight uses twelve percent of available capacity. Teams that implement proper dynamic batching, where requests queue for a short window before being sent together, routinely reduce their per-request GPU time by three to five times. That reduction directly shrinks the number of instances you need to reserve.

How much does it cost

These are rough estimates based on mid-size models (7B to 13B parameter range) on current generation GPU instance families. Frontier model API costs follow a different curve because you are paying for tokens rather than compute time, and self hosting is not a realistic option for those models.

What to look for before committing

• Thirty days of stable throughput data. Do not commit to a reservation on a workload you have been running for less than a month.
• A clear model of your peak to average ratio. If your peaks are ten times your average, you need on-demand or spot capacity for the peaks even after you reserve the baseline.
• Confirmation of GPU availability in your target region. Some instance families have limited availability and the reservation does not guarantee physical hardware immediately.
• Your roadmap for the next twelve months. A reservation is a bet that your load will remain at roughly that level. A major product pivot can strand the commitment.
• Whether batching has been implemented and measured. Commit after optimization, not before.
• The cancellation terms for partial reservation if the workload changes significantly. Some providers allow exchange for different instance types.

Expert opinion

The teams that control AI infrastructure costs are the ones that treat GPU capacity as a procurement question, not just an engineering question. The reservation decision needs your throughput data, your cost per user model, and your product roadmap in the same room. The engineering team usually has the first. Finance usually has the second. Nobody convenes that meeting until the bill is already too high.

>

Yashveer Singh, founder of Yashveer Labs

How this played out on a real project

A startup I worked with was running a document analysis feature on on-demand GPU instances. When they hit about fifteen thousand active users, the monthly inference bill crossed a threshold where it was meaningfully eroding their gross margin. The engineering team had been aware the cost was growing but had not modeled the reservation alternative. When we ran the numbers together, a one year reservation covering their baseline load would have saved roughly forty five thousand dollars per year at their current scale, scaling to over a hundred thousand per year at their projected growth.

The additional finding was that their batching implementation was naive. Each user request triggered a single synchronous inference call. We implemented a simple queue with a fifty millisecond batching window, which grouped requests from concurrent users into batches of three to eight. That change alone reduced GPU utilization by about thirty five percent, which meant the reservation they needed to commit to was smaller than originally calculated. For the broader token cost picture, see token economics and why your AI bill surprised you. For the build versus buy framing on AI infrastructure, building production grade AI features without an ML team covers the decision tree.

Common mistakes

1. Committing to a reservation before collecting thirty days of stable throughput data. The data is cheap to collect. The wrong commitment is expensive.
2. Reserving for peak load instead of average load. Peaks should stay on-demand or spot. Reserve the baseline.
3. Skipping batching optimization before shopping for reservations. The reservation you need is smaller after batching than before.
4. Ignoring regional pricing differences. GPU pricing varies by region and provider. The cheapest option in one region may be forty percent more expensive in another.
5. Treating the API tier and self hosted tier as mutually exclusive. Many mature products use API providers for frontier models and self hosted inference for smaller utility models.
6. Not modeling the cost per user. Knowing your monthly infrastructure spend without knowing cost per active user means you cannot tell whether the product economics are working.
7. Forgetting to account for the GPU time used by batch jobs, reprocessing pipelines, and internal tools in addition to user-facing inference.

A 90 day plan

1. Month one. Instrument your inference calls to capture per-request latency, tokens in, tokens out, and GPU time if self hosting. Get thirty days of data before making any infrastructure commitment.
2. Month two. Implement or improve batching. Set a target of batching at least four requests per GPU pass at peak load. Measure the reduction in GPU hours.
3. Month three. Run the reservation math on your actual thirty day baseline post-batching. Compare on-demand, one year reserved, and three year reserved for your average load. Commit to the option that fits your roadmap confidence.

For related reading, see the real cost of just use GPT, a postmortem for the full picture on AI cost surprises across the stack, and token economics and your AI bill for the token-level breakdown that complements the infrastructure picture.