What self-hosted AI actually costs: GPU box vs API bills

Published June 2, 2025

Everyone asks "is self-hosting cheaper?" as if there's a single answer. There isn't. There's a break-even curve, and where you land on it depends almost entirely on one variable people underweight: utilization. I do FinOps work for a living, so let me walk through the actual math instead of the vibes.

The two things you're actually comparing

API pricing is pay-per-token. You're charged for what you use, when you use it, and the provider absorbs the idle time, the hardware failures, the driver updates, and the capacity planning. You're buying a finished service.

Self-hosting is pay-for-capacity. You buy or rent a GPU box that's provisioned for peak load, and it costs roughly the same whether it's running at 5% or 95% utilization. You're buying a machine, and the token cost is whatever falls out of how well you use that machine.

This is the same tension as any make-vs-buy infrastructure decision — it's just newer.

GPU cost: buy vs rent

I'll give ranges, not quotes, because prices move and your negotiated rate will differ from list price. Treat everything here as "roughly, as of mid-2025" and re-check before you budget against it.

Buying. A single high-memory datacenter GPU (the class you'd want for a 30-70B parameter model at reasonable batch sizes) has historically run somewhere in the high four figures to low five figures USD per card, and a usable multi-GPU server (4-8 cards plus the host, networking, and storage around them) can run from the mid five figures into six figures USD depending on GPU count and generation. That's capital expenditure, up front, with a multi-year depreciation clock. You also need somewhere to put it — a datacenter rack, or at minimum a well-ventilated server room, plus redundant power.

Renting. Cloud GPU instances (on-demand) for comparable hardware have generally run on the order of a few USD per GPU-hour on the major clouds, less on specialized GPU cloud providers, and less still if you commit to reserved or spot capacity. Renting converts your capex into opex and removes the "what do we do with this hardware in three years" problem, at the cost of a higher long-run price per GPU-hour if you run it constantly.

The crossover point — where buying beats renting — is usually somewhere in the range of continuous, high-utilization use over a year or more. If you're running the box most hours of most days, ownership wins over time. If usage is bursty or you're still validating the use case, rent first. Don't buy hardware to prove a hypothesis.

The variable that actually decides this: utilization

Here's the thing nobody puts on the slide. An idle GPU is not "saving money for later" — it's a sunk cost accruing zero return, every hour it sits there. A GPU server that costs a fixed amount per month is a fixed cost regardless of whether you send it one request or one million. Your effective cost per token is literally:

cost per token = (fixed infra cost for the period) / (tokens actually served in that period)

Double your utilization, halve your effective cost per token, with no change to the hardware bill. This is why the same box can look like a bargain or a disaster depending entirely on how well the org actually uses it. A GPU server sized for a 50-person company's peak load, used only by three enthusiastic early adopters, is one of the more expensive ways to generate text in existence. The same box, fully adopted across the company with real workloads routed to it, can be dramatically cheaper than the equivalent API spend.

This is also why "we bought a GPU server and it's saving us money" claims need scrutiny. Saving money against what baseline, at what utilization, amortized over what period? Compare your actual fully-loaded cost-per-token against actual API pricing for equivalent quality, not against a hypothetical.

Throughput: what a GPU can actually serve

Tokens-per-second is the number that turns "we bought a GPU" into "we can serve N people." It depends heavily on model size, quantization, batching, and hardware, so don't anchor on a specific number — but the reasoning matters more than the number.

A serving stack like vLLM using continuous batching can serve many concurrent requests against one model copy far more efficiently than naively running one request at a time — this is the difference between a GPU box that supports a handful of users and one that supports a whole department. If you're evaluating hardware, evaluate the serving stack's batched throughput for your actual model and prompt lengths, not the marketing spec sheet's single-request latency number. Run a load test that resembles your real usage pattern (concurrent users, typical prompt/response length) before sizing hardware — don't size off vibes or a vendor's best-case benchmark.

When API pricing wins

• Low or spiky volume. If usage is unpredictable — a feature used by a subset of users, occasional batch jobs, anything without a steady baseline — you cannot hit the utilization needed to make owned or rented GPU capacity pay off. Pay-per-token wins by default here.
• You're still validating the use case. Don't buy hardware for a workload you're not sure will exist in six months. Use the API, measure real usage, then decide.
• Frontier capability matters more than unit cost. If the task genuinely needs the best available model, and open-weight alternatives underperform on it, the API isn't just cheaper — it may be the only option that clears the quality bar.

When self-hosting wins

• Steady, high volume. If you have a predictable, ongoing workload that keeps a GPU busy most of the time — internal chat used company-wide, a production feature at real scale, a batch pipeline running continuously — the fixed-cost-over-high-utilization math starts to beat pay-per-token, often decisively.
• Compliance forces it. Sometimes the decision isn't cost at all — it's that the API isn't a legal option (see the companion piece on self-hosted AI for compliance reasons). In that case you're not comparing cost, you're comparing "self-host" against "don't do this at all," and the cost conversation becomes about running it efficiently, not about whether to run it.

Hidden costs people forget to budget

• Model updates. New model versions come out regularly. Someone has to evaluate whether the new version is actually better for your use cases, test it against your prompts, and roll it out. This is ongoing work, not a one-time setup cost.
• Evals. If you care whether the system is actually working, you need a way to measure output quality over time, ideally before you change models or prompts, not after users complain. Building and maintaining even a lightweight eval set is real effort.
• On-call. If the GPU box goes down and it's serving production traffic, someone gets paged. That's a cost — both the tooling to alert and the human time to respond — that an API vendor absorbs invisibly into their SLA.
• Power and cooling. Datacenter GPUs draw serious power, and if you're running your own hardware rather than renting cloud capacity, power and cooling are a real, recurring line item — not huge relative to the hardware cost, but not zero either, and often missing from the initial budget conversation.
• Ops time. Patching, driver updates, capacity planning, security reviews — someone's time is going into keeping this running, and that person's salary is part of the true cost even if it never shows up on the GPU invoice.

These hidden costs are exactly the ones that turn a spreadsheet that looked favorable into a break-even that lands somewhere much less flattering — budget for them up front rather than discovering them a quarter in.

The FinOps framing

Treat this exactly like any other infrastructure spend decision: model the cost per unit of value delivered, not the sticker price of the hardware. I built spendark.com as a free calculator for exactly this kind of cloud cost reasoning — the same discipline applies here. Estimate your realistic utilization before you commit, re-check it against actual usage after three months, and be willing to admit if the math doesn't hold up. The GPU box that made sense on a spreadsheet with optimistic utilization assumptions can quietly become the most expensive line item in your infrastructure budget if adoption doesn't show up. Measure, don't assume.