FinOps for AI: track token spend like you track cloud spend

Published February 16, 2026

Ten years ago, cloud bills grew because nobody could see which team, service, or environment was actually spending. Untagged resources, no budgets, no owner — the bill just showed up and finance asked engineering to explain it after the fact. FinOps existed to fix that: attribution, budgets, unit economics, a shared vocabulary between engineering and finance.

Token spend is running the exact same playbook right now, except faster, because the marginal cost of "just call the API one more time" is so low that nobody thinks twice about it until the invoice arrives. This happens even in orgs with excellent cloud cost discipline — tagged resources, budget alerts, the works — that still have zero visibility into which feature is quietly burning tens of thousands a month in LLM calls. Same failure mode, new line item.

No visibility, no accountability

The core problem is structural, not a discipline failure. Cloud spend was already legible because it maps naturally to infrastructure — an EC2 instance belongs to a service, a service belongs to a team. Token spend doesn't map to anything by default. A single API key gets shared across three features because setting up a new key felt like unnecessary process. A staging environment quietly runs the same prompt volume as production because nobody turned off the eval loop. Nobody owns the number because nobody can see their slice of it.

The fix is the same fix cloud FinOps used: make usage attributable before you try to optimize it. You cannot right-size, cache, or budget what you can't attribute to an owner.

Tag and meter at the gateway level

Don't rely on each service calling the LLM API directly with its own credentials and hoping someone aggregates the bill later. Put a gateway — even a thin one — between your application code and the model provider, and tag every request that passes through it.

# gateway/middleware.py
from dataclasses import dataclass
from datetime import datetime, timezone

@dataclass
class UsageRecord:
    team: str
    feature: str
    environment: str  # prod, staging, dev
    model: str
    input_tokens: int
    output_tokens: int
    cost_usd: float
    request_id: str
    timestamp: datetime

def record_usage(response, tags: dict, pricing: dict) -> UsageRecord:
    model = tags["model"]
    input_cost = response.usage.input_tokens * pricing[model]["input_per_token"]
    output_cost = response.usage.output_tokens * pricing[model]["output_per_token"]

    record = UsageRecord(
        team=tags["team"],
        feature=tags["feature"],
        environment=tags["environment"],
        model=model,
        input_tokens=response.usage.input_tokens,
        output_tokens=response.usage.output_tokens,
        cost_usd=input_cost + output_cost,
        request_id=response.id,
        timestamp=datetime.now(timezone.utc),
    )
    emit_to_metrics_pipeline(record)  # your warehouse, StatsD, whatever you already use
    return record

Every caller passes team, feature, and environment as required metadata on the request — enforced at the gateway, not left as an optional convention. If a request comes through without tags, that's a bug in the caller, and the gateway should either reject it or attribute it to an "untagged" bucket that shows up loudly on the dashboard, not silently disappear into an aggregate number.

This gets you the same thing cloud tagging gets you: a bill you can slice by team, feature, and environment, instead of one number you can only argue about.

Set budgets and alerts

Once usage is attributable, budgets follow the same pattern as cloud budgets — a monthly ceiling per team or feature, with alerts at 50/80/100% rather than a hard cutoff that breaks production mid-month.

# budgets.yaml
budgets:
  - team: support-automation
    feature: ticket-triage
    monthly_limit_usd: 8000
    alert_thresholds: [0.5, 0.8, 1.0]
    on_breach: notify_slack  # not hard_stop — this is a customer-facing feature

  - team: internal-tools
    feature: docs-search
    monthly_limit_usd: 500
    alert_thresholds: [0.8, 1.0]
    on_breach: hard_stop  # internal tool, safe to degrade

The on_breach behavior matters — a customer-facing feature usually shouldn't hard-stop when it hits a budget, because "the support bot silently stopped working" is worse than an unplanned cost overrun. A hard stop is more defensible for internal or non-critical tools where degrading gracefully (falling back to a cheaper model, or just erroring with a clear message) is an acceptable outcome. Decide this per feature, not as a global policy.

Unit economics: cost per request, per user, per feature

A monthly total tells you the bill grew. It doesn't tell you why, or whether that's a problem. Unit economics does — the same way "cost per gigabyte served" is more useful than "total S3 bill" for deciding whether a cloud cost trend is healthy or not.

Track, per feature:

-- monthly unit economics by feature
SELECT
    feature,
    date_trunc('month', timestamp) AS month,
    count(DISTINCT request_id) AS requests,
    sum(cost_usd) AS total_cost,
    sum(cost_usd) / nullif(count(DISTINCT request_id), 0) AS cost_per_request,
    count(DISTINCT user_id) AS active_users,
    sum(cost_usd) / nullif(count(DISTINCT user_id), 0) AS cost_per_user
FROM usage_records
GROUP BY feature, date_trunc('month', timestamp)
ORDER BY month DESC, total_cost DESC;

This is the query that turns "the AI bill is up again" into an actual conversation about whether it should be.

Optimization levers, roughly in order of ROI

Once you can see spend by feature, here's the order I'd generally work through — cheapest effort and lowest risk first, most invasive last. Your mileage will vary by workload, but this is a reasonable default ordering.

1. Prompt caching. If your provider supports it (most do now), and your prompts have a stable prefix — a system prompt, a set of tool definitions, retrieved context that doesn't change every request — caching can cut input-token cost meaningfully on the cached portion for near-zero engineering effort. This is usually the first lever because it's close to free to implement and has no quality tradeoff.

2. Prompt trimming. Audit what's actually in your prompts. It's extremely common to find stale few-shot examples, verbose instructions nobody's revisited since the first draft, or retrieved context that includes more than the model needs. Trimming a bloated system prompt is often a bigger win than people expect, and it also tends to improve latency.

3. Model right-sizing. Not every call needs your most capable model. Classification, extraction, simple formatting, routing decisions — these often work fine on a smaller, cheaper model, with the expensive model reserved for the steps that actually need its reasoning. This is where the eval discipline matters: don't swap to a cheaper model on a hunch, verify on your own eval set that quality holds (see the companion piece on building model evals — the same rigor applies whether you're comparing open-weight models or comparing tiers of the same provider's lineup).

4. Batching. For non-interactive workloads — nightly summarization jobs, bulk classification, embedding generation — batch APIs (where available) typically run at a substantial discount over synchronous calls, in exchange for higher latency. If the workload doesn't need a real-time response, this is close to free savings.

5. Output length control. Verbose outputs cost more and often aren't more useful. Explicit length constraints or format instructions (structured output over free-form prose, where the use case allows it) reduce output-token spend directly.

Notice what's not high on this list: renegotiating pricing, switching providers on price alone, or building a custom inference stack to avoid API costs entirely. Those are real levers eventually, but they're expensive to execute and often come with hidden costs (ops burden, quality risk, migration time) that dwarf the savings for most teams below a certain scale. Work the cheap levers first.

Build the report leadership actually reads

The report that gets ignored is a raw token count or a line item labeled "AI/ML." The report that gets read ties spend to something a non-engineer already understands: cost per team, trend over time, and — critically — what it's buying.

A structure that's worked well for me:

This is the same discipline I ended up building spendark.com around for cloud costs — the pattern of "make it visible, attribute it, then optimize" doesn't change when the line item is tokens instead of compute. The tooling differs; the discipline doesn't.

Token spend isn't going to shrink as a category — usage is growing, and it should, if it's creating value. The goal was never "spend less." It's the same goal FinOps always had for cloud: spend on purpose, know why, and be able to explain the number to someone who's going to ask.