The LLM gateway pattern: one internal API for hosted and local models

Published January 20, 2025

If more than one service in your codebase imports an OpenAI or Anthropic SDK directly, you already have the problem this pattern fixes. It's a small architectural decision that pays off the first time you need to swap a model, and it compounds every time after that.

Why direct provider coupling hurts

The failure mode is boring and predictable. Service A calls the OpenAI SDK directly. Service B, built six months later by someone else, does the same thing but with different retry logic and no logging. Service C calls Anthropic instead, because that's what its author preferred that week. Now you have three integration points, three sets of API keys scattered across config, and no single place to answer "how much are we spending on LLM calls this month, and who's spending it."

Concretely, direct coupling costs you:

Vendor lock. Swapping a model means touching every service that calls it. What should be a config change becomes a multi-PR migration.

No usage visibility. Spend is scattered across provider dashboards, none of which know which internal team or feature triggered a given call. You find out about a cost spike from the invoice, not from a dashboard.

No policy enforcement. Rate limits, PII filtering, content policies — if you want these enforced consistently, you either duplicate the logic in every service or you enforce it nowhere and hope.

None of this is a hypothetical for a large org. It shows up as soon as you have two teams making LLM calls independently.

The gateway pattern

The fix is the same fix you'd apply to any other external dependency: put one internal API in front of it. Every service talks to your gateway using a single, stable, OpenAI-compatible interface. The gateway is the only thing that talks to the actual providers — hosted APIs like OpenAI, Anthropic, and others, plus self-hosted models running on vLLM or Ollama.

From the calling service's point of view, nothing changes when the backend changes. You swap a model name in the gateway's routing config, not in application code. A team building a new feature makes one HTTP call to an internal endpoint; whether that call ends up hitting a hosted API or a local GPU box is the gateway's decision, not theirs.

This is not a novel idea — it's the same reason you don't let every microservice open its own connection straight to a third-party payment processor. You put a service in front of it.

What a gateway actually gives you

Model swapping without code changes. Point a route at a different model or provider in config. Services calling that route don't need a deploy.

Spend tracking per team or per key. Every request goes through one place, so you can tag it — by API key, by service, by team — and get a real answer to "who's spending what" without reconciling three provider invoices.

Rate limits. Per-team or per-key limits enforced centrally, instead of every service implementing (or forgetting to implement) its own.

Audit logs. One place to see every prompt and completion that left the building, which matters both for debugging and for anyone who eventually asks a compliance question.

PII filtering hooks. A single choke point where you can scan or redact before a request leaves your infrastructure toward a third-party API — much harder to guarantee if every service is calling providers directly.

Fallbacks. If your primary provider is down or rate-limiting you, the gateway can retry against a secondary provider or a local model without the calling service knowing anything happened.

LiteLLM, and the alternatives

The most common open-source choice for this right now is LiteLLM. It speaks the OpenAI-compatible API shape on the way in, and translates to whatever the underlying provider actually expects on the way out — OpenAI, Anthropic, Bedrock, Vertex, a local vLLM or Ollama endpoint, and a long list of others. It also ships a proxy/gateway mode specifically for this use case, with built-in spend tracking, key management, and rate limiting.

It's not the only option. Depending on what you already run:

• Cloud-native gateways from AWS, Azure, and GCP if you're already deep in one cloud and want the gateway to be a managed service rather than something you operate.
• API management layers like Kong or Apigee, if you already use them for non-LLM APIs and want LLM traffic to go through the same policy engine.
• Building a thin one yourself — for a small number of routes, a lightweight internal proxy is sometimes less operational overhead than adopting a new dependency. I wouldn't default to this, but it's a legitimate option if your needs are narrow.

I'd default to LiteLLM for most teams starting from zero: it's open source, self-hostable, has a genuinely large provider list, and the proxy mode is built for exactly this pattern rather than bolted on.

A minimal config example

This is illustrative, not a drop-in production config — treat the exact keys as a shape to adapt, not gospel.

model_list:
  - model_name: default-chat
    litellm_params:
      model: gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY

  - model_name: default-chat
    litellm_params:
      model: claude-sonnet-4-5
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: local-llama
    litellm_params:
      model: openai/llama-3.1-8b-instruct
      api_base: http://vllm-internal:8000/v1
      api_key: "not-needed"

router_settings:
  routing_strategy: usage-based-routing
  fallbacks:
    - default-chat: ["local-llama"]

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/LITELLM_DATABASE_URL

Two entries under the same model_name gives you a pool the router can load-balance or fail over across. The fallbacks block means a default-chat request that fails against both hosted providers drops to the local model rather than failing outright — worse quality, but the request doesn't just die.

Pitfalls

The gateway becomes a single point of failure. You've traded "three services with independent, uncorrelated failure modes" for "one service whose downtime takes out all LLM-dependent features at once." Run it with real redundancy — multiple instances behind a load balancer, health checks, and a plan for what happens to callers when it's down. Don't stand up a single instance and call it done.

Latency overhead. Every request now makes two hops instead of one. In practice this overhead is usually small relative to model inference time, but it's not zero, and it's worth measuring on your own traffic rather than assuming it's negligible.

Streaming quirks. Streaming responses are the part most likely to break when you insert a proxy layer, especially if you're translating between different providers' streaming formats. Test streaming specifically — don't assume that because non-streaming requests work, streaming does too. Long-running streamed responses are also where you'll find timeout and buffering issues that don't show up in a quick manual test.

Config sprawl inside the gateway itself. The whole point of this pattern is to centralize complexity instead of scattering it — but centralized complexity is still complexity. A routing config with a dozen models, fallback chains, and per-team overrides needs the same care as any other piece of infrastructure config: reviewed changes, not hand-edited in production.

Where to start

If you've got more than one service calling a model provider directly, that's the signal. Stand up LiteLLM in proxy mode, point your newest service at it first, prove out spend tracking and fallbacks on low-stakes traffic, then migrate the rest. You don't need to solve PII filtering and multi-region failover on day one — the value of "one place to change the model" shows up immediately, before you've built out the rest.