One GPU server for a 50-person company: minimum viable internal AI

Published July 21, 2025

Most of what gets written about self-hosted AI is sized for companies with a platform team and a dedicated ML budget. If you're at a 50-person company and you've been told to "figure out internal AI," you don't need that. You need one box, a handful of well-chosen pieces of software, and the discipline to not build more than you need yet.

This is that setup.

The core idea: one box, not a platform

At this scale, the goal is a single GPU server that a chunk of the company can use for chat, drafting, summarization, and basic coding assistance, with enough control to satisfy IT and security, and none of the complexity that a 500-person org would need. One machine. Not a Kubernetes cluster, not a multi-region deployment, not a fine-tuning pipeline. One box, sized correctly, running well.

Hardware: 1-2 GPUs, sized to the model you actually need

Don't start from "what's the biggest model we can afford." Start from "what tasks do we actually need this for," pick a model size that comfortably handles those tasks, then size hardware to that model.

Rough VRAM tiers to think in terms of (these move with quantization choices, so treat as directional, not exact):

• ~24GB VRAM (a single higher-end workstation/prosumer GPU): comfortably runs quantized 7-14B models, and smaller quantized versions of larger models with some quality trade-off. Good for a pilot, or for a company where the use case is mostly drafting and simple Q&A.
• ~48-80GB VRAM (a single datacenter-class GPU): comfortably runs quantized 30-70B models, which is where most of the practical quality-for-general-tasks lives right now. This is the realistic sweet spot for a 50-person company that wants something genuinely useful, not just a demo.
• 2x GPUs in the 48-80GB class: either lets you run a larger model split across cards, or — often the better move for a company this size — run two model instances side by side (say, a fast small model for quick tasks and a larger one for harder tasks) or simply get headroom for concurrent users without queuing.

For a 50-person company, a single well-specified datacenter GPU, or two mid-tier ones, is almost always enough. You are not going to have 50 people hammering it concurrently with long contexts — real concurrent usage at this size is usually a handful of people at once, even with decent adoption.

The stack

Inference: Ollama or vLLM

For a company this size, start with whichever matches your actual concurrency needs:

• Ollama if you expect light, bursty use — a few people trying it, one team actively using it. Dead simple to run and update.
• vLLM if you already know you want this available company-wide from day one, or if early usage shows queuing under Ollama. Better throughput under concurrent load via continuous batching, more moving parts to manage.

It's fine to start on Ollama and migrate to vLLM later once usage justifies it — don't over-engineer the pilot.

Gateway: a LiteLLM-style proxy

Put a gateway between your users' tools and the inference server. This single piece of infrastructure buys you a lot: one place to issue and revoke API keys, per-team or per-user usage tracking (useful both for cost visibility and for spotting who's actually using the thing), rate limiting so one runaway script doesn't starve everyone else, and a stable OpenAI-compatible endpoint so you can swap the backend model later without reconfiguring every tool that talks to it.

# docker-compose.yml — minimal internal AI stack for a small company
services:
  ollama:
    image: ollama/ollama:latest
    runtime: nvidia
    volumes:
      - ollama_data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

  litellm:
    image: ghcr.io/berriai/litellm:main-stable
    depends_on:
      - ollama
    environment:
      - LITELLM_MASTER_KEY=${LITELLM_MASTER_KEY}
    volumes:
      - ./litellm-config.yaml:/app/config.yaml
    command: ["--config", "/app/config.yaml", "--port", "4000"]
    ports:
      - "4000:4000"

  openwebui:
    image: ghcr.io/open-webui/open-webui:main
    depends_on:
      - litellm
    environment:
      - OPENAI_API_BASE_URL=http://litellm:4000/v1
      - OPENAI_API_KEY=${LITELLM_MASTER_KEY}
      - WEBUI_AUTH=true
    volumes:
      - openwebui_data:/app/backend/data
    ports:
      - "3000:3000"

volumes:
  ollama_data:
  openwebui_data:

That's the whole minimum viable stack in one file. Not a lot of moving parts — that's the point.

SSO in front

Don't ship this with local username/password accounts. Put SSO (SAML or OIDC, whatever your identity provider already supports — Okta, Entra ID, Google Workspace) in front of the chat UI and the gateway. This is a small amount of setup work and it's the difference between "auditable internal system" and "shadow IT with extra steps." Most chat UIs, including Open WebUI, support OIDC directly — use it from day one rather than retrofitting it once fifteen people have local accounts.

Basic logging

You don't need a full observability platform. You need: who made each request, when, to which model, and — depending on your data sensitivity and retention policy — enough of the prompt/response to investigate a problem if one comes up. The gateway layer is the natural place to capture this, since every request already flows through it. Decide your retention period deliberately (not "forever by default") and document it — this is usually the first question security will ask.

Chat UI: Open WebUI or similar

Give people a interface, not just an API endpoint. Open WebUI (or a comparable self-hosted chat UI) gives non-technical staff a ChatGPT-like experience against your internal models, with per-user history, and it talks to your gateway over the same OpenAI-compatible API your other internal tools will use. This matters more than it sounds — adoption lives or dies on whether using it feels as easy as the tools people already have.

What NOT to build yet

This is the part that saves you the most time and money.

No fine-tuning. Fine-tuning is expensive in engineering time, requires a real evaluation process to know if it even helped, and for the vast majority of internal-tool use cases, a good base model with a solid system prompt gets you 90% of the value at a fraction of the effort. Don't reach for fine-tuning until you have a specific, measured gap that prompting can't close.

No custom RAG pipeline until there's a proven need. Retrieval-augmented generation over your internal docs sounds like the obvious next step, and it's also a genuine engineering project — chunking strategy, embedding model choice, vector store, retrieval quality tuning, keeping the index fresh as documents change. Don't build this speculatively. Wait until you have a concrete, recurring request ("I need this to answer questions against our internal wiki") and even then, start with the smallest version that could work before building anything elaborate.

No multi-model zoo on day one. One good general-purpose model is enough to start. Resist the urge to stand up five specialized models before you know which tasks people actually bring to the system.

The pattern across all three of these is the same: resist building infrastructure ahead of demonstrated need. It's much cheaper to add a piece later, once a pilot has told you it's actually necessary, than to maintain something speculative from day one.

Rollout: start with one team, measure usage

Don't announce this company-wide on day one. Pick one team — ideally one with a mix of skeptics and enthusiasts, not just the most AI-eager people in the building — and give them access first. You want honest signal on whether this is actually useful, not just enthusiasm from people who'd use it regardless.

Watch three things during the pilot: actual usage (does it get used after the first week's novelty wears off), what tasks people bring to it (this tells you if your model choice and system prompts are matched to real needs), and where it fails (this tells you the boundary of what your current setup can handle, and whether that boundary matters).

Only expand company-wide once the first team's usage looks like a habit, not a novelty. When you do expand, revisit your GPU sizing against real concurrent usage numbers from the pilot rather than guessing again — you'll have actual data by then, which beats any estimate in this article.

The failure mode to avoid isn't under-building — it's over-building before anyone's used the thing. One box, a gateway, SSO, a chat UI, and a small pilot team will tell you more in a month than a quarter of architecture planning will.