Choosing an open-weight model for internal use: build an eval, ignore the leaderboard

Published April 14, 2025

Someone on your team will eventually ask "which open-weight model should we use for X," and the instinctive answer is to check a leaderboard. Don't. Leaderboard benchmarks measure performance on tasks that are almost never your tasks, on prompts that are almost never your prompts, at a quantization that is almost never what you'll actually deploy.

The only benchmark that means anything is the one built from your own use case. This is more work than reading a leaderboard. It's also the only approach that produces an answer you can trust.

Why the leaderboard doesn't transfer

A few concrete reasons, not just "benchmarks are imperfect":

None of this means leaderboards are useless for a first pass — they're a reasonable way to narrow a field of thirty models down to four or five worth testing properly. They're just not a substitute for testing on your own data.

Build a 30-50 case eval set from real use

Pull real examples from what you're actually trying to automate — not synthetic examples you write to be "representative," but actual production data:

30-50 cases is enough to catch systematic failure modes without becoming its own multi-week project to maintain. Skew the set toward the hard 20% — a case the model gets right effortlessly tells you less than a case that's genuinely ambiguous.

Structure each case as an explicit input/expected-output pair, even when "expected output" is a rubric rather than an exact string:

# eval_cases/support_ticket_042.yaml
id: support_ticket_042
category: billing_dispute
input: |
  Customer message: "I was charged twice this month and no one
  has responded to my last two emails. This is unacceptable."
expected_behavior:
  - Classifies as: billing_dispute, high_urgency
  - Does NOT promise a specific refund amount (no pricing authority)
  - Escalates or flags for human review given repeated contact
  - Tone: acknowledges frustration, does not sound scripted
notes: >
  Real case from Q1 2025 backlog. Original model response incorrectly
  classified this as "general_inquiry" and missed the urgency signal
  from "no one has responded."

That notes field matters more than it looks — it's the institutional memory of why this case is in the set. Six months from now, when someone asks "can we remove this case, it's slowing down the run," the note is what stops you from deleting the one case that caught a real regression.

Scoring: rubric plus LLM-judge plus human spot-check

A rubric with a pass/fail per criterion, not a single "good/bad" label — this is what makes the eval diagnostic rather than just a scoreboard.

rubric:
  - criterion: correct_classification
    description: "Ticket category matches expected_behavior"
    weight: 3
  - criterion: no_unauthorized_commitment
    description: "Model does not promise refunds, discounts, or SLAs it has no authority over"
    weight: 3
  - criterion: urgency_signal_caught
    description: "High-urgency language (repeated contact, strong sentiment) results in escalation flag"
    weight: 2
  - criterion: tone_appropriate
    description: "Response acknowledges the customer's stated frustration without sounding templated"
    weight: 1

Automate the first pass with an LLM judge, using a stronger model than the one under test to score each case against the rubric. This scales — you can re-run it on every change — but it isn't free of its own failure modes: judge models have their own biases (they tend to favor longer, more hedged answers, for instance), so don't treat the judge's score as ground truth.

Human spot-check a sample every run, not just at the start. A reasonable cadence: review 100% of failing cases (however the judge scored them) plus a random 20% of passing cases, to catch the judge rating something as correct that a human would flag. This is the step teams skip once the automated eval is running smoothly, and it's exactly the step that catches judge drift.

Test at the quantization you'll actually deploy

This is the single most commonly skipped step, and it's the one that most directly determines whether your eval result means anything. If you're going to run the model at 4-bit quantization in production because that's what fits your GPU budget, evaluate the 4-bit version. Full-precision numbers from a leaderboard, or even from your own eval run on an unquantized checkpoint, tell you nothing about the model you'll actually ship.

Quantization degrades models unevenly — some architectures and some tasks (especially anything requiring precise numeric reasoning or exact-format output) degrade more than others at the same bit width. The only way to know your specific model's degradation on your specific tasks is to run the eval at the target quantization.

A simple harness sketch (Python)

You don't need an eval framework to start — a loop over YAML files and a scoring function gets you most of the value.

import glob
import yaml
import json
from dataclasses import dataclass, field
from pathlib import Path

@dataclass
class EvalResult:
    case_id: str
    scores: dict[str, bool] = field(default_factory=dict)
    total_weight: int = 0
    earned_weight: int = 0
    judge_notes: str = ""

def load_cases(directory: str) -> list[dict]:
    cases = []
    for path in glob.glob(f"{directory}/*.yaml"):
        with open(path) as f:
            cases.append(yaml.safe_load(f))
    return cases

def run_model_under_test(model_client, case: dict) -> str:
    # model_client wraps whatever inference server you're testing
    # (vLLM, llama.cpp, TGI) at the target quantization
    return model_client.complete(case["input"])

def judge_response(judge_client, case: dict, response: str, rubric: list[dict]) -> EvalResult:
    result = EvalResult(case_id=case["id"])
    judge_prompt = f"""
You are scoring a model response against a rubric. For each criterion,
answer only "PASS" or "FAIL" followed by a one-sentence reason.

Case input: {case["input"]}
Expected behavior: {case["expected_behavior"]}
Model response: {response}

Criteria:
{json.dumps(rubric, indent=2)}

Respond as JSON: {{"criterion_name": {{"pass": bool, "reason": str}}, ...}}
"""
    judged = judge_client.complete(judge_prompt, response_format="json")
    parsed = json.loads(judged)

    for criterion in rubric:
        name = criterion["criterion"]
        weight = criterion["weight"]
        result.total_weight += weight
        passed = parsed.get(name, {}).get("pass", False)
        result.scores[name] = passed
        if passed:
            result.earned_weight += weight

    return result

def run_eval_suite(directory: str, model_client, judge_client) -> list[EvalResult]:
    cases = load_cases(directory)
    results = []
    for case in cases:
        response = run_model_under_test(model_client, case)
        rubric = case.get("rubric") or load_default_rubric(case["category"])
        result = judge_response(judge_client, case, response, rubric)
        results.append(result)
    return results

def summarize(results: list[EvalResult]) -> None:
    total_score = sum(r.earned_weight for r in results)
    total_possible = sum(r.total_weight for r in results)
    print(f"Overall: {total_score}/{total_possible} ({100 * total_score / total_possible:.1f}%)")

    failing = [r for r in results if r.earned_weight < r.total_weight]
    print(f"\n{len(failing)} cases with at least one failed criterion:")
    for r in failing:
        failed = [k for k, v in r.scores.items() if not v]
        print(f"  {r.case_id}: failed {failed}")

The load_default_rubric and model_client/judge_client wiring are stand-ins for your actual inference stack — the shape that matters is: load real cases, run the candidate model, score with a judge against an explicit rubric, and produce a per-criterion breakdown, not just a single number.

Re-run on every change, keep it in CI

Treat this exactly like a regression test suite, because that's what it is. Wire it into CI so it runs automatically on:

# .github/workflows/eval.yml
name: Model Eval Suite
on:
  pull_request:
    paths:
      - "prompts/**"
      - "eval_cases/**"
      - "model_config.yaml"
jobs:
  run-eval:
    runs-on: [self-hosted, gpu]
    steps:
      - uses: actions/checkout@v4
      - name: Run eval suite
        run: python run_eval.py --directory eval_cases/ --min-score 0.85
      - name: Fail if below threshold
        run: |
          if [ "$(cat eval_score.txt)" -lt 85 ]; then
            echo "Eval score below threshold, blocking merge"
            exit 1
          fi

Set the threshold from your own baseline run, not an arbitrary number — the point is catching regressions relative to where you currently are, not hitting a score someone picked out of thin air. When a case starts failing that used to pass, that's the signal a prompt change or quantization swap broke something real — which is exactly the thing a leaderboard score would never have told you.

The eval set is never finished. Add a case every time production surfaces a failure the current set didn't catch. That's what turns it from a one-time model-selection exercise into an actual safety net.