The vibes-based engineering trap

There is a stage every team building on LLMs lives through. Someone tweaks the system prompt on Tuesday afternoon, runs three or four examples by hand, and says it feels better. The team agrees. They ship. On Friday, a customer reports that the assistant has started confidently fabricating order IDs that don't exist. Pulling up the diff, no one can quite explain why "feels better" survived the four examples and missed an entire class of regression. That is the trap.

The trap is not that the engineer was lazy. It is that the workflow had no instrument capable of detecting the regression. Four hand-picked examples are not a measurement. They are a vibe with a small sample size. The model output looked plausible on each, and plausibility is what we are most easily fooled by — by design, an LLM produces output that is locally fluent regardless of whether the underlying claim is true.

"It feels better" is a defensible engineering practice exactly nowhere. It is, however, the default — because the alternative looks expensive. It looks like building infrastructure when you wanted to ship a feature. The trick is to notice that the infrastructure is not optional; you are paying for it either way. You either pay up front, in a few days of building a small eval harness, or you pay later, in production incidents and slow rollback decisions. The second bill is bigger, and your customers help pay it.

The rest of this guide is the smallest possible eval stack that lets you say something defensible about whether change X made the system better or worse. It assumes you can call an LLM API; nothing more. It is not the whole field — production teams add adversarial sets, drift monitors, online A/B testing, human-in-the-loop, and so on — but with these five pieces in place, you stop arguing about vibes and start arguing about numbers, which is a much shorter argument.

The golden set — 20 to 100 examples you take seriously

The first artifact is a fixed list of inputs that represent what the system has to handle. Not a sample of one hundred thousand. A curated list of twenty to a hundred. Small enough that you can read all of it, large enough that you can compute averages and not be drowned in noise.

People over-think this. The golden set does not need to be statistically representative of your traffic distribution in the careful sense. It needs to span the failure modes you care about. That is a different objective, and it pushes you toward a different selection strategy.

Diverse beats balanced

If 80% of your real traffic is "what is your refund policy" and 20% is everything else, a representative golden set is 80% refund-policy questions. That set will tell you almost nothing useful — every model variant scores 95% on it, and the 5% gap where models differ is buried in the easy 80%. You want the rare and the weird over-represented. The 20% diverse tail is where prompt changes have leverage, where regressions hide, and where users notice the difference between systems. Bias the set toward the tail.

Two acquisition strategies, both required

The first source is production traffic sampling. Once you have real users, dump anonymized inputs to a table and sample weekly. Stratify by intent, by length, by whether the user came back, by whatever signal you have. Look at this dump with your own eyes — you will discover the things customers ask that you did not design for. Add those to the golden set.

The second source is hand-crafted adversarial. Inputs that probe specific weaknesses: prompt-injection attempts, ambiguous pronouns, queries with no good answer, queries that require admitting the system does not know. These rarely show up in production traffic at useful density, but they are the inputs that turn good demos into bad incidents. You have to write them yourself.

You need both. Production sampling tells you what users actually do; adversarial inputs tell you what attackers and edge cases will do. Either alone leaves a blind spot. A reasonable mix for a serious system: 60% sampled, 40% adversarial, with the adversarial set growing every time you hit a real incident in production.

Rubrics — turn "is it good?" into a checklist

The second artifact is a rubric — a written set of criteria a response either does or does not satisfy. The single most important rule about rubrics is this: vague rubrics produce vague evals. "Is the answer helpful?" looks like a rubric but is not one. Two reasonable graders will disagree on it constantly, because "helpful" is not defined, and so the rubric is doing none of the work.

Useful rubrics are decomposed into binary, rule-shaped checks. Each criterion should answer a question that can in principle be settled by looking at the response and a short reference, without asking the grader to make a judgment call. "Did the assistant include the order ID exactly as it appears in the input?" is a rule. "Did the assistant cite a real refund-policy document and not paraphrase it from memory?" is a rule. These you can run on autopilot and trust the answer.

From vague to useful — a worked example

Bad rubric, single dimension:

What goes wrong: graders anchor on different aspects of "helpful." Grader A thinks helpful means complete. Grader B thinks helpful means brief. They both rate every response a 7 because the scale invites averaging across implicit sub-judgments they have not agreed on. Inter-rater agreement is poor. The rubric is doing no work.

Useful rubric, decomposed and binary:

Four binary checks. A response gets 0–4. The score is a count of rules satisfied, not an opinion. Two graders running this rubric will disagree on individual rules sometimes, but those disagreements are diagnostic — they tell you the rule is ambiguous and needs sharpening, or that the response is genuinely on the boundary.

LLM-as-judge — using a model to grade a model

Running a binary rubric across 50 golden-set examples by hand takes a couple of hours. Doing it on every prompt change is unsustainable. The lever is using an LLM to apply the rubric — feed it the input, the response, and the rubric, and ask it to return a score per criterion. This is "LLM-as-judge."

It works better than you would expect, and worse than the hype suggests. The shape of where it works:

What LLM-as-judge is good at

• Rule-following checks. "Did the response include a code block?" "Did it begin with an acknowledgement?" The judge reads carefully; it doesn't miss obvious things.
• Factuality against a reference. "Does this response contradict the retrieved context?" Given source documents and a response, an LLM judge is remarkably good at flagging unsupported claims. Better than humans, often, who get bored on the fifth response.
• Format compliance. JSON validity, schema conformance, structural constraints — strong domain for an LLM judge.

What LLM-as-judge is bad at

• Subjective quality. "Is the response a good explanation?" The judge has its own opinions about what good explanations look like — usually the long, hedged, balanced kind. Its judgment is not your users' judgment.
• Long-output grading. Performance degrades as the candidate response gets longer. The judge stops attending uniformly, misses regressions in the middle, weights the opening and closing.
• Its own family of biases. Position bias: in pairwise judgments, the response shown first is preferred more often than chance. Verbosity bias: longer responses are rated higher even when they're worse. Sycophancy: a response framed as "improved" or "the better version" is rated higher. Self-preference: a model judge tends to prefer outputs from its own family. You have to know these exist or you will measure them by accident.

The cost shape

An LLM judge run on a 50-prompt golden set with a 4-criterion rubric, comparing two candidate responses each, is roughly 50 × 2 × 4 = 400 judgments. At a frontier model price of ~$3–15 per million input tokens, and ~1–2K input tokens per judgment, you're looking at $2–$20 per full eval run. Cheap enough to run on every PR. Expensive enough that you don't want to be wasteful. Most teams use a smaller, cheaper model for the judge and validate against the frontier model periodically.

The agreement problem — does your judge actually agree with humans?

An LLM judge that disagrees with careful human graders is not a judge — it's a parallel process that you have learned to call "the grader." The whole point of automating with a judge is that its scores are a usable substitute for human scores. So you have to verify that they are.

The standard instrument is Cohen's kappa (κ), which measures agreement between two raters on categorical labels, adjusted for the agreement you would get by chance. Two raters who agree on every label get κ = 1.0. Two raters who agree at chance levels get κ ≈ 0. Two raters worse than chance get negative kappa, which mostly just means you've done something very strange.

The interesting thing about kappa is that you can read it as a meta-eval of your rubric. The kappa between two careful humans on the same rubric tells you whether the rubric is even gradable. The kappa between your LLM judge and a human tells you whether the judge can stand in for the human.

Reading kappa scores

• κ < 0.4 — your rubric is broken. Two graders applying it produce nearly independent labels. Either the rule wording is ambiguous, the criterion is subjective, or the examples don't actually map cleanly to pass/fail. Rewrite before measuring anything else.
• 0.4 ≤ κ < 0.6 — moderate. You can use this, with caveats. Differences in eval scores between two systems need to be substantial to be trusted.
• 0.6 ≤ κ < 0.8 — good. The rubric works. Small differences between systems become detectable. This is the working range.
• κ ≥ 0.9 — suspicious. Either your rubric is asking a trivially easy question ("does the response contain the word 'refund'?"), or your graders are colluding (one is anchoring to the other's labels). Verify the rubric is actually discriminating something hard.

The sweet spot is roughly 0.65–0.85: the rubric is hard enough to be measuring something, agreed-on enough to be reliable. Push outside that band in either direction and you should investigate before trusting the numbers.

You'll grade five prompts yourself in the next section. We'll show your kappa against a pre-baked LLM-judge in real time. It is an instructive exercise — sometimes you discover your own internal rubric was sharper than the one you wrote down.

When scalar scores fail — the pairwise trick

Here is the central failure mode of scalar evals: when two systems are both reasonable, their average scores bunch up. Model A averages 7.5, Model B averages 7.3, and you don't know whether the gap is signal or noise. Scalar evals are most informative when one system is clearly worse, and least informative when the choice is hardest — which is exactly when you needed the information.

The fix is pairwise comparison. Instead of asking "score this response from 1 to 10," you show both candidate responses side by side and ask "which is better, A or B?" then aggregate as a win rate. The same five examples that produced an indistinguishable 7.5-vs-7.3 will often produce a 4–1 pairwise split. The same model, the same prompts, the same judge — but in pairwise mode it is forced to express a preference at the margin, and the margin is where the truth lives.

The demo below is a small working eval bench. The golden set is five prompts. Two candidate systems, A and B, produce responses for each. A 3-criterion binary rubric is applied. You can flip between scalar grading and pairwise grading and see how the verdict changes.

If you ran scalar mode first, you saw two close averages and a verdict that hedged. Switch to pairwise. Pairwise is uniformly more sensitive at the margin because it converts "both look fine" into "one of these is slightly less worse on prompt 3." When the underlying systems are similar but not identical, this is the eval mode that distinguishes them. When the systems are wildly different, scalar mode catches it too — pairwise is the upgrade you reach for when the answer matters more.

Grade these yourself — meet your kappa

One more demo, smaller. Below are five responses from Model A. A pre-baked LLM judge has graded each as PASS or FAIL against the criterion "no fabricated facts." Grade them yourself by clicking. As you grade, the kappa between you and the judge updates live. Anything ≥0.6 means you and the judge are aligned enough to trust its work on the next 500. Anything <0.4 means you should investigate before you let it grade unsupervised.

Putting it in your dev loop — the production stack

A working eval is only useful if it runs on every change. The discipline is to wire the eval into the workflow such that a prompt change without a passing eval cannot be merged. Otherwise the team's behavior reverts to vibes within a quarter, no matter how good the infrastructure was at the start.

A minimal production loop

1. Golden set in version control. A JSON or YAML file alongside the code. Every entry has an ID, the input, optional metadata (intent, source, severity), and optional expected behavior tags. Reviews of PRs include reviews of golden-set additions.
2. Eval harness as a script. One command runs your golden set against the current system and emits a structured report — per-criterion pass rates, win rates against the previous version, regressions flagged. Total runtime under two minutes for <100 prompts, or the team will skip it.
3. CI gate on the eval. The PR can't merge if the overall win rate against main drops below 50% or a specific named regression test fails. Make the gate stricter for higher-stakes routes.
4. Kappa-validated judge, periodically re-calibrated. Every quarter, hand-grade 30 examples, compute kappa against the judge, alert if it has drifted below 0.6. Models update, judges drift, your team learns more about what counts as a good response.
5. An incident → golden-set pipeline. Every production incident ends in a postmortem that adds the failing case (and a handful of near-neighbors) to the golden set as a permanent regression test.

Latency and cost budgets

Two practical numbers to hold: eval runs must take less than five minutes wall-clock, or developers will work around them; an eval run must cost less than a coffee, or finance will work around them. Most teams hit both numbers by using a smaller judge model for routine PR-level evals and a frontier model only for monthly calibration runs.

The false-confidence trap

The final caveat, and it is the most important one: you cannot eval your way to a good system in isolation. A perfect 100% pass rate on a 50-example golden set does not mean the system works in production. It means it works on the cases you wrote down. The golden set is a sample, not a coverage proof. Real users find inputs you did not imagine, and they find them every week.

The defense is the loop. Sample production traffic continuously. Grow the golden set. Run the eval. Pay attention when the eval starts saying "fine" but customers start saying "broken" — that gap is the most actionable signal you have, and it means your eval has stopped measuring something that matters. Rewrite the rubric, expand the set, recalibrate the judge. The eval is a living instrument, not a finished artifact, and the discipline is in keeping it honest.