When to Keep Humans in the Loop: The Case for Manual Eval

Keep humans in the loop for judgments that are subjective, high-stakes, or novel — anything a rubric can't fully specify in advance. Automate the rest: factual checks, format compliance, and anything with a clear right answer. The decision isn't "human or automated" as a permanent camp; it's a per-dimension call you revisit as failure modes get discovered and codified.

Quick Answer: Automate objective, repeatable checks (format, factuality against a source, numeric thresholds). Keep humans for subjective quality, high-stakes edge cases, and novel failure modes an automated judge hasn't seen yet. Bootstrap from human labels to a calibrated LLM judge, then keep humans spot-checking forever — not just at launch.

The Decision Rule: What to Automate vs. What Stays Human

A judgment is a good candidate for automation when it's repeatable, has a defined correct answer, and doesn't require contextual taste. It should stay human when it's subjective, carries real consequences if wrong, or is a failure mode nobody has seen and codified yet.

This isn't a binary choice you make once per product. It's a per-dimension decision that belongs in your eval plan alongside your golden dataset and your scoring rubric. Some dimensions of the same feature will sit on opposite sides of the line.

The pattern across every row: automation works where "correct" is a fact, not an opinion. The moment a judgment depends on audience, context, or stakes that shift case-by-case, you're buying false confidence by automating it prematurely. Numeric thresholds only work for evals that can actually fail cleanly — if you can't write down the failing condition, you don't have a metric yet, you have a hope.

Why teams over-automate anyway

Automated evals are cheap to run at scale and produce a tidy dashboard number, which makes them organizationally seductive even when the underlying judgment doesn't compress into a number. The tell is a metric that keeps passing while users keep complaining. That gap is usually evidence the automated check is measuring a proxy, not the thing anyone actually cares about.

Hamel Husain, an ML engineer who has written extensively and practically about production LLM evaluation, has argued that teams frequently skip a rigorous human-labeling phase entirely and jump straight to an automated judge — and then can't explain why their eval scores and their user complaints disagree. The fix isn't "never automate." It's "never automate a dimension you haven't first watched a human get right and wrong."

Where Human Judgment Still Wins — and Why It's Not Going Away Soon

Humans stay essential for subjective quality, high-stakes decisions, and failure modes nobody has cataloged yet — three categories where an automated check either can't be specified precisely enough or can't be trusted to catch something it wasn't built to look for. This isn't a temporary gap that better models will close; it's structural.

Subjective quality is the biggest category. Whether a response feels warm versus curt, confident versus arrogant, or appropriately concise versus abrupt depends on audience and context a rubric can approximate but never fully capture. Two reasonable people can disagree about a "good" response to the same prompt — that's not a measurement failure, it's the nature of the judgment. If a task produces genuine disagreement between two skilled humans, an automated judge trained to mimic "the average human" just launders that disagreement into false precision.

High-stakes decisions are the second category — anywhere a wrong answer has outsized cost: medical, legal, financial, safety-adjacent, or anything touching a vulnerable user. The case for human review here isn't about measurement accuracy; it's about accountability, since a 98% automated pass rate means little if the 2% failure is the one that harms someone. OpenAI's GPT-4 system card described using dozens of external red-teamers, because some risks only surface when a person actively tries to break the system in ways no automated suite was built to probe.

Novel failure modes are the third, and the most easily forgotten. Every automated eval — rule-based or LLM-judge — can only detect failure patterns someone already anticipated and encoded. A genuinely new way the model goes wrong will sail through every existing check clean, because nothing was written to catch it. Humans reviewing raw transcripts, not just aggregate scores, are still the primary mechanism for discovering the failure mode you'll write tomorrow's automated check for.

A useful lens for spotting which category a judgment falls into: ask whether the output actually accomplishes the underlying job the user hired the product to do, in jobs-to-be-done terms, rather than whether it merely matches a surface pattern. JTBD framing routinely surfaces judgments — did this actually help the user make progress? — that no keyword match or format check will ever capture.

Emotional and journey-stage judgments

Some quality dimensions are tied to where a user is in their experience, not just what the output says. A response that's technically accurate but tone-deaf to a frustrated user mid-escalation is a failure a static rubric won't catch, because the "correct" tone shifts with the emotional arc mapped in a customer journey. These context-dependent judgments are near-impossible to automate reliably and are exactly where a human reviewer, briefed on the situation, earns their keep.

The Bootstrap Pattern: Humans Label, Then Calibrate an LLM Judge, Then Spot-Check Forever

The bootstrap pattern is a three-stage sequence: humans label a representative sample first, an LLM judge is calibrated against those labels until it agrees with humans at an acceptable rate, and then humans keep spot-checking the judge's output indefinitely rather than walking away once it's "calibrated." Skipping any of the three stages is where most eval programs quietly go wrong.

1. Human labeling first. Before any automated judge touches the task, a small group of qualified humans scores a representative sample using the same rubric the judge will eventually use. This is also where you should be building your golden dataset — the 50 or so examples that anchor every future comparison.
2. Calibrate the LLM judge against those labels. Run the candidate judge model on the same samples and measure agreement with the human labels — not just aggregate accuracy, but where and how it diverges. Knowing when to trust an LLM-as-judge means knowing its disagreement pattern, not just its overall score.
3. Spot-check continuously. Even a well-calibrated judge drifts — as the underlying model changes, as your product's outputs shift, as new failure modes emerge that the judge was never calibrated against. A recurring human sample, not a one-time calibration, is what keeps the judge honest.

This pattern mirrors, at a much smaller scale, what Anthropic described in its Constitutional AI research: a human-labeled seed set trains and grounds a process that AI feedback then scales — but the human seed doesn't disappear once the automated layer is running, it keeps anchoring it. The lesson generalizes past any one lab's method: automation scales a judgment humans have already validated; it doesn't replace the validation step.

How much agreement is "enough"?

Inter-rater reliability statistics give you a vocabulary for this instead of a gut feeling. Cohen's kappa, the classic measure of agreement between two raters correcting for chance agreement, is commonly interpreted using the Landis and Koch scale: values above roughly 0.6 are considered "substantial" agreement, and above 0.8 "almost perfect." Apply the same logic to judge-vs-human agreement — a judge sitting below "substantial" agreement with your own reviewers isn't ready to run unsupervised, no matter how good its aggregate score looks.

A Human-Review Protocol That Doesn't Fool Itself

A trustworthy human-review protocol needs three things: reviewers blind to which system or version produced each output, outputs presented in randomized order, and a rubric defined and shared before review begins — not invented while scoring, and not adjusted mid-review to fit what a reviewer is seeing. Skip any of the three and your human baseline is quietly contaminated by the same biases you were trying to avoid by using humans in the first place.

Blind review means the reviewer doesn't know whether they're looking at the new model or the old one, version A or version B, or the "improved" prompt versus the baseline. Expectation bias is real and well-documented in human-subjects research generally: a reviewer told "this is the new version" will tend to find it better, independent of actual quality.

Randomized order matters for a related reason. The Zheng et al. paper on judging LLM-as-a-judge (the research behind the widely used MT-Bench and Chatbot Arena evaluations) documented position bias — both human and LLM judges systematically favor whichever response they see first or second, regardless of content — alongside verbosity bias (longer answers scored as better independent of quality) and self-enhancement bias (a judge favoring outputs similar in style to its own). Randomizing presentation order is the cheapest fix for the first of these.

A rubric fixed in advance stops reviewers from unconsciously shifting their standard to match whatever they're currently looking at — the single most common way review consistency erodes over a long scoring session.

A minimal protocol that operationalizes all three:

• Write the rubric first, with 3-5 named criteria and a description of what a 1, 3, and 5 look like for each — before any reviewer sees a single output.
• Strip identifying metadata (model name, timestamp, prompt version) from anything a reviewer will see.
• Randomize presentation order per reviewer, and if comparing two systems, randomize which side is "A" and which is "B" independently for each item.
• Use at least two reviewers on a meaningful sample and calculate agreement (Cohen's kappa or a simpler percent-agreement figure) rather than trusting a single opinion.
• Log disagreements as data, not noise — a pair of reviewers who disagree on the same item repeatedly is telling you the rubric itself is ambiguous.
• Rotate reviewers across review cycles so no single person's idiosyncratic taste becomes the de facto standard.

Stanford's HELM (Holistic Evaluation of Language Models) project, one of the more comprehensive public efforts to standardize LLM benchmarking, has been explicit that several of the quality dimensions it cares about — including robustness and certain fairness measures — resist a single fully automatic metric and depend on structured human or hybrid assessment. That's a research-scale validation of the same principle that applies at product scale: some things you have to look at.

Building a Blended Eval Plan

A blended eval plan isn't automated evals plus human evals running in parallel as a formality — it's a deliberate allocation where every dimension in your rubric has an explicit owner: an automated check, a human reviewer, or both, with a stated cadence for each. Treating "human review" as a vague catch-all defeats the purpose; it needs the same rigor as any automated metric.

A practical way to build one:

1. List every quality dimension your feature needs to satisfy (correctness, tone, safety, format, latency-adjacent behavior, and anything task-specific).
2. For each, apply the decision rule from the first section: repeatable-and-defined goes automated; subjective, high-stakes, or novel stays human.
3. For anything automated, run the bootstrap pattern before trusting it — don't skip straight to an LLM judge on a dimension no human has scored yet.
4. For anything human, apply the blind/randomized/rubric protocol, and set a sampling rate you'll actually sustain (5-10% of production traffic is a common realistic starting point for ongoing spot-checks, heavier during a launch window).
5. Revisit the allocation whenever a new failure mode surfaces — that's usually a sign a "human" dimension has become well-enough understood to partially automate, or that an "automated" dimension needs a human dimension added back in.

Key Takeaways

• Automate the repeatable, not the subjective — format checks, factual lookups against ground truth, and anything with a single correct answer are the clearest wins for automation.
• Three categories stay human by default: subjective quality judgments, high-stakes decisions where a rare miss is costly, and novel failure modes no automated check was built to catch.
• Bootstrap, don't skip straight to a judge — label with humans first, calibrate an LLM judge against those labels, and keep spot-checking after launch, not just before it.
• Cohen's kappa above roughly 0.6 is a reasonable bar for "substantial" human-judge agreement before trusting a judge to run with lighter supervision.
• Blind review, randomized order, and a rubric fixed in advance are the minimum protocol to keep human review from reproducing the same biases (position, verbosity, self-enhancement) documented in LLM-judge research.
• A metric that keeps passing while users keep complaining is the clearest signal a dimension was automated before it should have been.
• Revisit the automation line regularly — today's human-only judgment is tomorrow's automated check, once enough labeled examples exist to calibrate against.