Inter-annotator agreement — how consistently your human labelers apply the same standard to the same example — sets the real ceiling on any model eval. If two expert reviewers only agree with each other 75% of the time, no model can be honestly scored as "95% correct" against that same ground truth. Disagreement isn't noise to average away; it's data about the task itself.
Quick answer: Measure agreement (
Cohen's kappaor a related statistic) before you trust any golden set as ground truth. Split the disagreements you find into "fixable guideline gaps" and "genuine ambiguity" — rewrite the rubric for the former, and cap your model's target score at the human agreement rate for the latter.
Why Human Disagreement Sets a Ceiling on Model Performance
A model cannot score better against ground truth than ground truth agrees with itself. If your labelers only agree with each other 80% of the time on a task, reporting a model at 95% "accuracy" against a single fixed answer key is a statistical fiction — you've just measured how well the model matches one annotator's mood on one day, not the task.
Most eval sets are treated as if they arrived pre-verified, a list of objectively correct answers a model either hits or misses. In practice, someone — often one overworked PM or a single contractor — wrote every label under time pressure, making judgment calls that a second equally qualified person might make differently. A golden dataset is only as trustworthy as the process that produced its labels, and that process is human.
This matters most for subjective or high-stakes tasks: content moderation severity, medical triage urgency, sentiment on sarcastic text, "is this support ticket urgent or merely high-priority." On these tasks, published inter-annotator agreement in NLP and ML research routinely lands in the 0.5-0.7 Cohen's kappa range even among trained experts — nowhere near the near-perfect agreement teams quietly assume.
Objective tasks (does this SQL query return the right rows, is this date formatted correctly) rarely show this problem; ambiguity concentrates in exactly the tasks where model behavior matters most.
Treating your eval set as infallible while it was never independently verified is the most common blind spot in eval design covered in a broader complete guide to AI evals. The fix starts before you write a single test case: measure whether your own experts agree with each other first.
Skipping this step doesn't just risk an inflated score — it risks an actively misleading one. A team that ships a model at "94% accuracy" against an unverified golden set, then watches real-world complaints roll in anyway, tends to blame the model or the prompt. Often the real defect was upstream: the eval never measured what it claimed to measure, because nobody checked whether the answer key agreed with itself.
Measuring Agreement: Cohen's Kappa and Its Cousins
Cohen's kappa, developed by psychologist Jacob Cohen in 1960, measures how much two raters agree beyond what chance alone would produce — a raw percent-agreement number is misleading because two careless labelers flipping a coin on a binary task will still "agree" roughly half the time. Kappa subtracts out that chance baseline, so a score near 0 means "no better than random" and a score near 1 means near-perfect real agreement.
You don't need to derive the formula to use it well. Conceptually, kappa asks: of the agreement we observed, how much of it is real signal versus luck? A tool like scikit-learn, R's irr package, or a spreadsheet formula computes it from a simple confusion matrix of rater A's labels against rater B's labels — the input you already have from any dual-annotation pass.
Different setups call for different variants of the same idea:
| Metric | Best for | Handles chance agreement? | Number of raters |
|---|---|---|---|
| Percent agreement | Quick sanity check only | No | 2+ |
Cohen's kappa | Two raters, categorical labels | Yes | Exactly 2 |
Fleiss' kappa | Three or more raters, categorical labels | Yes | 3+ |
Krippendorff's alpha | Any rater count, missing data, ordinal/interval labels | Yes | 2+ |
Statisticians Landis and Koch proposed a widely cited (if debated) rule-of-thumb scale in 1977 for interpreting a kappa score, still the default reference point in most applied ML writing:
| Kappa range | Interpretation |
|---|---|
| < 0.20 | Slight agreement — treat the label as unreliable |
| 0.21 – 0.40 | Fair agreement — guidelines likely need work |
| 0.41 – 0.60 | Moderate agreement — usable with caution |
| 0.61 – 0.80 | Substantial agreement — a defensible eval bar |
| 0.81 – 1.00 | Almost perfect agreement — the ideal, rare in practice |
Most production teams target the substantial-or-better band before freezing a golden set for high-stakes evals, and treat anything below moderate as a signal to stop and adjudicate rather than ship. Krippendorff's alpha, developed by communications scholar Klaus Krippendorff, is worth reaching for the moment you have more than two annotators or any missing ratings — a common real-world situation Cohen's kappa wasn't built to handle cleanly.
A worked illustration
Say you're building a golden set for a support-ticket triage classifier with four severity labels. You have two senior reviewers independently label the same 200 tickets before either sees the other's answers. Their confusion matrix shows they agree on 168 of 200 — 84% raw agreement, which sounds solid until you compute kappa and get 0.58, "moderate" on the Landis-and-Koch scale, because the label distribution is skewed enough that chance agreement alone would already explain a large share of that 84%.
That gap between raw agreement and kappa is the whole point of using kappa in the first place: 84% felt reassuring, but 0.58 tells you the two reviewers are only moderately more consistent than you'd expect from two people applying a rough guess. It's a prompt to adjudicate the 32 disagreements, not a passing grade.
Cohen also proposed a weighted kappa variant in 1968 for exactly this situation on ordinal labels: it penalizes a "low" vs. "critical" mismatch more heavily than a "low" vs. "medium" one, instead of treating every disagreement as equally wrong. If your severity labels have a natural order, weighted kappa is almost always the more honest statistic than the unweighted version.
Genuine Ambiguity vs. Bad Guidelines: Which One Are You Looking At?
Low agreement has two very different root causes, and confusing them wastes weeks: either your labeling guidelines are underspecified (fixable by rewriting them) or the task itself contains genuine, irreducible ambiguity (not fixable by writing — only by redefining or accepting it). Diagnose which one you have before you touch the rubric.
Researchers Lora Aroyo and Chris Welty, in their CrowdTruth work on crowdsourced annotation, argue that persistent disagreement is frequently a measurement of the task's real ambiguity rather than a defect in the annotators:
Disagreement is not noise to be eliminated — it is signal about where a task's ground truth is inherently plural, and averaging it away just hides the information.
Use these signals to tell the two apart:
| Signal | Points to bad guidelines | Points to genuine ambiguity |
|---|---|---|
| New vs. tenured annotators | New annotators disagree far more than tenured ones | Even your most experienced reviewers split |
| Disagreement pattern | Clustered around one specific edge case or label pair | Spread evenly across many unrelated examples |
| After a guideline rewrite | Agreement jumps sharply | Agreement barely moves |
| Rationale check | Reviewers cite a missing rule ("guide never said what to do here") | Reviewers cite different but equally valid criteria |
Genuine ambiguity shows up constantly once you go looking for it — it's the same phenomenon that makes a well-run Jobs to Be Done interview surface three different underlying "jobs" from three customers who bought the identical product for different reasons. The label undercounts the reality; forcing one answer erases information rather than resolving it.
The same logic shows up in mapping a customer journey's emotion curve: two researchers coding the same session recording will place the friction point at a different step unless the stage boundaries are defined as precisely as you're asking your annotators to define a label. Fuzzy construct, fuzzy agreement — in journey mapping and in eval labeling alike.
A practical tell worth adding to the table above: ask each disagreeing reviewer to write one sentence justifying their label before they see the other's answer. If both sentences cite the same rule and reach different conclusions, that's a guideline gap — the rule didn't cover this case cleanly. If the sentences cite genuinely different, defensible criteria ("I weighted the customer's tone," "I weighted the stated deadline"), that's ambiguity baked into the construct itself, and no amount of guideline polishing will fully close it.
The Adjudication Playbook: From Disagreement to a Defensible Golden Set
Adjudication is the structured process of turning raw disagreement into a golden set you can actually stand behind — not by picking a winner arbitrarily, but by making every resolved case traceable to a documented reason. Skipping straight to "senior person breaks ties" without logging why quietly reintroduces the same ambiguity one label at a time.
- Compute agreement per label, not just overall. An aggregate kappa of 0.65 can hide one category sitting at 0.30 — find it before it contaminates your whole eval.
- Pull every disagreement into a review queue. Don't just keep whichever label a majority chose; keep the actual disputed cases as a working set.
- Adjudicate with a pre-agreed rule — majority vote across three-plus raters, a senior reviewer as tie-breaker, or a synchronous discussion to consensus. Decide the rule before you see the cases, not after.
- Log the "why" for every adjudicated case as a short addendum to your labeling guide — a future annotator, or a future you, needs the reasoning, not just the resolved label.
- Revise the guideline, then re-annotate a fresh sample — never the same examples you just adjudicated, which would flatter your new rules.
- Re-measure agreement and freeze the golden set only once it stabilizes at or above your target band from the kappa scale above.
This is also the point to set explicit numeric pass/fail thresholds for the model, rather than an open-ended "looks about right" bar — a threshold defined against an unadjudicated, disagreement-riddled golden set is a threshold built on sand.
Fix the Task Definition or Accept a Lower Bar
Once you've diagnosed the source of disagreement, you have exactly two honest paths forward — there's no third option where you simply demand more agreement through willpower. Either the task definition needs to change, or your target score needs to reflect the ceiling that genuinely exists.
If it's bad guidelines: rewrite the rubric with concrete examples of borderline cases, not just abstract rules. Add a worked example for every edge case that showed up in adjudication. Retrain annotators against the new version and re-measure — agreement should visibly improve within one or two revision cycles, or you've actually found genuine ambiguity wearing a guideline problem's clothes.
If it's genuine ambiguity, you have three legitimate options, and they aren't mutually exclusive:
- Split the label. A category that experts keep splitting in different directions ("urgent" vs. "high priority") may really be two categories that need separate definitions, not one ambiguous one.
- Allow soft or multi-label ground truth. Instead of one answer per example, store the distribution of valid labels and score partial credit — appropriate for genuinely contested cases, especially ones an LLM-as-judge will also be asked to grade, since a judge model inherits the same ceiling a human panel does.
- Cap your target score at the empirical agreement ceiling. If your best human reviewers agree with each other 78% of the time, reporting a model target of 95% isn't rigor — it's a number nobody, human or machine, could actually hit.
The uncomfortable version of this advice: a lower, honestly-derived attainable bar is more useful to a team than an aspirational one nobody — including your own experts — could reach.
Communicating the ceiling upward
Stakeholders who haven't sat through an adjudication session tend to hear "we're targeting 78%, not 95%" as the team lowering its ambitions. Reframe it the other way: 78% against a task humans can't fully agree on is a stronger, more defensible number than 95% against an answer key nobody actually verified. Bring the kappa score itself into that conversation — a concrete "our own experts agree 0.62 of the time" is a far more persuasive ceiling argument than an abstract appeal to nuance.
Building Disagreement-Awareness Into Your Eval Workflow
Agreement measurement isn't a one-time audit before launch — it's a standing step that recurs every time you add a label, onboard a new annotator, or expand into a new example domain. Treat it the same way you'd treat any other regression check: run it again whenever the inputs to your golden set change, not just once at the start.
This is exactly the discipline Prodinja's eval-planning flow is built around: it's designed to treat whatever golden set you feed it as the authoritative source of truth for scoring a model — which is precisely why the adjudication work in this article has to happen before that set gets loaded, not after. The prototype walks you through defining a golden set, its categories, and its pass/fail thresholds.
It doesn't manufacture agreement your own annotators haven't actually reached, and it won't paper over a disagreement you haven't resolved. The rigor still has to come from you and your reviewers — the tool's job is to make sure that rigor, once established, actually gets enforced consistently instead of drifting eval to eval.
In practice, a lightweight cadence works: re-run agreement measurement whenever you add a label, swap in a new annotator, or expand the golden set with a new batch of examples — not on a fixed calendar, but tied to the events that actually change what's being measured. A golden set that hasn't been re-checked since the labels it was built on last changed isn't stale in an abstract sense; it's actively describing a task that no longer exists.
Key Takeaways
- Inter-annotator agreement is the ceiling, not a footnote — no model can be scored as more "correct" than your human labelers are correct with each other.
Cohen's kappameasures agreement beyond chance, not raw percent match; a Landis-and-Koch-style scale (roughly 0.6+ as "substantial") is a reasonable bar for a production-grade golden set.- Use
Fleiss' kappaorKrippendorff's alphaonce you have three-plus annotators or any missing ratings —Cohen's kappaalone assumes exactly two complete raters. - Diagnose before you fix: clustered disagreement that vanishes after a guideline rewrite is a guideline problem; even, persistent disagreement among your best experts is genuine ambiguity.
- Adjudicate with a documented rule, not an ad hoc tie-break — log the reasoning behind every resolved case so it survives beyond the person who made the call.
- When ambiguity is real, split the label, allow soft/multi-label ground truth, or cap your target score at the measured human agreement rate — don't chase 100% against a task humans themselves can't agree on.
- Re-measure agreement any time the task, labels, or annotator pool changes — it's a recurring check, not a one-off certification.
Frequently Asked Questions
What counts as a good inter-annotator agreement score for an AI eval?
A Cohen's kappa of 0.61-0.80 ("substantial agreement" on the Landis and Koch scale) is a reasonable bar for a production-grade golden set on a subjective task. Below 0.4, treat the label as unreliable and adjudicate before trusting any model score derived from it.
How do you calculate Cohen's kappa without a statistics background?
Build a confusion matrix of rater A's labels against rater B's labels on the same examples, then run it through a free calculator, a spreadsheet formula, or a one-line call in scikit-learn (cohen_kappa_score). You don't need to compute the underlying chance-agreement math by hand to use the output correctly.
What if annotators still disagree even after I rewrite the guidelines?
Persistent disagreement after a guideline revision — especially among your most experienced reviewers — is a strong sign you've found genuine ambiguity, not a documentation gap. At that point, split the ambiguous label into two, allow multi-label ground truth, or set your model's target score at the measured human agreement rate instead of 100%.
Is 100% agreement between labelers realistic for a golden set?
No, and expecting it is itself a design flaw. Even trained experts on well-defined objective tasks rarely hit perfect agreement, and subjective tasks routinely land in the moderate-to-substantial kappa range at best — build your eval's pass bar around the ceiling you actually measure, not an idealized one.
Should I throw out examples where annotators disagree?
Not automatically — a disagreement is information, not noise to discard. Adjudicate it first with a documented rule; only exclude an example if adjudication concludes the case itself is malformed (unclear input, a genuinely broken prompt) rather than simply hard to label.