Ground truth is whatever your labels say it is — which means the taxonomy, the annotator instructions, and the disagreement-resolution process are product decisions, not back-office data work. An AI PM who delegates labeling strategy to an ops team or a vendor is quietly outsourcing the definition of "correct" for their own product.
Quick Answer: Ground truth labeling strategy is a PM responsibility because the label taxonomy defines what your model is optimized to predict, annotator agreement tells you whether that definition is even coherent, and the loop between edge cases and guideline updates is the actual mechanism by which your product's quality improves over time.
Why Ground Truth Is a Product Decision
Ground truth is a product decision because every label is a miniature product spec: it encodes what counts as correct, what counts as an edge case, and whose judgment gets to decide. Get the taxonomy wrong and no amount of model architecture, prompt engineering, or compute fixes the downstream quality ceiling.
Most teams treat labeling as a procurement problem — find annotators, write a short instruction sheet, ship a spreadsheet, wait for results. That framing hides the actual failure mode. Ambiguous labels produce ambiguous products, because the model learns whatever inconsistency is baked into its training signal, and that inconsistency then shows up as unpredictable behavior in production.
A 2021 Google Research paper by Sambasivan et al., "Everyone Wants to Do the Model Work, Not the Data Work," studied AI practitioners across industries and found that data quality issues — not model architecture — were a recurring, underappreciated cause of downstream failures the teams involved hadn't anticipated. The paper's core argument is directional but hard to ignore: teams that treat data work as lower-status, delegable busywork tend to pay for it later, in production incidents that trace back to a labeling decision nobody examined closely.
Andrew Ng's "data-centric AI" push, publicized through DeepLearning.AI's 2021 data-centric competition, makes a parallel case from the practitioner side: for a fixed model, improving label consistency and coverage often moves quality more than swapping architectures. Neither source hands you a precise multiplier — but both point at the same asymmetry: the labeling decision is leveraged, and PMs who ignore it are ignoring their highest-leverage lever.
This is also where the line between an AI PM and a software PM gets concrete. A software PM ships a spec and reviews the build against it. An AI PM ships a label taxonomy and reviews whether reality — messy, ambiguous, contested reality — fits inside the categories they defined. If you want the fuller role definition, the complete guide to the AI PM role covers where this responsibility sits alongside eval design and model selection.
The three decisions hiding inside every labeling project
- Taxonomy — what categories exist, and how granular they are.
- Guidelines — the written rules annotators use to resolve ambiguous cases.
- Escalation — who decides when the guidelines don't cover a case, and how that decision gets fed back.
Skip any one of the three and the other two degrade. A perfect taxonomy with no escalation path calcifies the moment reality produces a case it didn't anticipate.
Designing a Label Taxonomy That Survives Contact with Reality
A good label taxonomy is designed for the disagreements it will provoke, not just the categories it will contain — because every taxonomy eventually meets an example that doesn't fit cleanly, and how it handles that moment determines whether your ground truth stays trustworthy or quietly rots.
Start from the decision the label feeds, not from an intuitive ontology of the domain. If the label ultimately routes a support ticket, trains a classifier, or scores an eval, work backward from that downstream action to the minimum set of distinctions that actually change it. A taxonomy with categories nobody's downstream logic ever uses is complexity with no payoff.
Flat, hierarchical, or multi-label — pick deliberately
| Taxonomy shape | Best for | Main failure mode |
|---|---|---|
| Flat (mutually exclusive) | Simple routing, clear-cut classification | Forces borderline cases into a wrong bucket |
| Hierarchical (parent → child) | Domains with natural nesting (intent → sub-intent) | Annotators disagree on depth before they disagree on category |
| Multi-label (tags, not buckets) | Cases with genuinely overlapping properties | Downstream logic must handle combinations, which is often skipped |
Most teams default to flat because it's easiest to build tooling for, then discover their domain is multi-label in disguise — a support ticket that's simultaneously a billing question and a bug report doesn't disappear just because your schema forces a single tag.
Practical taxonomy rules that hold up:
- Cap depth at three levels. Beyond that, annotators spend more time navigating the tree than judging the example.
- Write a one-sentence test for every category — if you can't state the deciding question in one sentence, the category is under-specified.
- Include an explicit "unclear/needs review" label from day one. Forcing every case into a confident bucket manufactures false precision; an honest "unclear" bucket is signal, not failure.
- Version the taxonomy. Treat every change like a schema migration — note what changed, when, and which examples were re-labeled under the new definition.
This is close cousin work to the discovery methods PMs already use for user needs — a taxonomy built without grounding in how the jobs to be done actually surface in the data will encode the categories you assumed mattered, not the ones users' real requests fall into. It's worth running the same rigor across a customer journey mapping exercise before finalizing category boundaries — the moments where user intent is genuinely ambiguous in the journey are exactly where your taxonomy will be tested hardest.
Inter-Annotator Agreement: Proving Your Labels Mean What You Think
Inter-annotator agreement (IAA) is the measurement that tells you whether your label definitions are precise enough for two independent people to apply them the same way — and if it's low, no amount of model tuning downstream will fix what's actually a definitional problem.
The naive approach — checking raw percent agreement — overstates reliability because it doesn't correct for chance. If 90% of your examples fall into one dominant category, two annotators guessing randomly would still agree 80%+ of the time. That's why the field uses chance-corrected metrics instead.
The metrics PMs should actually know
| Metric | Origin | Handles | Typical read on the number |
|---|---|---|---|
| Raw percent agreement | — | Simple sanity check only | Misleading with skewed class distributions |
Cohen's kappa | Jacob Cohen, 1960 | Two annotators, nominal categories | Corrects for chance agreement |
Fleiss' kappa | Joseph Fleiss, 1971 | Three or more annotators | Extension of Cohen's kappa to multiple raters |
Krippendorff's alpha | Klaus Krippendorff | Multiple annotators, missing data, ordinal/interval scales | Most flexible; standard in content analysis research |
A widely cited interpretation scale from Landis & Koch (1977) treats kappa values above roughly 0.80 as "almost perfect," 0.61–0.80 as "substantial," 0.41–0.60 as "moderate," and below 0.40 as agreement too weak to trust as ground truth. Those bands are a rough field convention, not a law of nature — but a taxonomy producing kappa in the 0.3s is telling you something concrete: your category boundaries are underspecified, not that your annotators are careless.
A worked example. Say you're labeling support tickets as "bug," "how-to question," or "feature request." Two annotators independently label 200 tickets. Raw agreement comes back at 78% — reassuring on its face. But the "how-to question" category dominates the dataset at 70% of tickets, so a coin-flip annotator would already land near 70% agreement by chance alone. Once you compute Cohen's kappa correcting for that base rate, it drops to 0.35 — "fair" at best on the Landis & Koch scale. That gap between raw agreement and kappa is the tell: your annotators aren't sloppy, your "bug" vs. "how-to question" boundary is.
The fix in that scenario is never "retrain the annotators harder." It's almost always a taxonomy or guideline problem: pull the disagreement cases, read them together, and find the actual boundary condition annotators are silently resolving in different directions.
Low agreement is a taxonomy signal wearing a data-quality costume. Treat every kappa below your target threshold as a prompt to re-open the guideline, not a prompt to fire the annotators.
Coordinating this measurement loop is itself a cross-functional skill — it's the same muscle covered in guidance on working with data scientists on experiments versus features, where the PM's job is translating a statistical signal (low kappa) into a product action (a guideline rewrite) that a purely technical read might miss.
The Feedback Loop Between Edge Cases and Guidelines
The feedback loop between edge cases and guidelines is what keeps a labeling taxonomy alive instead of ossified — every ambiguous case an annotator escalates is a live signal that your written guideline didn't anticipate reality, and routing it back into the guideline is how the taxonomy earns its "living" description.
Most guideline documents are written once, reviewed at kickoff, and never revisited — treated like a one-time spec instead of a spec under continuous pressure from new data. That's backwards. The guideline is the interface between your product's definition of correctness and the messy examples that test it, and interfaces need versioning discipline.
What breaks without a feedback loop
- Annotators silently develop their own private conventions for unresolved cases, so labels drift apart over time even though the written guideline never changed.
- The same edge case gets escalated repeatedly by different annotators because nobody closes the loop by updating the source document.
- Model errors downstream get diagnosed as model problems when they're actually inherited label inconsistency from an unresolved edge case months earlier.
- Nobody can explain, months later, why a given label was applied — because the reasoning lived in a Slack thread instead of the guideline.
A working loop, in practice
- Annotator hits an ambiguous case and flags it rather than guessing.
- Flagged cases get triaged on a fixed cadence (weekly is a reasonable default) by whoever owns the taxonomy.
- Each triage session produces one of three outcomes: clarify the existing rule, add a new sub-category, or explicitly declare it out of scope — all three are valid, but silence is not.
- The guideline document gets a dated changelog entry, and any prior labels affected by the change get flagged for re-review, not silently left inconsistent.
- The resolved example gets added to the guideline as a worked illustration, so the next annotator who hits something similar has a precedent to reference.
This is exactly the discipline behind eval-driven roadmaps — a roadmap built around quality gates is only as trustworthy as the ground truth those evals are scored against, and an eval suite scored against a taxonomy that's silently drifted will report quality that no longer means what it used to.
A Mini Playbook: Run Your Guideline Like a Living Spec
Running a labeling guideline like a living spec means giving it the same version control, changelog discipline, and review cadence you'd give a product requirements document — because functionally, that's what it is: a spec for what "correct" means, under continuous revision as new evidence arrives.
Step-by-step
- Draft v1 from a seed sample, not a hypothesis. Pull 50-100 real examples before writing a single rule; a taxonomy designed from imagined cases misses the distribution your annotators will actually face.
- Pilot with two annotators on the same 100 examples before scaling. Compute
Cohen's kappaon the pilot before hiring a larger labeling team — a taxonomy problem is far cheaper to fix with two annotators than with twenty. - Set an explicit agreement threshold before you see the number. Deciding your bar is 0.7 kappa after you've already seen 0.5 invites rationalizing a weak result; commit to the threshold first.
- Build the escalation channel before labeling starts at scale, not after the first complaint — a lightweight flag-and-triage flow, even a shared spreadsheet, works fine at first.
- Schedule recurring guideline reviews on the calendar, not "as needed" — ambiguity doesn't announce itself loudly enough to compete with a launch deadline unless it has a standing slot.
- Version every guideline change and re-run agreement checks after material revisions — a guideline update that isn't validated against a fresh agreement check is a guess, not a fix.
- Retire categories that never get used, and split categories that absorb a disproportionate share of disagreement — both are taxonomy-shape signals, not annotator-performance signals.
A quick self-check table
| Question | Green flag | Red flag |
|---|---|---|
| Can you state each category's boundary in one sentence? | Yes, cleanly | Requires a paragraph and "it depends" |
| Do escalations get resolved and documented? | Weekly triage, changelog kept | Flagged cases pile up unresolved |
| Has agreement been measured recently? | Within the last guideline revision | Measured once, at kickoff, and never since |
| Is there an "unclear" category? | Yes, tracked and reviewed | Every case forced into a confident bucket |
Keeping Ground Truth Honest Over Time
Keeping ground truth honest over time means treating the moment an edge case surfaces — not the quarterly guideline review — as the real unit of work, because insight decays fast: the annotator who hit the ambiguous case remembers the nuance today and will not reconstruct it accurately from a stale ticket next month.
Prodinja's Journals are built around that decay problem directly. As you're reviewing labeling output or sitting in an annotator sync, Journals let you capture a labeling-guideline Assumption ("we're treating partial refunds as billing, not disputes") or an edge-case Friction ("annotators keep splitting on ambiguous multi-intent tickets") the moment it surfaces, rather than trying to reconstruct it from memory at the next scheduled review. Over time those entries become the raw material for the guideline changelog itself — a running, honest record of where your ground-truth definition has been tested and where it's had to bend.
That's the whole point of running a labeling guideline like a living spec: the document is only as trustworthy as the discipline behind capturing what challenged it, as it happens.
Key Takeaways
- Ground truth is a product decision, not a data-ops task — the label taxonomy defines what your model is optimized toward, and PMs who delegate it are delegating their product's definition of correctness.
- Design the taxonomy for its disagreements, not just its categories — cap hierarchy depth, write a one-sentence boundary test per category, and always include an explicit "unclear" bucket.
- Measure agreement with a chance-corrected metric —
Cohen's kappa,Fleiss' kappa, orKrippendorff's alpha— because raw percent agreement overstates reliability on skewed distributions. - Low inter-annotator agreement is almost always a taxonomy signal, not an annotator-competence signal — treat it as a prompt to revisit the guideline, not a prompt to retrain the labelers.
- Build the edge-case-to-guideline feedback loop before labeling starts at scale — a fixed escalation and triage cadence keeps the taxonomy from silently drifting.
- Version the guideline like a spec — dated changelog entries, re-validated agreement after material revisions, and worked examples for future annotators to reference.
- Capture ambiguity where it surfaces, not where it's remembered — the nuance in an edge case decays fast, which is exactly the gap a lightweight, real-time capture habit is meant to close.
Frequently Asked Questions
What is inter-annotator agreement and why does it matter for AI PMs?
Inter-annotator agreement measures how consistently independent labelers apply the same category to the same example, corrected for the agreement you'd expect by chance alone. For an AI PM, it's the earliest, cheapest signal that a label taxonomy is too ambiguous to trust as ground truth — before that ambiguity gets baked into a trained model.
How do you fix low inter-annotator agreement?
Low agreement is fixed by revisiting the taxonomy and guideline, not by retraining annotators. Pull the specific disagreement cases, identify the boundary condition annotators are resolving differently, clarify or split the category, and re-measure agreement on a fresh sample after the change.
What's a good inter-annotator agreement score?
Under the commonly cited Landis & Koch scale, a Cohen's kappa above roughly 0.80 reads as "almost perfect" and 0.61-0.80 as "substantial," while anything below about 0.40 signals a taxonomy too ambiguous to serve as reliable ground truth. Treat these as a directional field convention to calibrate against your own risk tolerance, not an absolute pass/fail line.
Who should own labeling guidelines — the PM or the data science team?
The PM should own the taxonomy and guideline content because they encode product-level definitions of correctness, while data scientists typically own the statistical tooling used to measure agreement and surface disagreement patterns. Ownership works best as a shared loop: PMs decide what a label means, and the measurement tells both sides when that definition needs revisiting.
How is data labeling different for AI products versus traditional software features?
Traditional software features are specified once and verified against a fixed spec, while AI products are trained on labels that function as an evolving, empirically-tested definition of correctness. That difference is central to what distinguishes an AI PM's day-to-day from a traditional software PM's, particularly around ongoing quality ownership rather than one-time requirements sign-off.