Designing a Rubric a Machine Can Score Consistently

A rubric scores consistently when every criterion names an observable behavior, not a personality trait. Replace adjectives like "helpful" or "clear" with a checklist of concrete conditions a grader can literally check for. Then validate the rubric by having two graders — human or LLM — score the same output; if scores diverge, the rubric is broken.

Quick answer: Adjectives force every grader to invent their own definition. Write criteria as observable behaviors with binary or anchored 1-5 scales, then run the two-grader test — same output, same rubric, two independent graders — before you trust a single score it produces.

Why "Helpful" Is Not a Rubric Criterion

"Helpful" fails as a rubric criterion because it's a verdict, not a measurement — it names the conclusion you want the output to earn, not the evidence a grader should look for. Two careful graders can both believe an answer is helpful and still land on different scores, because each is quietly applying a private definition of the word.

This is the most common failure mode in home-grown eval rubrics, and it gets worse, not better, once an LLM is doing the grading. A language model asked to rate "helpfulness" will confidently produce a number — LLMs rarely refuse to answer — but that confidence is manufactured, not earned. Ask the same judge model to re-grade the same output a day later with an identical prompt and you'll often get a different score, because "helpful" gave it nothing stable to anchor to.

You can see the fix already deployed at scale. Google's publicly available Search Quality Rater Guidelines don't ask thousands of human raters to judge whether a page is "trustworthy" — they define explicit, checkable signals (author expertise, citations, whether the content matches the query's intent) and have raters check for each one individually.

Stanford's HELM benchmark (Percy Liang and the Stanford CRFM team) does the same thing for language models. Instead of one "quality" score, it decomposes evaluation into separately-named, separately-measured dimensions — accuracy, calibration, robustness, fairness — so a low result always points at a specific dimension, not a vague overall impression.

If you're building an evaluation practice from the ground up rather than patching one rubric, our complete guide to AI evals covers where rubric design fits into the full lifecycle — golden datasets, thresholds, and judge selection included.

The adjectives worth watching for are the ones that sound like quality but describe no specific text:

• Helpful, clear, thorough, professional, engaging, accurate-sounding, appropriate, on-brand.

None of these are wrong to care about. They're wrong to leave un-translated in a rubric a second person — or a model — has to apply without asking you what you meant.

From Adjectives to Observable Behaviors: The Checklist Method

The fastest fix is a three-step translation: name the adjective, list every concrete thing it's silently standing in for, then rewrite each one as a yes/no clause a grader can point to in the actual text. Do this once per quality dimension, and the adjective can retire from the rubric entirely.

1. Write the adjective down. Start with "helpful," "clear," or whatever the current criterion literally says.
2. Brainstorm the tells. Recall the last five outputs you personally called "helpful" versus "unhelpful." What specifically differed between them?
3. Turn each tell into a checkable clause. "Contains a next step" is checkable. "Feels useful" is not.
4. Keep only clauses a stranger could verify from the text alone, with no access to your intent.

Notice that none of these translations reference the grader's feelings — every clause resolves to something present or absent in the text itself. That's the actual test for whether a criterion is finished: could two people who've never spoken check it and get the same yes-or-no answer?

Anchoring each behavior to the job the output was hired to do also keeps this checklist from ballooning into an unreviewable list of fifty rules. That's a technique borrowed directly from jobs-to-be-done thinking: define success by the outcome the user is actually trying to achieve, and only the behaviors that serve that outcome earn a line in the rubric.

Choosing Your Scale: Binary vs. 1-5

Use a PASS/FAIL scale for any criterion where partial credit doesn't make sense — safety, factual accuracy, format compliance. Reserve an anchored 1-5 scale for criteria that are genuinely a spectrum, like tone or completeness. Mixing the two purposes on one scale is the most common cause of inconsistent scores in home-grown rubrics.

A binary criterion asks a single yes-or-no question: did the response contain PII, or didn't it? There's no meaningful "3 out of 5" for that question, and forcing one onto a scale just adds noise a grader has to invent an answer for. A spectrum criterion, by contrast, describes a real range of acceptable-to-ideal outcomes, where a 1-5 scale genuinely carries information a binary couldn't.

Gating criteria deserve their own separate pass, and turning them into hard failure conditions rather than points lost from an average is worth doing deliberately — our piece on numeric thresholds for failable evals goes deeper on where to draw that line and why averaging a safety criterion into a composite score tends to hide failures instead of surfacing them.

It's also worth deciding scales per context, not once for the whole product. A rubric written for an onboarding message and one written for a churn-save message shouldn't necessarily share anchors — the emotional stakes differ by where the interaction sits on the customer journey, and an anchor that reads as a "5" during onboarding might only earn a "3" when a customer is already frustrated.

Writing Anchor Descriptions That Remove Guesswork

An anchor tells a grader exactly what a 1, a 3, and a 5 look like in concrete, self-contained terms — not relative ones. "Somewhat better than a 2" is not an anchor; it's a shrug that only means something if the grader also happens to have a 2 sitting in front of them.

• Never use comparative language ("better," "more," "less") inside an anchor description — comparatives require a second output to compare against, which the grader may not have.
• Anchor at minimum both ends and the midpoint — write a concrete sentence for 1, 3, and 5 before you bother writing anything for 2 or 4.
• Reuse identical anchor wording across every example in the golden dataset you use to pressure-test the rubric, so the scale means the same thing from example to example — see our guide on how to build your first golden dataset if you haven't assembled one yet.
• If you can't write a concrete sentence for a scale point, cut it. A scale with points nobody can describe is a scale nobody can use consistently.

Before and After: A Full Rubric Rewrite

Here's the fix applied end to end, on a criterion you'll recognize from plenty of support-response evals: "Score 1-5: how helpful is this response?" That single line carries three separate problems, and none of them get fixed by swapping in a longer adjective.

The before version gives no definition of "helpful," offers no anchors so the same number can mean different things to different graders, and quietly bundles accuracy, tone, and completeness into one digit — so a low score never tells you which of the three actually failed.

The after version splits that one adjective into four separately-gradeable criteria, three binary and one anchored:

A low score on any binary line now points at exactly what broke — the response skipped a fact, gave no next step, or re-asked for something already provided. The one remaining 1-5 criterion carries only the dimension that's genuinely a spectrum, tone, so its average across a batch of outputs actually means something.

This is also why a single "helpful" score is so hard to act on: a 3 could mean "accurate but cold," or it could mean "warm but wrong." Splitting the criterion doesn't just improve grader agreement — it makes the score legible enough to actually fix the thing it's flagging.

The Two-Grader Test: How to Know Your Rubric Actually Works

The two-grader test is simple: take five to ten outputs, have two independent graders score each one against the same rubric with no conferring, then compare. If they land on the same score — or an adjacent one, on a 1-5 scale — more than roughly 80% of the time, the rubric is specific enough to trust.

If not, every disagreement tells you exactly which clause needs a sharper anchor — the gap between two graders is diagnostic information about the rubric, not noise to average away.

A rubric only one grader can apply consistently isn't a rubric. It's a first draft.

This isn't a new idea invented for LLM evals — it's inter-rater reliability, a decades-old concept from psychometrics. Cohen's Kappa, the statistic Jacob Cohen published in 1960 for measuring agreement between two raters beyond what chance alone would produce, is still the standard tool for it. Landis and Koch's widely-cited 1977 interpretation scale treats a kappa above roughly 0.8 as "almost perfect" agreement — a useful bar to aim a rubric at, even if you never compute the statistic formally and just eyeball percent agreement instead.

Run it as a repeatable loop:

1. Pull 5-10 outputs from your existing sample, ideally ones that already span easy cases and known edge cases.
2. Score independently. Grader A scores every output against the rubric; grader B does the same with zero visibility into A's scores.
3. Compare, criterion by criterion — not just the final composite — since one bad clause can hide inside an otherwise-fine average.
4. Read every disagreement. For each mismatch, find the specific rubric clause each grader interpreted differently.
5. Rewrite that clause, sharpen the anchor or split the criterion, and re-run the test on a fresh sample.

Whether the second grader is human or an LLM changes what kind of disagreement you'll find, but not whether you need to run the test. Our piece on when to trust an LLM as judge covers the additional checks worth adding once a model, not a second person, is doing half the grading.

OpenAI Evals, the open-source framework OpenAI publishes for exactly this kind of testing, is built around the same assumption: a rubric or grading function is only as trustworthy as the agreement you've actually measured, not the agreement you assumed it would produce.

Building the Discipline Into Your Workflow

The habit that makes rubric quality durable is treating "write an explicit rubric" as a mandatory step for every quality dimension you track, not an optional nicety added after an eval already exists. Teams that skip it end up debugging grader disagreements months later, with no record of what either grader originally meant by the score they gave.

In practice that means a few small process rules: no quality dimension ships without a written rubric attached; rubrics get reviewed the same way code does, by someone other than the author; and any time an anchor gets edited, the two-grader test runs again before the new wording is trusted.

Key Takeaways

• Adjectives are conclusions, not criteria — swap "helpful," "clear," and "thorough" for a checklist of behaviors a grader can point to in the text.
• Binary scales are for gating, 1-5 scales are for spectrums — don't force a yes/no question onto a five-point scale, or squeeze a real range into pass/fail.
• Anchor every scale point in concrete, non-comparative language — "somewhat better than a 2" is not an anchor, it's a shrug.
• One rubric, one job — four single-purpose criteria beat one criterion quietly bundling four different qualities into a single number.
• The two-grader test is the only real validation — run the same rubric past two independent graders on the same outputs before trusting it at scale.
• Disagreement is rubric feedback, not grader error — read every mismatch and rewrite the clause that caused it, then retest.
• Rubric design is a workflow habit, not a one-time task — build it into how every quality dimension gets defined, not just the first one you happen to write.