A model that scores 90% in aggregate can still be failing a quarter of its users if that number is an average hiding a 61% for one language or region. Fairness evals exist to catch exactly this: they score model outputs separately for each user segment, so a passing headline number can't hide a failing group underneath it.
Quick Answer: Fairness evals slice your eval set by group — language, region, or a lawful demographic proxy — and score each slice against the same pass/fail threshold you'd use for anyone. An aggregate pass sitting on top of a slice failure is a ship-blocking bug, not statistical noise.
What a Fairness Eval Actually Measures
A fairness eval is an ordinary model eval — a golden dataset, a scoring rubric, a pass/fail threshold — run once per user segment instead of once over the whole set. It exists because a single blended average always launders variance, letting a majority group's strong performance quietly absorb a minority group's weak one.
Most eval suites are built once, run once, and reported as one number: overall accuracy, overall pass rate, overall judge score. That single figure is exactly what a complete guide to AI evals walks you through building first — golden dataset, rubric, threshold — but nothing in that recipe says to stop at one number. A fairness eval reuses the same machinery and adds one field: a segment tag on every example.
Averages hide group-level failure for three predictable reasons:
- Sample-size imbalance. If 900 of your 1,000 test cases are English and 100 are Spanish, the English score dominates the mean almost by arithmetic default, no matter how badly the model does on Spanish.
- Correlated failure modes. Bugs rarely spread evenly. A tokenizer that mishandles accented characters, a prompt template written and tuned only against English examples, or a training set skewed toward one region will all concentrate their damage in the same slice, every time.
- Judges that share the blind spot. If you're using an LLM to grade responses, that judge model can carry the same language or cultural skew as the model it's grading, scoring a weak Spanish response as "good enough" because its own calibration is thinnest there too.
None of this shows up in a single blended score. It only shows up once you stop averaging and start comparing.
Choosing the Segments That Matter
The segments worth slicing on are the ones where users plausibly face a different product experience — not every column in your database. Defensible starting axes are input language, region or jurisdiction, device or modality, and account tenure; demographic proxies like age band, disability status, or gender belong in the evaluation layer only, and only with legal review, never as a live decisioning input.
Segments to start with
A practical first cut rarely needs more than four or five slices:
- Input language or dialect — the single highest-leverage cut for most global products, and the one buried inside the worked example below.
- Region or regulatory jurisdiction — especially where the same feature operates under different legal regimes (GDPR vs. CCPA, EU AI Act vs. no equivalent statute).
- Device or modality — voice input, mobile, and assistive technology often surface different failure modes than a desktop text box.
- Account tenure or experience level — new users phrase requests differently than power users, and a rubric tuned on power-user phrasing can quietly under-serve newcomers.
- Demographic proxies, where lawful and voluntarily disclosed — age band or disability status, added only after legal sign-off, and scored, never used as a routing input.
Why this is a JTBD problem before it's a stats problem
Two users in different segments are frequently trying to get different jobs done, in the language of the Jobs to Be Done framework. A Spanish-speaking small-business owner filing a billing dispute isn't a translated copy of an English-speaking user's request — it can be a structurally different job, with different urgency and different acceptable failure modes.
Mapping each segment's customer journey emotion curve usually shows the gap concentrated at one specific moment — the escalation step, the confirmation screen — rather than spread evenly across the whole interaction.
Legally, the US treats using a protected characteristic as a model input very differently from measuring outcomes across protected groups. Using race, sex, age, or a tight proxy for any of them as a live input to a lending, hiring, or housing decision is generally prohibited outright.
Measuring whether outcomes differ across those same groups — the entire point of a fairness eval — is not only legal in most contexts, regulators increasingly expect it. The four-fifths rule from the EEOC's Uniform Guidelines on Employee Selection Procedures is the classic version of this test: if a group's selection rate falls below 80% of the highest-scoring group's rate, that's a signal of adverse impact worth investigating, regardless of intent.
Disparate Performance vs. Disparate Treatment
Disparate treatment means a model or process explicitly uses a protected characteristic, or a close proxy for one, to produce a different outcome — an intent-based problem. Disparate performance (often called disparate impact once it produces an adverse outcome) means a group-blind model still produces unequal results across groups regardless of intent — an effects-based problem, and the one a fairness eval is built to catch.
| Dimension | Disparate Treatment | Disparate Performance / Impact |
|---|---|---|
| Definition | Model or rule explicitly differentiates by protected class or proxy | Outcomes differ across groups even though no protected attribute was used as input |
| Where it shows up | A hardcoded rule, a segment branch, a labeling instruction | Accuracy, error rate, or latency gaps in an otherwise group-blind model |
| Does intent matter? | Central to the claim — the differentiation is the violation | Irrelevant — the measured effect is what counts |
| Typical root cause | An explicit if/else on a protected attribute, biased labeling guidance | Training-data imbalance, language coverage gaps, judge-model blind spots |
| How it's caught | Code/prompt review, audit of decision logic | A segment-sliced eval comparing scores across groups |
| Fix | Remove the differentiating logic or input entirely | Rebalance data, add targeted examples, adjust or retrain per segment |
The distinction matters because the fixes are incompatible. You cannot "retrain your way out" of disparate treatment — the differentiating rule has to come out. You cannot "just remove a rule" to fix disparate performance — there usually isn't one to remove; the gap lives in the data and the model's learned behavior, not a single line of logic.
NIST's Special Publication 1270, Towards a Standard for Identifying and Managing Bias in Artificial Intelligence, groups root causes into systemic, statistical/computational, and human/cognitive bias — a useful checklist when a slice fails and you're trying to work out which category you're looking at.
The Gender Shades study by Joy Buolamwini and Timnit Gebru is the field's best-known illustration of disparate performance in the wild. Commercial facial-analysis systems showed accuracy gaps of over 30 percentage points between the best- and worst-performing subgroups, with no protected attribute anywhere in the input — the model never "knew" gender or skin type, and the gap showed up anyway.
Worked Example: A 90% Aggregate Hiding a 61% Slice
Here's the pattern in a concrete case: a support-ticket triage assistant that classifies incoming tickets into the correct queue. Evaluated in aggregate across 1,000 held-out tickets, it hits 90% routing accuracy against an 80% threshold — a clean pass, ready to ship by the numbers alone.
Slice the same eval set by the ticket's input language and the picture changes:
| Segment | Tickets (N) | Accuracy | Threshold | Result |
|---|---|---|---|---|
| Aggregate (all tickets) | 1,000 | 90% | 80% | Pass |
| English | 750 | 97% | 80% | Pass |
| French | 100 | 79% | 80% | Fail (borderline) |
| Spanish | 150 | 61% | 80% | Fail |
The aggregate passes because English tickets are three-quarters of the dataset and are scoring almost perfectly — that's enough to drag the blended number well above the bar even while Spanish tickets are failing badly and French is missing by a hair. Ship on the aggregate alone and you've shipped a triage assistant that misroutes nearly two in five Spanish-language tickets.
Getting to this table takes three concrete steps:
- Tag the golden dataset by segment before scoring, not after. If you're building your first golden dataset, add the segment label as a required field from example one — retrofitting tags onto an existing set is where most teams quietly give up on doing this at all.
- Set one threshold, and hold every segment to it. The guidance in choosing numeric thresholds for failable evals applies per slice exactly as it applies in aggregate — a segment gets its own row in the report, not its own bar to clear.
- Report pass/fail per segment alongside the aggregate, every run. A dashboard that only ever prints one number will never surface this table on its own; someone has to ask for the slice, every time, until it's automatic.
Building Segment-Level Evals Into Your Pipeline
Segment-level scoring is mechanically simple to add and easy to get quietly wrong, usually at the judge. The fix is validating your grader per segment with the same rigor you validate the model it's grading.
Watch the judge, not just the model
If part of your rubric relies on an LLM to grade free-text output, that grader needs its own segment check. The guidance on knowing when to trust an LLM-as-judge applies with extra force here: a judge trained and calibrated mostly on English examples can systematically over-score mediocre non-English responses, masking exactly the gap the eval exists to find. Spot-check judge-vs-human agreement separately within each segment, not only in aggregate, before trusting its scores to gate a release.
Structuring the eval plan itself
This is where the mechanics benefit from being planned, not improvised per release. Prodinja's Evals workspace lets you structure an eval plan so segment tags sit alongside the rubric and threshold from the start.
A plan built this way can score per-segment and surface a table like the one above next to the headline pass/fail, instead of requiring someone to remember a separate breakdown after the fact. It's a structuring aid for the eval plan you'd otherwise assemble by hand, not an AI that decides fairness for you.
A workable rollout sequence:
- Define segments before writing the rubric, so the threshold conversation happens once, for all groups, instead of being re-litigated per slice later.
- Size each slice deliberately. A segment with only 12 examples will swing wildly between runs — treat its result as directional, not gating, until the sample is large enough to trust.
- Version the segment definitions themselves. Regions get redrawn, language detection gets more granular, account-tenure buckets shift — track those changes the same way you'd track a rubric change.
- Re-run the full slice table on every model or prompt change, not just the aggregate — a prompt tweak that helps the average can easily be the exact change that widens a gap you already had.
What to Do When a Slice Fails
A failing slice is a diagnosis, not a verdict — the response depends on why the gap exists, not just that it exists. Work through root cause before reaching for a fix, because the wrong fix (retraining when the real problem is a hardcoded rule, or vice versa) burns a cycle without closing the gap.
- Rule out disparate treatment first. Audit the prompt, routing logic, and any post-processing for an explicit branch on a protected attribute or tight proxy. If you find one, remove it — no amount of retraining offsets an explicit discriminatory rule sitting upstream of the model.
- Check data representation. Open-source toolkits like IBM's
AI Fairness 360compute standard disparity metrics —demographic parity,equalized odds— that can help confirm whether the gap tracks training-data imbalance for that segment. - Check for a structural gap, not a volume gap. Sometimes the segment isn't underrepresented in count, but underrepresented in variety — a language slice with plenty of examples that all share one narrow phrasing pattern.
- Decide on a remediation path: targeted data augmentation, a segment-aware prompt variant, a temporary human-review gate for that slice, or, if none of those close the gap fast enough, an explicit scope disclosure that the feature isn't yet released for that segment.
- Re-run the full segment table, not just the failing row, after any fix — a change targeted at one slice can move another slice you weren't watching.
- Document the decision either way. "We shipped with a known 61%-vs-90% gap and here's our remediation timeline" is a defensible product decision. A silently ignored slice failure is not.
Key Takeaways
- An aggregate score is a weighted average, and averages hide exactly what a fairness eval is designed to expose — a strong majority-group score can fully offset a failing minority-group one.
- Define segments around where the product experience genuinely differs — language, region, device, tenure — not around every demographic column available in your data warehouse.
- Demographic proxies belong in evaluation, never in live decisioning, and only after legal review confirms the segment is lawful to measure and act on.
- Disparate treatment and disparate performance are different problems with different fixes — an explicit discriminatory rule has to be removed; a data-driven performance gap has to be closed through data, prompts, or retraining.
- Tag your golden dataset by segment from the first example, and set one threshold that every segment is held to, rather than grading each group on its own curve.
- Validate your LLM-as-judge per segment, separately from the model it grades — a judge with the same blind spot as the model will happily rubber-stamp a failing slice.
- A failing slice calls for a documented remediation decision, not silence — ship with disclosure and a timeline, or fix it before shipping, but don't let it disappear into an aggregate that still says "pass."
Frequently Asked Questions
What is a fairness eval in AI or machine learning?
A fairness eval is a standard model evaluation — golden dataset, scoring rubric, pass/fail threshold — run separately for each user segment instead of once across the full dataset. It exists specifically to catch cases where a blended average score conceals a group whose real performance falls below an acceptable bar.
Is it legal to test AI models by race, gender, or other demographic groups?
Measuring outcomes across demographic groups is generally legal, and in regulated domains like lending and employment, increasingly expected by regulators using tests like the EEOC's four-fifths rule. What's restricted is using those same characteristics, or tight proxies for them, as a live input to the decision itself — that's disparate treatment, a different and much narrower problem.
How is disparate impact different from disparate treatment?
Disparate treatment is an intent-based problem: a model or process explicitly differentiates by a protected characteristic. Disparate impact (or disparate performance) is an effects-based problem: outcomes differ across groups even though the model never saw a protected attribute as input. A fairness eval is built to catch the second kind — it can't tell you whether a rule was written with intent.
How big does a segment sample need to be before I trust its score?
There's no universal cutoff, but treat any segment under roughly 30-50 examples as directional rather than gating — small samples swing sharply between runs on chance alone. Grow the slice toward the same order of magnitude as your other segments before letting its result block or approve a release on its own.
What tools can help run fairness evals across segments?
Open-source toolkits such as IBM's AI Fairness 360 implement standard disparity metrics like demographic parity and equalized odds out of the box, and are a reasonable starting point if you're building this capability from scratch. Whatever tooling you use, the underlying requirement is the same: a segment-tagged golden dataset, a shared threshold, and a report that shows every slice, not just the average.