Reference-Free vs Reference-Based: Do You Even Have a Right Answer?

Some tasks have one right answer you can check a model's output against; many don't. Extraction, classification, and structured generation have a ground truth to match. Summarization, writing, and open-ended chat don't — there is no single correct summary, just better and worse ones. That distinction should decide your entire eval methodology before you pick a single metric.

Quick answer: If the task has a canonical correct output — extraction, classification, routing, structured data — use reference-based metrics like exact match, F1, or semantic similarity. If it doesn't — writing, summarization, open-ended chat — reference-based scoring will mislead you. Use a rubric or an LLM-as-judge instead.

Does This Task Have a Right Answer? Start Here

Ask one question before building any eval: could two competent humans produce meaningfully different outputs and both be right? If no — there's a single correct extraction, label, or value — you're in reference-based territory. If yes, you're evaluating quality, not correctness, and need a different toolkit entirely.

This isn't a minor technical detail. It's the fork that determines which metric you compute, whether you need a golden dataset or a rubric, and whether a human or an LLM judge ever needs to look at the output at all. Get the fork wrong and every metric downstream measures the wrong thing.

Run a quick diagnostic on your own task list:

1. Is there a mechanical check? A string match, schema validator, or execution test that can confirm correctness without human judgment.
2. Would two subject-matter experts agree independently? If they'd converge on the same output, a reference exists even if you haven't written it down yet.
3. Does "good" depend on audience, tone, or context? If the right answer changes depending on who's reading it, you're not measuring correctness — you're measuring fit.

If your answers lean toward "yes, a mechanical check exists," you're reference-based. If they lean toward "it depends who's asking," you're reference-free. Most teams building their first evaluation program benefit from mapping this out task by task before touching a single metric — our complete guide to AI evals covers the full planning sequence this decision sits inside of.

Reference-Based Evaluation: Exact Match, F1, and Semantic Similarity

When a task has one correct answer, score it against that answer directly. Exact match works for atomic outputs like labels or extracted values. F1 handles partial credit for multi-field extraction. Semantic similarity — embedding distance or entailment — catches correct answers phrased differently than your reference.

Each of these solves a slightly different failure mode, so pick based on what "close but wrong" looks like for your task:

• Exact match — for categories, boolean flags, single values, and routing decisions. Binary pass/fail, trivially automatable, zero ambiguity about what counts.
• F1 / token overlap — for structured extraction spanning multiple fields or spans, where a partially correct extraction deserves partial credit rather than a zero.
• Semantic/embedding similarity — for outputs correct in meaning but variable in wording, such as a paraphrased answer to a factual question.
• Execution-based matching — for code or SQL generation, where you run the output and diff the result set rather than comparing text. Two queries with completely different syntax can be equally correct.

The classic extraction benchmark SQuAD (Rajpurkar et al., 2016) popularized exactly this exact-match/F1 pairing for span extraction, and it's still the right instinct for any task where the model is pulling a fact out of a document rather than composing one. The catch: reference-based scoring is only as good as the reference. A mislabeled gold example silently caps your ceiling — no model can beat a wrong answer key.

That's why building the reference set is its own discipline, not an afterthought. Our guide on how to build your first golden dataset with 50 examples walks through sourcing and labeling those references before you compute a single score. And once you have a metric, you still need a bar — see our piece on setting numeric thresholds for evals that can actually fail for how to pick a pass/fail cutoff that isn't arbitrary.

Reference-Free Evaluation: Rubrics and LLM-as-Judge

When multiple outputs can be equally correct — a summary, an email draft, a brainstorm — there's no string to match against. Score these against a rubric that defines what good looks like: coverage, tone, structure, factual grounding. Then apply that rubric consistently, either with human raters or an LLM-as-judge.

Building a rubric starts with a question most teams skip: what job is this output actually hired to do? A summary written to help someone decide whether to read the full document has different success criteria than one meant to replace reading it entirely. Anchoring rubric criteria in the underlying job — the same lens behind Ulwick's Jobs-to-be-Done framework — keeps your criteria concrete instead of drifting toward vague "quality." Our complete guide to Jobs-to-be-Done is a useful reference for framing that job precisely.

A rubric axis list for summarization might look like this:

1. Faithfulness — does the summary avoid claims the source doesn't support?
2. Coverage — does it include the points a reader would consider essential?
3. Concision — is it shorter than the source without losing the above?
4. Readability — does it flow, or read like clipped fragments stitched together?

Human rubric scoring is the gold standard for reliability but doesn't scale past a few hundred examples without becoming a full-time job. That's where LLM-as-judge earns its place — not as a replacement for human judgment, but as a way to apply a human-defined rubric at volume.

Research on this pattern, including Zheng et al.'s 2023 work on MT-Bench and Chatbot Arena, found GPT-4-style judges agreeing with human preference at a rate roughly comparable to the agreement between two human annotators on the same pairs — a meaningfully strong signal, though not a perfect one.

The same research also documented real judge biases worth guarding against:

• Position bias — favoring whichever response appears first in a pairwise comparison.
• Verbosity bias — rating longer answers higher regardless of actual quality.
• Self-preference bias — a judge model rating outputs from its own model family more favorably.

None of these disqualify LLM-as-judge; they just mean you calibrate before you trust. Our deep dive on when to trust an LLM-as-judge covers exactly how to run that calibration against a human-scored sample before scaling it up.

The Classic Mistake: Scoring Creative Writing Like a Translation

BLEU and ROUGE were built to score machine translation and summarization against a reference by counting overlapping n-grams — they assume there's basically one right phrasing. Apply them to creative or open-ended writing and you penalize every valid alternative phrasing, rewarding outputs that happen to reuse the reference's exact words over outputs that are genuinely better.

BLEU (Papineni et al., 2002) was designed for a narrow world: machine translation, evaluated against multiple reference translations, where phrasing options really are constrained by the source sentence. It correlates reasonably well with human judgment in that narrow world. Pull it out of that world and the assumptions stop holding.

Callison-Burch, Osborne, and Koehn's widely cited 2006 re-evaluation of BLEU in machine translation research found the metric's correlation with human judgment broke down under conditions common in real deployments — different systems, different genres — sometimes losing any meaningful correlation at all.

The same pattern shows up in dialogue and open-ended generation. Liu et al.'s 2016 paper, memorably titled "How NOT to Evaluate Your Dialogue System," found that BLEU- and ROUGE-family metrics correlated weakly with human judgments of response quality — a warning that arrived a full decade before most teams building LLM products encountered it firsthand.

Picture the failure directly: a team building a marketing-email generator writes one "gold" example and scores every model output against it with BLEU. A genuinely stronger rewrite — different structure, sharper subject line, same message — scores lower than a mediocre draft that happens to reuse the gold example's phrasing. The metric is measuring similarity to one example, not quality.

ROUGE (Lin, 2004) has the same lineage and the same limitation for summarization specifically, which is why newer evaluation efforts like Stanford's HELM benchmark (Liang et al., 2022) push toward multiple complementary metrics — calibrated against human judgment — rather than leaning on any single overlap score as ground truth.

A Decision Tree for Matching Method to Task

Work through four questions in order: does a mechanical check exist, does exactly one answer count as correct, does correctness depend on audience or context, and can the task be split into a checkable part and a judgment part? The answers route you to exact match, semantic similarity, rubric scoring, LLM-as-judge, or a hybrid of two methods.

1. Can you write a program that checks correctness? A schema validator, SQL execution, a regex, a unit test.
   • Yes → use exact match or execution-based scoring. Done.
   • No → go to question 2.
2. Is there exactly one correct answer, possibly phrased differently? Factual extraction, classification labels, routing decisions.
   • Yes → use semantic similarity or embedding-based paraphrase matching.
   • No → go to question 3.
3. Does "good" depend on audience, tone, brand voice, or context rather than a fixed fact?
   • Yes → build a rubric first, then go to question 4.
4. Can the rubric be scored reliably at the volume you need, with a human?
   • Low volume → human rubric scoring, ideally as pairwise preference rather than absolute rating.
   • High volume → LLM-as-judge, calibrated against a small human-scored sample first.

The hybrid middle: tasks that are half-reference, half-judgment

Plenty of real tasks don't sit cleanly on one side. Code generation, SQL generation, and agent tool-calling all have a checkable core — did the code run, did the query return the right rows, was the correct tool invoked — wrapped in a judgment layer: was the code well-structured, was the query efficient, was the reasoning sound.

The fix isn't to pick one method. It's to split the task and apply both:

Where an output sits on this table often shifts with where it lands in the customer's experience, not just what type of task it is. An onboarding nudge and a billing-dispute resolution note might both be "open-ended chat" by task type, but the second carries much higher stakes for factual accuracy against account records. Mapping outputs against a customer journey clarifies which stage's stakes justify a strict, closer-to-reference-based bar versus a looser rubric.

Where Prodinja Fits Into Eval Planning

None of this is a one-time decision — most products have a mix of reference-based and reference-free tasks running side by side, and new features keep adding more of both. Prodinja's eval-planning flow is built around this exact fork: rather than defaulting every task to a golden dataset or every task to an LLM-judge, it's designed to walk you through classifying each task first, then matching the method to it.

That includes the harder case this article is really about: flagging when no gold reference answer exists at all, so you don't reach for exact match out of habit and quietly mis-score an open-ended task. It's one piece of Prodinja's broader simulated evals layer — a prototype experience for planning eval strategy, not a system that has run these evaluations for you.

Key Takeaways

• Ask first, measure second. Whether a task has one correct answer determines your entire eval methodology before you touch a metric.
• Reference-based metrics — exact match, F1, semantic similarity — only work when a canonical answer exists: extraction, classification, structured or executable output.
• Reference-free tasks — writing, summarization, chat — need rubrics or LLM-as-judge, not string-matching against a single "gold" example.
• BLEU/ROUGE were built for machine translation, not creative writing; applying them to open-ended generation penalizes valid alternative phrasing.
• Hybrid tasks are common. Score the checkable part (execution, schema) with reference-based methods and the judgment part with a rubric.
• Calibrate any LLM-judge against a small human-scored sample before trusting it at scale — and watch for position, verbosity, and self-preference bias.
• Anchor rubric criteria in the job the output is hired to do, not a generic notion of "quality."