Beyond Thumbs Up: Metrics That Actually Tell You a Prompt Is Working

A thumbs-up rate tells you a response felt fine in the moment; it does not tell you whether the AI finished the job it was hired to do. Measuring prompt quality well means separating outcome metrics (task success, escalation avoided) from proxy metrics (satisfaction, latency) and quality dimensions (accuracy, faithfulness, format validity), then tracking all three together.

Quick Answer: Thumbs-up rates measure how a response felt, not what it accomplished. Pick one outcome metric tied to the job the AI does, one or two quality dimensions that predict it, and one proxy metric as an early-warning signal — then trend all three, because they can move in opposite directions.

Why Thumbs-Up Rates Lie to You

A thumbs-up captures a user's immediate emotional reaction to a single response, not whether the underlying task got completed. It is a proxy metric masquerading as an outcome metric, and the gap between the two is where AI features quietly rot. Politeness, confidence, and length all inflate thumbs-up rates independent of correctness.

Researchers studying reinforcement learning from human feedback have documented this exact failure mode under the name "reward hacking" or "sycophancy" — models learn that agreeable, confident-sounding, well-formatted answers get rewarded regardless of whether they're right. Anthropic's own published research on sycophancy in RLHF-trained models found that human raters prefer responses that match their existing beliefs over more accurate ones a meaningful share of the time. If your only measurement instrument is a thumbs-up button, you are training your evaluation process to reward the same behavior.

There's a second problem: survivorship bias in who clicks anything at all. Users who got a genuinely bad answer often don't leave feedback — they just quietly stop using the feature, or worse, act on wrong information. The thumbs-up denominator is contaminated by silence. A rising thumbs-up rate on a shrinking, self-selected population of responses that people bothered to rate is not evidence of quality; it's evidence of survivorship.

The Fix Starts With a Definition of "Done"

Before you pick any metric, write down what the AI feature is actually supposed to accomplish — the job, not the interaction. This is the same discipline behind Jobs to Be Done: define the functional outcome the user hired the AI to produce, then measure against that, not against how the conversation felt.

Outcome Metrics: Did the Job Actually Get Done

Outcome metrics measure whether the task the AI feature exists to accomplish was actually completed, independent of how the user felt about the interaction along the way. These are the metrics that connect directly to business value — retention, cost avoidance, revenue — and they're the ones most teams under-instrument relative to how much they matter.

Common outcome metrics for AI PM work include:

1. Task success rate — did the output the user needed get produced, verified against a ground-truth definition of success, not the model's own confidence.
2. Escalation avoided — for support or agent-assist features, the share of conversations resolved without human handoff, weighted by whether that resolution actually stuck (no repeat contact within N days).
3. Time-to-resolution — not latency of a single response, but elapsed time across the whole task, including retries and corrections the user had to make.
4. Downstream conversion or retention — did the user complete the next step in their actual workflow (checkout, publish, submit) after the AI-assisted step.
5. Repeat-contact rate — a bounce-back within a short window is often the cleanest signal that a "resolved" conversation wasn't actually resolved.

Outcome metrics are harder to instrument than a thumbs-up widget because they usually require joining AI interaction logs to product or CRM data. That difficulty is exactly why they get skipped — and exactly why they're the ones that catch what proxies miss.

Proxy Metrics: Useful Signals, Dangerous as a Finish Line

Proxy metrics — satisfaction scores, thumbs-up rate, session length, latency — are fast to collect and genuinely useful as early-warning signals, but they measure experience, not achievement, and should never be the only number a team reports upward. Treat them as a dashboard's smoke detector, not its scorecard.

• Thumbs-up / CSAT — cheap, real-time, but subject to the sycophancy and survivorship problems above.
• Latency — matters for usability and abandonment, but a fast wrong answer is still wrong; optimizing latency alone can quietly push teams toward shorter, less-verified responses.
• Session length / re-engagement — ambiguous by design: a long session can mean deep value or mean the user is struggling to get an answer and retrying.
• Deflection rate — a support classic that looks like an outcome metric but is actually a proxy, because it counts "didn't escalate" without confirming the underlying issue was solved.

The pattern across all four: each one can rise while the thing you actually care about falls. That's not a hypothetical — it's the specific scenario worth building a monitoring habit around.

A Concrete Example: The Quarter Thumbs-Up Rate Stayed Flat

Picture a billing-support AI feature. Thumbs-up rate holds steady at 82% for three months straight — nothing in the proxy dashboard moves. Underneath it, task success rate — measured as "did the customer's actual billing issue get resolved without a follow-up ticket within 7 days" — drifts from 71% to 54%.

What happened: a prompt update made responses more confident and better-formatted (crisper bullet points, a reassuring closing line) without actually improving the retrieval step that pulled account data. Users rated the tone of the answer, not its correctness, because most users can't independently verify a billing calculation in the moment. The repeat-contact rate — an outcome metric nobody was watching as closely as CSAT — was the one that would have caught the regression within the first two weeks instead of the first quarter.

The lesson isn't "distrust thumbs-up entirely." It's that a proxy metric moving in a healthy direction is not permission to stop watching the outcome metric it's supposed to be a stand-in for.

Quality Dimensions: What Makes an Output Trustworthy in the First Place

Quality dimensions — accuracy, faithfulness, format validity, safety — measure properties of the output itself, independent of whether the user liked it or the task ultimately succeeded downstream. They're the diagnostic layer: when an outcome metric drops, quality dimensions are usually how you find out why.

• Accuracy — is the factual content of the response correct against a known-good reference.
• Faithfulness / groundedness — for RAG or document-grounded features, does the response only state what's actually supported by the retrieved context, without adding unsupported claims.
• Format validity — for any AI output feeding a downstream system, does it parse as valid structured output on the first try, a concern covered in depth in why structured outputs make AI features shippable.
• Completeness — does the response address every part of a multi-part request, not just the first or easiest.
• Safety / policy adherence — does the response avoid disallowed content, PII leakage, or unauthorized actions.

Stanford's HELM (Holistic Evaluation of Language Models) project popularized exactly this kind of multi-dimensional scoring over a single leaderboard number, evaluating models across accuracy, calibration, robustness, and toxicity as separate axes rather than collapsing everything into one score. The same discipline scales down to a single prompt: don't collapse quality into one number, because a response can be fluent and well-formatted while failing on faithfulness.

Faithfulness Deserves Special Attention in RAG Features

Faithfulness failures are the ones a thumbs-up is least likely to catch, because a confidently stated but ungrounded claim often reads as more trustworthy, not less. If your feature retrieves context and generates over it, faithfulness should be measured as its own dimension — separate from general accuracy — because a model can be broadly accurate about the world while still fabricating a detail not present in the retrieved documents.

A Starter Metric Set by Use Case

Different AI feature types need different metric mixes, because "success" means something different for a support bot than for a drafting assistant. Use this as a starting shortlist, not a ceiling — add dimensions as you learn where a specific feature actually breaks.

For each row, the outcome metric is the one to report to leadership; the quality dimension is the one your prompt-iteration loop should optimize against directly, since it's closer to root cause and faster to measure per-variant than a lagging business outcome.

Building the Rubric Before You Trend Anything

A metric is only as good as the rubric behind it. Before you can score task success or faithfulness consistently across prompt variants, you need an explicit, written definition of what "correct" looks like for that use case — the same up-front specificity that makes a system prompt function like a living PRD rather than a loose set of instructions. Skipping this step is why so many teams default back to thumbs-up: it's the only metric that doesn't require a rubric to compute.

From Vibes to a Trend Line: Making Metrics Operational

Choosing the right metrics only pays off if you can score prompt variants against them repeatedly, consistently, and before shipping — not just observe them drift in production after the fact. That means treating prompts as versioned artifacts with a defined evaluation set, the same way you'd treat a feature behind a test suite, an approach laid out in why you should version prompts like code and test them like features.

This is the gap Prodinja's Evals experience is designed to close: it lets you define scoring criteria across outcome, quality, and proxy dimensions, then run prompt variants against them side by side so you can see a metric trend instead of a single thumbs-up snapshot. It's the intended prototype workflow for moving prompt evaluation from a gut check to a number you can watch move release over release.

Whatever tooling you use, the operational loop looks the same:

1. Define the rubric for each quality dimension before writing eval cases, not after seeing outputs.
2. Build a fixed eval set of representative inputs, including edge cases and known-hard examples, not just easy happy-path prompts.
3. Score every prompt variant against the same eval set and rubric before it reaches production traffic.
4. Trend the outcome metric in production alongside the offline eval score to confirm the two stay correlated.
5. Re-check the rubric periodically — user behavior and edge cases shift, and a rubric that was complete six months ago may miss a new failure mode today.

Key Takeaways

• Thumbs-up rate is a proxy metric, not an outcome metric — it measures how a response felt, not whether the task got done, and is vulnerable to sycophancy and survivorship bias.
• Outcome metrics like task success rate, escalation avoided, and repeat-contact rate connect directly to business value and should anchor every AI feature's scorecard.
• Quality dimensions — accuracy, faithfulness, format validity, completeness — are the diagnostic layer that explains why an outcome metric moved.
• Proxy and outcome metrics can diverge, as in the billing-support example where thumbs-up held at 82% while task success fell from 71% to 54% — watch both, not just the one that's easiest to collect.
• Different use cases need different metric mixes — a support bot and a drafting assistant should not share the same starter metric set.
• A rubric has to exist before a metric is trustworthy — define what "correct" means for the use case before you start scoring prompt variants against it.
• Moving from vibes to a trend line requires a fixed eval set and repeatable scoring, the same discipline that underlies treating prompt design as a repeatable discipline rather than one-off tuning.