WER, and Why Word Error Rate Lies to Product Managers

Word Error Rate treats every mistake as equally bad, so a model can post an impressive 5% WER while butchering every customer name, SKU, and dollar figure in your transcripts. For product decisions, the errors that matter are the ones that break a user's task — not the aggregate edit distance between a transcript and ground truth. Measuring STT quality means pairing WER with entity error rate, intent accuracy, and correction rate.

Quick Answer: WER counts every substitution, insertion, and deletion as one point, regardless of whether the word was "the" or a patient's medication name. A low WER can still fail users if the missed words are the ones the task depends on — so pair it with entity error rate, intent accuracy, and real correction-rate data from usage, not vendor benchmarks.

What Word Error Rate Actually Measures

Word Error Rate is a normalized edit-distance score: (substitutions + insertions + deletions) / total reference words, computed by aligning a hypothesis transcript against a human-verified reference. It comes from decades of speech-recognition research and remains the industry's default reporting metric because it's cheap to compute and easy to compare across vendors and papers.

The formula rewards nothing about which words were wrong. A model that mangles three filler words ("um," "so," "like") posts the same WER penalty as one that mangles three drug dosages. Both are "3 errors" in the numerator. That equivalence is the entire problem for product decisions, because your users don't experience errors as a rate — they experience them as specific, individual failures at specific moments.

Where WER Comes From and Why It Persists

The metric traces back to speech-recognition evaluation work at institutions like NIST, which has run standardized STT benchmarks (SCTK scoring toolkit, Switchboard and CALLHOME corpora) since the 1990s. Vendors including Google, Amazon, and OpenAI still headline WER in their model cards because it's the one number every competitor reports the same way, making it the path of least resistance for marketing comparisons.

That comparability is also its weakness for you. A benchmark WER is measured on a generic corpus — often read speech, call-center audio, or podcast clips — that has nothing to do with your product's vocabulary. Whisper, Deepgram Nova, and AssemblyAI all publish strong WER numbers on public benchmarks like LibriSpeech or Common Voice; none of those benchmarks contain your customers' company names, product SKUs, or internal jargon.

Why WER Lies: The Blind Spots That Hurt PMs

WER's core flaw is that it's error-agnostic: it can't distinguish a typo in a stopword from a mangled proper noun that derails an entire downstream workflow. A voice assistant that mishears "set a reminder for tomorrow" as "set a reminder for tomorrows" scores an error identical in weight to one that turns "email Sarah Chen" into "email Sarah Chin" — the first is harmless, the second sends output to the wrong task entirely.

Three specific blind spots recur across every STT product review:

1. Uniform error weighting. Every substitution costs exactly one point whether it's an article or an account number. Products that route action based on transcribed content (names, dates, amounts) need those specific tokens right far more than they need generic fluency.
2. No task-completion signal. WER is computed against a static reference transcript, not against whether the user's underlying job — booking a meeting, filing an expense, dictating a spec — actually succeeded.
3. Benchmark-to-production mismatch. Vendor-reported WER comes from curated test sets with clean audio and standard accents; your production traffic includes cross-talk, accents, background noise, and domain vocabulary the benchmark never modeled.

Deb Roy's and others' work on conversational speech (echoed in MIT Media Lab research on naturalistic language) has long noted that transcription accuracy and communicative success are related but distinct constructs — a transcript can be "close enough" on paper and still fail the listener's actual goal.

A Concrete Illustration

The middle row has a higher WER than the top row yet causes zero user pain, while the top row's smaller-looking error is the one that actually breaks the workflow. If you're only reporting an aggregate percentage, you'd rank these backwards.

Better Proxies: Entity Error Rate, Intent Accuracy, Correction Rate

Three metrics predict user pain far better than aggregate WER, because each measures errors against what the transcript is actually used for rather than against raw text similarity. None require abandoning WER — they sit alongside it as the metrics that actually belong in a product review.

Entity Error Rate (EER)

Entity Error Rate scores WER only on the tokens that matter: names, numbers, dates, product terms, and other named entities extracted from the reference transcript. It's computed the same way as WER but restricted to a tagged entity subset, so a model can have a mediocre overall WER and an excellent EER, or vice versa — and EER is the one that predicts whether a CRM field, calendar invite, or support ticket gets filled in correctly.

• Tag entities in your reference transcripts using a named-entity recognizer or, better, your own domain dictionary (customer names, SKUs, internal tool names).
• Report EER separately for each entity class — person names typically fail at a much higher rate than dates or numbers, and blending them hides which class needs attention.
• Track EER trend over model or vendor changes independently from headline WER; the two frequently move in opposite directions after a fine-tune.

Intent Accuracy

Intent accuracy measures whether the downstream action the system took matches what the user actually meant, independent of exact word matching. A transcript can be imperfect and still route to the correct intent ("set a timer for ten minutes" transcribed with one wrong filler word still triggers the right timer action). This metric borrows directly from the Jobs to Be Done framing: what job was the user trying to get done with their voice, and did the system get them there?

Grounding this in a proper jobs framework — the kind covered in our guide to the complete Jobs to Be Done framework — keeps intent accuracy from becoming its own vague catch-all. Define the finite set of jobs your voice surface supports, then score intent-match rate per job, not as one global number.

Correction Rate

Correction rate is the share of dictated utterances a user manually edits after the system transcribes them — the single most direct, zero-inference signal you have, because it's measured from real behavior instead of a labeled test set. If users correct 30% of transcribed customer names but only 2% of transcribed dates, that gap tells you exactly where the model is failing for your users specifically, no benchmark required.

• Instrument every editable transcription field to log whether, and where, the user changed the raw STT output.
• Segment correction rate by field type, entity class, and user cohort — aggregate correction rate hides the same problem aggregate WER does.
• Treat a rising correction rate as a leading indicator of churn risk on voice features, since it's a direct proxy for user-perceived friction, not a proxy for a proxy.

Quick comparison:

Building a Domain-Specific Test Set

A domain-specific test set is a curated collection of real or realistic utterances, weighted toward the entities and phrasings your product actually encounters, scored against your own reference transcripts rather than a vendor's public benchmark. Building one is the single highest-leverage step a PM can drive, because it's the only way to make an accuracy number mean something for your users specifically.

1. Pull real usage transcripts, not synthetic scripts. Sample a stratified set of actual dictation sessions across accents, background-noise conditions, and device microphones — synthetic, studio-recorded test sets systematically understate real-world error rates.
2. Tag domain entities exhaustively. Every customer name, product SKU, internal acronym, and number format your users actually say needs a labeled reference, not just common nouns.
3. Weight the set toward what breaks tasks. If 80% of your voice traffic is scheduling and 20% is free-form notes, your test set's entity distribution should roughly mirror that, not an even split.
4. Re-run it on every vendor or model change. A vendor's WER improvement on their own benchmark doesn't guarantee an improvement on your entity mix — that's the whole point of building this set in the first place.
5. Version and freeze test-set snapshots so accuracy comparisons across model versions are apples-to-apples, the same discipline you'd want in a customer journey baseline before and after a UX change.

Why Real Transcripts Beat Vendor Benchmarks

Vendor benchmarks answer "how good is this model on data curated to make models look good." Your domain test set answers "how good is this model on the sentences my users actually say." Those are different questions, and only one of them predicts your correction rate. A model that leads every public leaderboard can still underperform a lower-ranked competitor on your specific vocabulary — pharmaceutical names, regional accents, product jargon — because none of that showed up in the benchmark's training or test distribution.

This is the same discipline good PMs already apply to feature evaluation generally: trust your own usage data over a vendor's marketing claim, the same way you wouldn't size a rollout off someone else's voice latency budget without checking it against your own network and device conditions.

Designing for Errors You Can't Fully Eliminate

No STT vendor will ever hit zero error rate on your domain vocabulary, so the product decision that matters most is how gracefully your interface surfaces and lets users fix the errors that do occur. This reframes the metrics conversation: instead of chasing an unreachable "perfect WER," design the correction path so entity errors get caught before they cause harm.

That means treating dictation as a capture-then-confirm pattern rather than a fire-and-forget transcription, an approach explored in depth in our piece on voice dictation as a capture feature. It also means the interaction design for fixing a wrong word matters as much as the recognition model itself — see our broader look at error correction in multimodal interfaces for the interaction patterns that make corrections fast rather than punishing.

For a PM building the business case, both moves — instrumenting correction rate and designing the correction UI — pay off independent of which STT vendor you eventually pick, because they're under your control even when raw model accuracy isn't. Our complete guide to multimodal voice walks through how these pieces fit into a broader voice-input strategy beyond accuracy metrics alone.

Where Prodinja Fits

Key Takeaways

• WER weighs every error equally, so it can't tell you whether a model's mistakes land on filler words or on the proper nouns your workflow depends on.
• Entity Error Rate isolates accuracy on names, numbers, and dates — the tokens most likely to break a task when transcribed wrong.
• Intent accuracy measures whether the right action fired, independent of whether the transcript matched word-for-word, borrowing directly from Jobs to Be Done thinking.
• Correction rate is the most direct signal available because it's measured from real user edits, not a labeled benchmark — segment it by entity type and field to find where it hurts most.
• Vendor-published WER numbers are measured on generic benchmarks, not your domain vocabulary — build and freeze your own test set from real usage transcripts.
• Perfect recognition isn't achievable, so the correction interface matters as much as the model — design for graceful, fast fixes rather than chasing an unreachable zero-error target.