Word error rate: read a speech benchmark correctly

What the score actually counts

Word error rate is based on substitutions, deletions and insertions. A substitution occurs when a recognized word replaces the reference word. A deletion occurs when the system leaves out a reference word. An insertion occurs when the system adds a word that is not present in the reference. The total edit count is divided by the number of reference words and expressed as a percentage.

The reference matters as much as the output. If it treats a contraction differently, includes a filler that the system omits, or writes a number in another form, the apparent disagreement can change. An evaluation therefore needs a stated normalization procedure. Without that context, a precise-looking score can conceal different definitions of an error.

Why error rate is not an accuracy promise

Subtracting word error rate from a hundred does not produce a reliable probability that the next word in your file will be correct. The denominator is the number of reference words, and inserted words increase the numerator without adding to that denominator. A sufficiently poor output can therefore exceed a hundred percent error.

Errors also have different practical consequences. A missing filler may be unimportant in a readable meeting note. A missing negation can reverse the meaning of an instruction. A single wrong surname can make a quotation unusable. A benchmark treats these through its edit metric; your review needs to consider the purpose and consequence of the text as well.

Compare models within the same evaluation

The original Whisper paper reports multiple datasets. FLEURS and Common Voice contain different recordings and evaluation conditions. A lower value on one dataset does not establish that a model would beat another system tested elsewhere. The useful comparison is a row of models evaluated under the same stated procedure.

The English example below illustrates why the dataset name belongs next to the score. Tiny, Base and Small all improve in the published English FLEURS row as model size increases. Their Common Voice values are different. Neither column is a promise that a new interview or lecture will have that error rate. The table describes the original paper’s models, not a fresh test of browser quantization.

Character scores need a different label

The paper’s multilingual normalization splits Chinese, Japanese, Thai, Lao and Burmese into characters. Those results are effectively character error rates. MurmurScribe labels them CER in its language table so a reader does not mistake them for word error rates.

This is more than a naming detail. A character and a word are different units, and languages differ in how written text marks boundaries. Averaging all rows together without understanding the scoring units would produce a misleading summary. Compare a language with itself under a consistent method. If you need to compare languages for a project, define the practical outcome you care about instead of using a mixed metric as a ranking.

Run a useful local sample

Choose a representative recording passage rather than the cleanest sentence you can find. Include the kinds of names, accents, background noise and interruptions present in the actual job. Write a careful reference by listening, and have another competent listener resolve unclear passages if the result matters.

Keep the input, task and language setting fixed when comparing models. Note any manual correction convention, such as whether fillers are retained. Review both the numerical differences and the mistakes that change meaning. If a larger model removes punctuation errors but still misses the key technical term, the practical benefit may be smaller than the overall score suggests.

The local transcriber can help produce the candidate drafts. It is not an automatic benchmark system, and the page does not claim to calculate a certified evaluation. Preserve the original recording and your reference so another reviewer can understand what you compared.

Decide what to review first

Start with passages that will be quoted or used for decisions. Check names, amounts, dates, units and negation directly against the recording. Then inspect speaker transitions and places where voices overlap. Fluency is not a reason to skip a sentence; an invented phrase can read more smoothly than the words that were actually spoken.

Mark uncertainty in the working draft rather than silently selecting a plausible word. If the recording cannot resolve a phrase, the transcript should preserve that limitation. A clean document is not always a faithful record. The final editing convention should be stated so someone reading later can tell whether they are seeing verbatim speech, a readable transcript or an editorial summary.

Use a benchmark as a way to choose what to test. Use listening and a project-specific review process to decide what to keep.

Worked example

Consider an illustrative reference containing 100 words. A draft has 5 substitutions, 3 deletions and 2 insertions. The calculation is (5 + 3 + 2) ÷ 100 × 100 = 10% WER. These are invented example inputs used to show the method, not a measured result. If the same ten errors include a reversed negation and an incorrect dose, the practical risk can be much greater than the percentage communicates. If a different scoring convention writes a spoken number as several words, the reference length and alignment may change. Keep the convention with the score.

Open the local transcriber to test an authorized recording and review the draft against the sound.

Make the evaluation reproducible

A score becomes more useful when another person can reconstruct it. Keep the model name and revision, the task, the language selection, the source recording identifier and the reference transcript. State whether the evaluation used word or character units and which text normalization steps were applied. Those details prevent an ordinary formatting change from being mistaken for a recognition improvement.

Separate the evaluation copy from the publication copy. The publication may use house style, headings or expanded abbreviations that are helpful to readers but different from the scoring reference. Editing one without documenting it can make later comparisons inconsistent. If the reference changes because a reviewer heard a word differently, update the comparison and explain why.

A practical review sheet can group errors by consequence as well as by edit type. Mark a proper-name error, a changed quantity or a reversed instruction separately from punctuation and presentation. This does not replace WER with an invented metric. It adds the context needed to decide whether the remaining draft is useful for the intended work.

Do not select a model solely because it wins on a passage you have repeatedly tuned around. Keep another representative passage for a final check, especially if the recording contains more than one speaker or setting. A model choice should survive ordinary variation in the job. When it does not, record the limitation and choose a workflow with sufficient human review rather than advertising the best-looking test as a universal result.