Word Error Rate

Word error rate (WER) is the standard score for speech recognition. It compares the recognizer’s transcript (the hypothesis) with a human-written reference and counts how many word-level edits turn one into the other, divided by the length of the reference. Lower is better, and 0 is a perfect match.

Definition

\[ \mathrm{WER}=\frac{S+D+I}{N} , \]

where \(S\) is the number of substituted words (a wrong word in the right place), \(D\) the number of deleted words (in the reference, missing from the hypothesis), \(I\) the number of inserted words (in the hypothesis, absent from the reference) and \(N\) the number of words in the reference. The counts come from the alignment of the two word sequences that needs the fewest edits, which is the Levenshtein (edit) distance computed over words. Take the reference “the cat sat on the mat” and the hypothesis “the cat sit on a mat today”: “sat” became “sit” (a substitution), “the” became “a” (a substitution) and “today” was added (an insertion), so \(S=2\), \(D=0\), \(I=1\), \(N=6\) and WER = 3/6 = 0.5.

Computing it

The alignment is a dynamic program over a table whose entry \(d_{ij}\) is the fewest edits needed to turn the first \(i\) reference words into the first \(j\) hypothesis words:

\[ d_{ij}=\min\big(d_{i-1,j-1}+[r_i\ne h_j],\; d_{i-1,j}+1,\; d_{i,j-1}+1\big), \]

with \(d_{i0}=i\) and \(d_{0j}=j\). The three options are match or substitute, delete \(r_i\), and insert \(h_j\). The last cell \(d_{NM}\) is \(S+D+I\). Several alignments can tie, which changes the split between S, D and I but not the total. The cost is proportional to the product of the two lengths.

import re
def wer(ref, hyp):
    r, h = ref.split(), hyp.split()
    d = [[0]*(len(h)+1) for _ in r+[0]]
    for i in range(len(r)+1): d[i][0] = i          # i deletions
    for j in range(len(h)+1): d[0][j] = j          # j insertions
    for i in range(1, len(r)+1):
        for j in range(1, len(h)+1):
            d[i][j] = min(d[i-1][j-1] + (r[i-1] != h[j-1]),   # match or substitution
                          d[i-1][j] + 1,                      # deletion
                          d[i][j-1] + 1)                      # insertion
    return d[-1][-1] / len(r)
tests=[("the cat sat on the mat","the cat sit on a mat today"),
 ("please do not cancel my order","please do not cancel my order"),
 ("please do not cancel my order","please do cancel my order"),
 ("please do not cancel my order","please do not cancel my order now"),
 ("yes","yes thank you for watching this video")]
for r,h in tests: print(f"{wer(r,h):.3f}  ref={r!r} hyp={h!r}")
ref="Dr. Smith paid $5, didn't he?"; hyp="doctor smith paid five dollars did not he"
norm=lambda s:re.sub(r"[^a-z' ]","",s.lower())
print("raw %.3f | lowercase, punctuation removed %.3f | same text after spoken-form normalization of both sides %.3f"%(
  wer(ref,hyp), wer(norm(ref),norm(hyp)), wer("doctor smith paid five dollars did not he",hyp)))
utts=[(5,1),(50,10),(4,2)]
print("mean of per-utterance WER %.3f, pooled %.3f"%(sum(e/n for n,e in utts)/3,sum(e for n,e in utts)/sum(n for n,e in utts)))

0.500  ref='the cat sat on the mat' hyp='the cat sit on a mat today'
0.000  ref='please do not cancel my order' hyp='please do not cancel my order'
0.167  ref='please do not cancel my order' hyp='please do cancel my order'
0.167  ref='please do not cancel my order' hyp='please do not cancel my order now'
6.000  ref='yes' hyp='yes thank you for watching this video'
raw 1.167 | lowercase, punctuation removed 1.000 | same text after spoken-form normalization of both sides 0.000
mean of per-utterance WER 0.300, pooled 0.220

Properties to know

• WER has no upper bound of 1. It counts edits, and a hypothesis longer than the reference can need more edits than the reference has words. The last test above scores 6.0: a one-word reference against a seven-word output with six insertions. This pattern is typical of a recognizer that makes up text on silence or noise.
• Equal WER does not mean equal damage. “please do cancel my order” (a dropped “not”) and “please do not cancel my order now” (an added “now”) both score 0.167 against “please do not cancel my order”, yet the first reverses the request and the second changes nothing that matters. WER weights every word the same.
• Normalization decides the number. The reference “Dr. Smith paid \$5, didn’t he?” and the hypothesis “doctor smith paid five dollars did not he” say the same thing. Compared as raw strings WER is 1.167, lowercasing and stripping punctuation gives 1.0, and expanding the abbreviation, the number and the contraction on the reference side (so that both sides use the spoken form) gives 0.0. A WER is only meaningful together with the text normalization that produced it, and two systems are comparable only when scored with the same one.
• Aggregation matters. With three utterances of 5, 50 and 4 reference words and 1, 10 and 2 errors, the mean of the per-utterance WERs is 0.300, while the pooled value, total errors over total words, is 0.220. The mean treats a 4-word utterance like a 50-word one. The usual convention is the pooled corpus value, with the per-utterance distribution reported separately because a few badly wrong utterances can dominate the mean.
• Uncertainty. WER on a test set is an estimate; resampling utterances (or speakers, since utterances from one speaker are correlated) gives an interval, and small gaps between systems on small sets may not be real.

Limits

WER measures text, not meaning, so errors in names, numbers, negations, and dates count as edits but receive no extra weight for their meaning; applications that need them add a separate accuracy measure on those items. It depends on the quality of the reference, and for casual speech human transcribers disagree. It assumes words can be separated, so for languages written without spaces between words the same computation is done over characters, as character error rate (CER). The edit model has no notion of reordering: a swapped pair of words counts as two substitutions.

Neighbors

Character error rate uses the same formula with characters. Sentence error rate counts the fraction of utterances with any error and is stricter. For speech synthesis there is no single reference waveform, so a common automatic check transcribes the synthesized speech with a fixed recognizer and computes WER against the input text, which measures intelligibility as that recognizer hears it.