Speech processing
Speech-to-Text Benchmark Checklist: Accuracy, Latency, and Language Coverage
Compare live caption systems using word errors, critical names, first-caption delay, and a worked choice.
For live captions, the better recognizer is the one people can follow in time and trust on important words. Set a delay limit and a critical-word rule before comparing systems. A lower average word error rate can still lose if captions arrive late or omit names. This is a method, not a BuildAI benchmark.
Make the product limits concrete
In this example, the meeting team allows the first useful caption within two seconds for 95% of turns. It also requires at least 95% of speaker names and action dates to be correct, because a missed name or date changes follow-up work. These limits come from the meeting workflow, not from a speech benchmark. Decide what counts as a useful first caption, and have human reviewers mark the critical words in the reference transcript.
Collect and score the same audio
Use permissioned, held-out meetings with quiet speech, interruptions, accents, names, and overlapping voices. Split by meeting so related clips do not leak into tuning and final tests. Normalize case, punctuation, and written numbers by one rule before calculating word error rate (WER). WER is substitutions plus deletions plus insertions divided by reference words; Hugging Face Evaluate's implementation shows the formula. Report WER by condition, first-caption delay, stable-final delay, and missed critical words. If languages do not separate words with spaces, state the segmentation rule or add a character measure.
Worked example: two caption systems
Imagine 200 held-out turns, 100 of them containing one marked name or date. The numbers below are illustrative. System A has better WER, yet fails both product limits. System B meets the example limits.
| Measure | Product limit | System A | System B |
|---|---|---|---|
| WER, lower is better | Review by condition | 12% | 15% |
| First caption within 2 s | At least 190/200 turns | 182/200 | 194/200 |
| Critical words correct | At least 95/100 | 90/100 | 97/100 |
Choose B provisionally, then listen to failures. If A's missing first words come from voice activity detection, fix that component and rerun the whole caption path. If B makes meaning-changing errors in a specific accent group, the average may hide a launch problem. Public sets such as Common Voice can add coverage, but the final choice needs meetings like the product's own.
How much evidence is enough?
Two hundred turns are a starting example, not a promised sample size. A rate of 194/200 sits near the 95% line; repeat with more independent meetings before a launch claim. Report counts per language and noise condition, and include the range across meetings. Keep the model version, audio settings, transcript rules, and clip-level errors so a later update can be checked.
Download the results sheet. Fill one row for each test item; keep the item-level errors, not just the average.