Speech processing
A Practical Text-to-Speech Evaluation Guide
Test spoken order details for critical-value accuracy, listening quality, and first-audio delay.
A pleasant voice is not useful if it reads an order total incorrectly. For a voice checkout assistant, test the numbers and names first, then compare listening quality among systems that pass. A change in text normalization can reverse the result. This is a method, not a BuildAI listening test.
Set the script and limits
Write a held-out script that matches the product: order totals, dates, addresses, product names, and short help messages. Include “Your total is $15.50,” “Delivery is on 11 May,” and unfamiliar names. Mark each critical value in the source text before listening. The scoring unit is critical values spoken correctly ÷ critical values presented, reported separately for currency, dates, and names. In this example, the team requires every critical value correct in the pilot, and first playable audio within 1.2 seconds for 95% of requests. The delay limit comes from the voice interface budget, not from a TTS standard.
Listen without hiding errors
Randomize voice labels and order, keep playback volume similar, and ask fluent listeners whether speech is clear, natural, and complete. Check skipped or repeated words by listening to the recording; an automatic transcript can help find suspects but can introduce its own errors. ITU-T P.800 gives general guidance for subjective listening tests. The critical-value rule is the product team's own rubric.
Worked example: an order assistant
Suppose both voices read 200 marked critical values in 80 messages. Voice A is rated more natural but says “fifty” for “fifteen” once. Voice B reads all 200 correctly and is a little less natural. The figures are illustrative.
| Measure | Pilot limit | Voice A | Voice B |
|---|---|---|---|
| Critical values correct | 200/200 | 199/200 | 200/200 |
| First audio within 1.2 s | At least 76/80 | 78/80 | 77/80 |
| Listener naturalness, 1–5 | Compare after passing gates | 4.5 | 4.1 |
Use B for the next test. First inspect whether A's error came from text normalization rather than speech generation; a shared normalizer may change the comparison. Neither 200/200 nor 77/80 proves a low long-term error or delay rate. Test more independent messages, more names and formats, and busy-hour traffic before automating order confirmation. Keep critical details visible in text if spoken accuracy remains uncertain.
Record script ID, language, value type, expected and spoken value, listener decision, time to first audio, total duration, voice version, and normalization rule. Do not average currency errors into a large set of harmless words.
Download the results sheet. Fill one row for each test item; keep the item-level errors, not just the average.