Speech processing

A Practical Text-to-Speech Evaluation Guide

Test spoken order details for critical-value accuracy, listening quality, and first-audio delay.

A pleasant voice is not useful if it reads an order total incorrectly. For a voice checkout assistant, test the numbers and names first, then compare listening quality among systems that pass. A change in text normalization can reverse the result. This is a method, not a BuildAI listening test.

Set the script and limits

Write a held-out script that matches the product: order totals, dates, addresses, product names, and short help messages. Include “Your total is $15.50,” “Delivery is on 11 May,” and unfamiliar names. Mark each critical value in the source text before listening. The scoring unit is critical values spoken correctly ÷ critical values presented, reported separately for currency, dates, and names. In this example, the team requires every critical value correct in the pilot, and first playable audio within 1.2 seconds for 95% of requests. The delay limit comes from the voice interface budget, not from a TTS standard.

Listen without hiding errors

Randomize voice labels and order, keep playback volume similar, and ask fluent listeners whether speech is clear, natural, and complete. Check skipped or repeated words by listening to the recording; an automatic transcript can help find suspects but can introduce its own errors. ITU-T P.800 gives general guidance for subjective listening tests. The critical-value rule is the product team's own rubric.

Worked example: an order assistant

Suppose both voices read 200 marked critical values in 80 messages. Voice A is rated more natural but says “fifty” for “fifteen” once. Voice B reads all 200 correctly and is a little less natural. The figures are illustrative.

MeasurePilot limitVoice AVoice B
Critical values correct200/200199/200200/200
First audio within 1.2 sAt least 76/8078/8077/80
Listener naturalness, 1–5Compare after passing gates4.54.1
Hypothetical order-reading decision. Voice B passes both pilot gates; naturalness is considered only after value accuracy.

Use B for the next test. First inspect whether A's error came from text normalization rather than speech generation; a shared normalizer may change the comparison. Neither 200/200 nor 77/80 proves a low long-term error or delay rate. Test more independent messages, more names and formats, and busy-hour traffic before automating order confirmation. Keep critical details visible in text if spoken accuracy remains uncertain.

Record script ID, language, value type, expected and spoken value, listener decision, time to first audio, total duration, voice version, and normalization rule. Do not average currency errors into a large set of harmless words.

Download the results sheet. Fill one row for each test item; keep the item-level errors, not just the average.

Sources

  1. ITU-T P.800: subjective listening methods
  2. Hugging Face: Text-to-speech task guide
← All articlesOur article standard →