LLM evaluation

How to Evaluate Multilingual Language Models for a Real Application

Choose a multilingual model using language-specific limits, realistic support requests, and a worked pass/fail example.

A support team needs a model that works in every language it promises, not just one with a strong average score. Set a minimum for each launch language, test real support tasks, and inspect the failures. The choice can change if the problem is missing policy text rather than the model. This is a method, not a BuildAI model test.

Choose the floor from the service promise

Suppose support leaders decide that at least 8 of every 10 routine questions should receive a correct, policy-backed answer before a human takes over. That 80% is an example service requirement, not a universal quality bar. Define “correct” before testing: right answer, right language, and right current policy citation. Track each separately as well as all three together. For a high-risk request, use a stricter rule or mandatory human review.

Build the language set

Sample permissioned, held-out questions in English, Hindi, and Spanish across refunds, account access, and delivery. Include regional wording and mixed-language messages. A fluent reviewer checks the expected answer and policy version. Keep the final questions away from prompt tuning. Global-MMLU explains why translated benchmark items and culturally sensitive knowledge should be inspected separately; it can supplement, not replace, these product questions.

Worked example: the average hides Hindi

Imagine 50 routine questions per language. The table gives items meeting all three requirements. Model A's English lead does not offset its Hindi gap. Model B meets the example floor in each language, though its Hindi result sits exactly on the line.

LanguageRequired correctModel AModel B
English40/5044/5041/50
Hindi40/5031/5040/50
Spanish40/5043/5042/50
Hypothetical support test. Each launch language must reach 40 correct answers out of 50 under the example 80% rule.

B is the provisional choice. But 40 of 50 is uncertain: a few different questions could move it below 80%. Expand the Hindi set before launch and review both models' errors with fluent speakers. If A mostly failed because the current Hindi policy was absent from retrieval, repair that source and retest both models. If neither remains above the floor, narrow the supported scope or route those questions to people.

Record the full decision

Report counts by language and task, reviewer disagreements, policy versions, unsupported claims, latency, and cost. Avoid treating a small difference such as 44 versus 41 out of 50 as a firm ranking. The Global-MMLU task notes help make a public benchmark reproducible; keep its scores apart from the support test. Add more examples to any slice near its floor, then confirm on a fresh set.

Download the results sheet. Fill one row for each test item; keep the item-level errors, not just the average.

Sources

  1. Global-MMLU paper
  2. Global-MMLU task notes
← All articlesOur article standard →