Computer vision
How to Test Image Search Results in a Product Catalog
Grade image-search results, calculate ranking quality, and decide if a search system is useful at the top ranks.
For product search, ask whether the first few images help a shopper find the item they meant. Grade exact matches separately from useful substitutes and misleading lookalikes. Choose a system that meets a top-results target and the page's response-time limit. This is an evaluation method, not a BuildAI model ranking.
Write the grading rule first
For each query image, mark an exact item and size as 2, an acceptable substitute as 1, and an accessory, wrong size, or unrelated result as 0. If a wrong-size item would be unsafe or costly, give it a separate hard-failure label as well. Have a catalog expert resolve uncertain cases before seeing the system ranking. Include text queries separately if the site accepts them.
Set the product target from a task. In this example, a shopper should see an exact item in the first three results for at least 9 of every 10 queries, and 95% of searches should return in under 300 ms. The first limit reflects how many results the page shows without scrolling; the second comes from the page's total response budget. They are example limits, not universal benchmarks.
Test the full search path
Collect held-out query photos by category, including near duplicates, bad lighting, and lookalikes. Split by product family so a nearly identical photo does not leak from tuning to test. Keep the catalog, index, distance rule, filters, and reranker fixed when comparing encoders. Measure query latency and index refresh time as well as ranking quality. scikit-learn's nDCG documentation defines a rank-sensitive score for graded relevance; an exact-match-in-top-three rate is easier to explain to a merchant.
Worked example: a shoe photo
Suppose system A returns [exact item, accessory, substitute] and B returns [substitute, exact item, wrong size]. With the 2/1/0 labels, A's discounted gain for the first three is 2 + 0/1.585 + 1/2 = 2.5; B's is 1 + 2/1.585 + 0/2 ≈ 2.26. The ideal order [exact, substitute, zero] scores about 2.63. That makes nDCG@3 about 0.95 for A and 0.86 for B for this one query. Neither single query can establish a system winner.
| Result rank | System A | Grade | System B | Grade |
|---|---|---|---|---|
| 1 | Exact shoe | 2 | Substitute shoe | 1 |
| 2 | Accessory | 0 | Exact shoe | 2 |
| 3 | Substitute shoe | 1 | Wrong size | 0 |
Across a larger example set, imagine A finds the exact item in the first three for 92 of 100 queries, B for 86. A clears the 90% target; B does not. But 100 queries split across five categories leave only about 20 per category. Check the category failures and add more queries before claiming coverage of every category. If A misses the 300 ms limit while B passes, tune the index or choose a different operating point.
Keep the decision honest
Report exact-item top-three rate, graded nDCG@3, wrong-size hits, latency distribution, and category counts. Put an uncertainty range around each rate; a result close to 90% needs more queries. Check a fresh set after changing the index or model. Revisit the choice when the catalog grows or the page shows a different number of results.
Download the results sheet. Fill one row for each test item; keep the item-level errors, not just the average.