Computer vision
How to Evaluate Image Captioning Models Beyond a Single Score
Review image captions against image purpose, required facts, nearby text, and invented details.
A useful caption tells a person what the image adds to the page without inventing facts. For a catalog accessibility lead, the decision is whether generated captions can be used as written, need review, or should be replaced by a human description. This is a method, not a BuildAI image test.
Write the image's job before scoring
Use the page around each image. A product photo may need the product type and visible color; a sizing diagram may need the actual measurements and what they refer to. A decorative image may need no caption at all. The W3C image decision tree asks what purpose the image serves, and its informative image guidance ties alternative text to context. Do not repeat a nearby product title unless the image adds something the title omits.
Use a filled review card
For each image, a reviewer lists required visual facts, facts already in nearby text, and facts that cannot be read reliably. Mark each generated caption as complete, missing a required fact, or inventing a fact. Keep omissions and inventions separate; a fluent sentence can still mislead.
| Image on page | Required facts from image | Nearby text already says | Caption decision |
|---|---|---|---|
| Blue rain jacket photo | Blue color; hood visible | “Rain jacket, size M” | “Blue hooded rain jacket” adds the visible details |
| Size chart | Chest 92 cm maps to M | “Choose your size” | Describe the measurement relationship; short caption alone is insufficient |
| Decorative texture | None | Article title gives context | Empty alt text is appropriate |
Worked example: choose the review path
Suppose 100 held-out images include 60 product photos, 25 charts, and 15 decorative images. For charts, the team decides that every measurement must be human checked because a wrong size can cause a return. For product photos, it allows automatic publication only when no reviewed caption invents a visible attribute in a larger final sample. In a first 100-image pilot, candidate A invents two product colors and omits four chart measurements; candidate B invents none but omits seven chart measurements. Neither is ready for automatic chart captions. B is a better pilot for product photos, pending more review.
“No errors in 100” would still be weak evidence that errors are rare; this sample is small. Expand the test across product types and seasons, and use a human review path for high-impact images. Record image type, page context, required facts, omissions, invented facts, reviewer decision, and latency. If the page text changes, revisit the caption.
Download the results sheet. Fill one row for each test item; keep the item-level errors, not just the average.