Evaluating Multimodal Systems

Metrics per task, plus your own test set

Each multimodal task has standard metrics. Retrieval uses recall@k: the fraction of queries whose correct item appears in the top k results. Speech recognition uses word error rate (WER): substitutions, deletions and insertions divided by the number of reference words; lower is better, and it can exceed 1. Captioning has n-gram overlap metrics such as BLEU and CIDEr and embedding-based scores such as CLIPScore, all of which correlate only loosely with human judgement. Visual question answering often uses exact or normalised match accuracy against reference answers. Public benchmarks are useful for picking candidate models, but they may not resemble your data and some have leaked into training sets. The most valuable asset is a task-specific eval set: a few hundred real examples from your domain with verified answers, including hard cases such as blurry photos, handwriting and regional languages, re-run on every model or prompt change.

Recall@k for image retrieval

Three queries, each with one correct image. Recall rises as k grows because more of the ranked list counts.

def recall_at_k(ranked, relevant, k):
    hits = sum(1 for q in ranked if relevant[q] in ranked[q][:k])
    return hits / len(ranked)

ranked = {"q1": ["img4", "img1", "img9"],
          "q2": ["img2", "img7", "img3"],
          "q3": ["img5", "img6", "img8"]}
relevant = {"q1": "img1", "q2": "img2", "q3": "img8"}
for k in (1, 2, 3):
    print(f"recall@{k} = {recall_at_k(ranked, relevant, k):.2f}")

Output:

recall@1 = 0.33
recall@2 = 0.67
recall@3 = 1.00

Word error rate from scratch

Edit distance over words. One deleted word in seven gives 0.143; one substitution in four gives 0.25.

def wer(reference, hypothesis):
    r, h = reference.split(), hypothesis.split()
    # d[i][j] = edits to turn the first i ref words into the first j hyp words
    d = [[0] * (len(h) + 1) for _ in range(len(r) + 1)]
    for i in range(len(r) + 1):
        d[i][0] = i
    for j in range(len(h) + 1):
        d[0][j] = j
    for i in range(1, len(r) + 1):
        for j in range(1, len(h) + 1):
            cost = 0 if r[i - 1] == h[j - 1] else 1
            d[i][j] = min(d[i - 1][j] + 1, d[i][j - 1] + 1, d[i - 1][j - 1] + cost)
    return d[len(r)][len(h)] / len(r)

print(round(wer("book a table for two at eight", "book table for two at eight"), 3))
print(round(wer("turn on the lights", "turn off the lights"), 3))

Output:

0.143
0.25

LLM-as-judge needs calibration

Using a strong model to grade free-form answers scales well, but check it against human grades on a sample first. Judges can favour longer answers and miss visual errors they cannot verify themselves.

Quick check: A reference transcript has 10 words. The ASR output has 1 substitution, 1 deletion and 1 insertion. What is the WER?

• 0.1
• 0.2
• 0.3
• 0.27