The Quality-Cost Frontier

Dominated versus frontier options

Prompt variants differ in quality and cost (prompt length, number of examples, model size, number of calls). An option is dominated if another is at least as good and no more expensive. The remaining options form the Pareto frontier: each is the best quality available at its cost. Choose a point on the frontier that meets your quality bar at acceptable cost, and include latency if it matters. Recompute when prices or models change.

Choose wisely, protect what works

Pick prompts on the quality-cost frontier, block regressions, and avoid overfitting to the score.

Finding the frontier among six variants, run

I ran this with Python 3 (scipy 1.18.1 where imported) on example or seeded simulated data, not results from a real product. Of six example variants, five are on the frontier; few-shot with a medium model is dominated by few-shot with a small model, which is slightly better and half the price. Whether to pay $7.80 per thousand for 0.86 instead of $1.30 for 0.80 depends on the quality bar.

variants = {  # name: (quality score, cost per 1k requests in $)
    "short prompt, small model": (0.71, 0.40), "long prompt, small model": (0.78, 0.90),
    "short prompt, large model": (0.83, 4.10), "long prompt, large model": (0.86, 7.80),
    "few-shot, small model": (0.80, 1.30), "few-shot, medium model": (0.79, 2.60)}
frontier = [n for n, (q, c) in variants.items()
            if not any(q2 >= q and c2 <= c and (q2, c2) != (q, c) for q2, c2 in variants.values())]
for n, (q, c) in sorted(variants.items(), key=lambda x: x[1][1]):
    print(f"{n:<27} quality {q:.2f}  ${c:.2f}/1k  {'frontier' if n in frontier else 'dominated'}")

Output:

short prompt, small model   quality 0.71  $0.40/1k  frontier
long prompt, small model    quality 0.78  $0.90/1k  frontier
few-shot, small model       quality 0.80  $1.30/1k  frontier
few-shot, medium model      quality 0.79  $2.60/1k  dominated
short prompt, large model   quality 0.83  $4.10/1k  frontier
long prompt, large model    quality 0.86  $7.80/1k  frontier

Set the quality bar first

Decide the minimum acceptable score before looking at costs, then pick the cheapest frontier option above it.

Quick check: When is a prompt variant "dominated"?

• It has the highest quality
• Another variant is at least as good and no more expensive
• It is the cheapest
• It was written first