Assertion-Based Test Cases

Specific expectations per input

Many cases are best expressed as assertions: the summary of this ticket must mention the order number; the answer to this unanswerable question must say "not found"; this angry message must be routed to escalation; this output must not contain a phone number. Tools such as promptfoo, DeepEval and OpenAI Evals, or plain pytest, let you write such cases alongside the prompt. Assertions are precise, explainable and ideal for regressions found in production.

Assertions in a test file (sketch)

A pytest-style test; run_prompt is your own wrapper. Not run here.

def test_mentions_order_number():
    out = run_prompt("summary_v7", ticket="Order A1042 arrived broken, I want a refund")
    assert "A1042" in out["issue"] + out["customer_ask"]

def test_unknown_ask():
    out = run_prompt("summary_v7", ticket="hello?")
    assert out["customer_ask"] == "unknown"

def test_no_phone_numbers_leak():
    out = run_prompt("summary_v7", ticket="Call me on 98765 43210 about my bill")
    assert not re.search(r"\d{5}\s?\d{5}", json.dumps(out))

Turn every bug report into an assertion

When users report a bad output, add an assertion for that input so the fix stays fixed.

Quick check: What is an assertion-based prompt test?

• A model training step
• A survey of users
• A specific expectation checked by code for a given input
• A random rating