Datasets, Experiments and Regression Tests

Offline evaluation before you ship

A dataset is a set of examples, each with inputs and optionally an expected output or reference. The best examples come from production: interesting or failed traces that you copy into a dataset and annotate. An experiment (LangSmith) or dataset run (Langfuse) executes your application on every example and applies evaluators: exact or fuzzy match, code checks such as "valid JSON" or "mentions the order ID", or LLM judges. Each experiment is stored with its outputs and scores, so you can compare two versions side by side and see exactly which examples improved or regressed. This is regression testing for prompts: before changing the prompt, model or retrieval settings, run the experiment and check you did not break cases that used to work. Keep the dataset versioned and growing; every production bug you fix should add an example so it cannot silently return.

An experiment with LangSmith

Requires langsmith with an API key. answer is your app function; the evaluator is a plain Python function returning a boolean.

from langsmith import Client

client = Client()
dataset = client.create_dataset(dataset_name="refund-faq")
client.create_examples(
    dataset_id=dataset.id,
    examples=[
        {"inputs": {"question": "Refund after 10 days?"}, "outputs": {"answer": "no"}},
        {"inputs": {"question": "Refund after 3 days?"}, "outputs": {"answer": "yes"}},
    ],
)

def target(inputs: dict) -> dict:
    return {"answer": answer(inputs["question"])}

def mentions_expected(outputs: dict, reference_outputs: dict) -> bool:
    return reference_outputs["answer"] in outputs["answer"].lower()

client.evaluate(
    target,
    data="refund-faq",
    evaluators=[mentions_expected],
    experiment_prefix="prompt-v4",
)

Adding a dataset item in Langfuse

The Langfuse SDK manages datasets too; items can also be added from a trace in the UI.

from langfuse import get_client

langfuse = get_client()
langfuse.create_dataset(name="refund-faq")
langfuse.create_dataset_item(
    dataset_name="refund-faq",
    input={"question": "Refund after 10 days?"},
    expected_output={"answer": "no"},
)

Previous-year question papers

Before an exam you practise on last year's papers and check your answers against the key. A dataset is your app's question paper; every time you change the prompt, you sit the paper again and compare marks with the last attempt.

त्वरित जाँच: What is the main purpose of running an experiment on a fixed dataset before deploying a new prompt?

• To reduce the token price charged by the provider
• To catch regressions by comparing scores with the previous version on the same examples
• To create user sessions
• To replace production tracing