# Datasets, Experiments and Regression Tests — LLM Observability (Langfuse / LangSmith)

Source: https://www.skillbyai.com/en/llm-observability/lo-datasets

> Build an evaluation dataset from traces and run experiments to compare prompt or model versions.

## Offline evaluation before you ship

A **dataset** is a set of examples, each with inputs and optionally an expected output or reference. The best examples come from production: interesting or failed traces that you copy into a dataset and annotate. An **experiment** (LangSmith) or dataset run (Langfuse) executes your application on every example and applies **evaluators**: exact or fuzzy match, code checks such as "valid JSON" or "mentions the order ID", or LLM judges. Each experiment is stored with its outputs and scores, so you can **compare two versions side by side** and see exactly which examples improved or regressed. This is regression testing for prompts: before changing the prompt, model or retrieval settings, run the experiment and check you did not break cases that used to work. Keep the dataset versioned and growing; every production bug you fix should add an example so it cannot silently return.

## An experiment with LangSmith

Requires `langsmith` with an API key. `answer` is your app function; the evaluator is a plain Python function returning a boolean.

```python
from langsmith import Client

client = Client()
dataset = client.create_dataset(dataset_name="refund-faq")
client.create_examples(
    dataset_id=dataset.id,
    examples=[
        {"inputs": {"question": "Refund after 10 days?"}, "outputs": {"answer": "no"}},
        {"inputs": {"question": "Refund after 3 days?"}, "outputs": {"answer": "yes"}},
    ],
)

def target(inputs: dict) -> dict:
    return {"answer": answer(inputs["question"])}

def mentions_expected(outputs: dict, reference_outputs: dict) -> bool:
    return reference_outputs["answer"] in outputs["answer"].lower()

client.evaluate(
    target,
    data="refund-faq",
    evaluators=[mentions_expected],
    experiment_prefix="prompt-v4",
)
```

## Adding a dataset item in Langfuse

The Langfuse SDK manages datasets too; items can also be added from a trace in the UI.

```python
from langfuse import get_client

langfuse = get_client()
langfuse.create_dataset(name="refund-faq")
langfuse.create_dataset_item(
    dataset_name="refund-faq",
    input={"question": "Refund after 10 days?"},
    expected_output={"answer": "no"},
)
```

## Previous-year question papers

Before an exam you practise on last year's papers and check your answers against the key. A dataset is your app's question paper; every time you change the prompt, you sit the paper again and compare marks with the last attempt.

**Quiz:** What is the main purpose of running an experiment on a fixed dataset before deploying a new prompt?

- [ ] To reduce the token price charged by the provider
- [x] To catch regressions by comparing scores with the previous version on the same examples
- [ ] To create user sessions
- [ ] To replace production tracing

*Answer:* To catch regressions by comparing scores with the previous version on the same examples. Running both versions on the same examples gives a fair comparison and shows which cases improved or broke.