User Feedback and Scores

Turning reactions into data

The cheapest quality signal is your users. Explicit feedback is what they tell you: thumbs up or down, a star rating, a correction they type. Implicit feedback is what they do: copying the answer, retrying the same question, abandoning the chat, or escalating to a human agent. To be useful, feedback must be linked to the exact trace that produced the answer. The usual pattern: your backend returns the trace ID (or a run ID you generated in advance) with the response; the frontend sends it back with the user's rating; your backend writes a score (Langfuse) or feedback (LangSmith) against that ID. Keep in mind the biases: most users never rate, and angry users rate more often than happy ones, so feedback is better for finding bad examples and comparing versions than for an absolute quality number. Combine it with automated evaluation for coverage.

Feedback in LangSmith with a pre-set run ID

Generating the run ID up front lets you hand it to the frontend before the user reacts. answer is a @traceable function.

import uuid
from langsmith import Client

client = Client()
run_id = uuid.uuid4()

reply = answer("Where is my order?", langsmith_extra={"run_id": run_id})
# ... send reply and str(run_id) to the frontend ...

# later, when the user clicks thumbs-up:
client.create_feedback(run_id, key="user-thumbs", score=1, comment="helpful")

Comparing feedback by version

Aggregating raw feedback events per prompt version. Note how small these samples are.

from collections import defaultdict

feedback = [("v3", 1), ("v3", 0), ("v4", 1), ("v4", 1), ("v3", 1), ("v4", 0), ("v4", 1)]
stats = defaultdict(lambda: [0, 0])          # version -> [thumbs_up, total]
for version, thumbs_up in feedback:
    stats[version][0] += thumbs_up
    stats[version][1] += 1
for version, (up, n) in sorted(stats.items()):
    print(f"{version}: {up}/{n} positive ({up / n:.0%})")

Output:

v3: 2/3 positive (67%)
v4: 3/4 positive (75%)

Small samples lie

75% versus 67% from seven ratings means nothing; one extra click flips it. Wait for enough feedback, look at confidence intervals or use a proper experiment, and read the negative traces before declaring a winner.

Quick check: What must a feedback event include so it can be attached to the right answer?

• The ID of the trace or run that produced the answer
• The user's full chat history
• The model's API key
• The time the frontend was deployed