Success Metrics and Guardrail Metrics

Two kinds of numbers

Success metrics capture the value the feature should create: task completion, time saved, conversion, satisfaction. Guardrail metrics protect against harm: error rate, thumbs-down rate, escalations, safety flags, latency, cost per request, complaint volume. A launch should improve success metrics without breaching any guardrail. Define each metric precisely (numerator, denominator, time window), with a threshold and a source, and make sure you can measure it before day one.

Speed and seatbelts

A faster car is the goal; seatbelts, brakes and crash tests make sure the speed does not come at an unacceptable cost.

Instrument before launch

If a metric cannot be measured on the first day of dogfooding, it cannot protect the launch.

Quick check: Which is a guardrail metric for an AI feature?

• Office temperature
• Number of marketing emails sent
• Lines of code written
• Rate of safety flags per thousand responses