# Self-Check: Debugging a Slow Checkout — Logging, Metrics, Tracing & Alerting

Source: https://www.skillbyai.com/en/observability/ob-self-check-scenario

> Walk through a realistic incident using metrics, traces and logs in order, and check your reasoning at each step.

## A guided incident walk-through

At 19:40 the alert `CheckoutErrorBudgetFastBurn` pages you. Work through it as you would in real life.

**Step 1, confirm with metrics.** The RED panel shows request rate normal, 5xx ratio at 2.1%, and p99 latency up from 400 ms to 4 s. p50 is unchanged, so most requests are fine and a subset is very slow.

**Step 2, narrow down.** Splitting the error ratio by `route` shows only `POST /checkout`; splitting by `instance` shows all pods equally, so it is not one bad pod.

**Step 3, open traces.** An exemplar on the latency graph opens a trace: the `payments.charge` span takes 3.9 s and its child call to the UPI gateway ends with a timeout.

**Step 4, read logs.** Filtering payment-service logs by that `trace_id` shows `upstream timeout` with `gateway=upi`, and the same message for most failing traces.

**Step 5, mitigate.** Enable the feature flag that routes payments to the backup gateway, watch the burn rate fall, then write the postmortem.

## Why this order works

Metrics are cheap and cover all traffic, so they tell you *whether* and *how big*. Breakdowns tell you *where*. Traces tell you *which component*. Logs tell you *exactly what happened*. Starting with logs means searching a haystack without knowing which needle to look for.

## The queries behind each step

Steps 1 and 2 in PromQL, then step 4 as a LogQL search for the trace found in step 3.

```text
# step 1: error ratio and p99
sum(rate(http_requests_total{job="checkout",status=~"5.."}[5m])) / sum(rate(http_requests_total{job="checkout"}[5m]))
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{job="checkout"}[5m])))

# step 2: which route, which instance
sum by (route) (rate(http_requests_total{job="checkout",status=~"5.."}[5m]))
sum by (instance) (rate(http_requests_total{job="checkout",status=~"5.."}[5m]))

# step 4: logs for the trace from step 3
{app="payment-service"} | json | trace_id="a3c1e07d5b2f4e9a8c6d1f0b7e2a9c45"
```

**Quiz:** In the walk-through, p50 latency was unchanged while p99 rose tenfold. What does that suggest?

- [ ] Every request became ten times slower
- [ ] The metrics pipeline is broken
- [ ] The load balancer is dropping half the traffic
- [x] A subset of requests is very slow while the majority is unaffected

*Answer:* A subset of requests is very slow while the majority is unaffected. An unchanged median with a much larger tail means most requests are normal and a minority, here those hitting the UPI gateway timeout, are very slow.
