पाठ 34 / 35
Self-Check: Debugging a Slow Checkout
यह पाठ अभी हिंदी में अनुवादित नहीं है, इसलिए अंग्रेज़ी में दिखाया जा रहा है।
Walk through a realistic incident using metrics, traces and logs in order, and check your reasoning at each step.
लेखक: Rahul Agarwal · 15 साल से वेब डेवलपर
A guided incident walk-through
At 19:40 the alert CheckoutErrorBudgetFastBurn pages you. Work through it as you would in real life.
Step 1, confirm with metrics. The RED panel shows request rate normal, 5xx ratio at 2.1%, and p99 latency up from 400 ms to 4 s. p50 is unchanged, so most requests are fine and a subset is very slow.
Step 2, narrow down. Splitting the error ratio by route shows only POST /checkout; splitting by instance shows all pods equally, so it is not one bad pod.
Step 3, open traces. An exemplar on the latency graph opens a trace: the payments.charge span takes 3.9 s and its child call to the UPI gateway ends with a timeout.
Step 4, read logs. Filtering payment-service logs by that trace_id shows upstream timeout with gateway=upi, and the same message for most failing traces.
Step 5, mitigate. Enable the feature flag that routes payments to the backup gateway, watch the burn rate fall, then write the postmortem.
Why this order works
Metrics are cheap and cover all traffic, so they tell you whether and how big. Breakdowns tell you where. Traces tell you which component. Logs tell you exactly what happened. Starting with logs means searching a haystack without knowing which needle to look for.
The queries behind each step
Steps 1 and 2 in PromQL, then step 4 as a LogQL search for the trace found in step 3.
# step 1: error ratio and p99
sum(rate(http_requests_total{job="checkout",status=~"5.."}[5m])) / sum(rate(http_requests_total{job="checkout"}[5m]))
histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{job="checkout"}[5m])))
# step 2: which route, which instance
sum by (route) (rate(http_requests_total{job="checkout",status=~"5.."}[5m]))
sum by (instance) (rate(http_requests_total{job="checkout",status=~"5.."}[5m]))
# step 4: logs for the trace from step 3
{app="payment-service"} | json | trace_id="a3c1e07d5b2f4e9a8c6d1f0b7e2a9c45"त्वरित जाँच: In the walk-through, p50 latency was unchanged while p99 rose tenfold. What does that suggest?
- Every request became ten times slower
- The metrics pipeline is broken
- The load balancer is dropping half the traffic
- A subset of requests is very slow while the majority is unaffected
Answer
A subset of requests is very slow while the majority is unaffected — An unchanged median with a much larger tail means most requests are normal and a minority, here those hitting the UPI gateway timeout, are very slow.