Lesson 16 / 35
Anatomy of a Distributed Trace
Identify traces, spans, parent-child links, attributes and span status, and read a trace waterfall.
By Rahul Agarwal · 15 years in web development
Traces are trees of spans
A trace represents one end-to-end operation, usually a user request, identified by a 128-bit trace ID. It is made of spans. A span is one timed unit of work: an HTTP handler, a database query, a call to another service. Each span records a span ID, its parent span ID (empty for the root), a name, start and end timestamps, and a status (unset, ok or error).
Spans carry attributes, key-value details like http.request.method, http.response.status_code or db.system, and timestamped events such as "retry started" or an exception with its stack trace. The span kind says what role it plays: SERVER (handling an incoming request), CLIENT (making an outgoing call), PRODUCER/CONSUMER (messaging), or INTERNAL.
Because every span knows its parent, the backend can rebuild a tree and draw a waterfall: horizontal bars on a timeline. You can see at a glance which calls ran in sequence, which ran in parallel, and where the time went. Gaps between child spans often point to uninstrumented work.
Spans across services
One call chain from a trace: a user request enters the API gateway span, which calls the order service, which calls the payment service, which in turn calls an external UPI gateway. Each arrow is a parent-child link carried by the same trace ID. (Sibling calls, such as the order service's database query, hang off the same parent in the full tree.)
Reading a waterfall
A text rendering of what a trace UI shows. The payment span accounts for most of the 410 ms and ends in error because its child call to the UPI gateway timed out. Database and external calls are CLIENT spans recorded by the caller, so their service column shows the peer they called.
trace 4bf92f3577b34da6a3ce929d0e0e4736 0ms 410ms
POST /checkout api-gateway [server] |===========================| 410ms
auth.verify auth-service [server] |=| 32ms
orders.create order-service [server] |=======================| 350ms
SELECT stock postgres [client] |===| 48ms
payments.charge payment-service [server] |==================| 290ms ERROR
POST /v1/charge upi-gateway [client] |==================| 280ms timeoutTracking a courier parcel
A courier tracking page lists every hop of one parcel: picked up in Pune, sorted at the Mumbai hub, out for delivery in Thane, each with a timestamp. The AWB number ties all hops together like a trace ID, and each hop is a span. If the parcel sat at the hub for two days, the timeline shows exactly where the delay happened.
Quick check: What links a child span to the operation that caused it?
- Its parent span ID, within the same trace ID
- The hostname of the server it ran on
- Its start timestamp being later than the parent's
- A shared log level
Answer
Its parent span ID, within the same trace ID — Each span stores the ID of its parent span, and all spans of a request share the trace ID. Timestamps alone cannot reliably reconstruct the tree.