Lesson 16 / 35

Anatomy of a Distributed Trace

Identify traces, spans, parent-child links, attributes and span status, and read a trace waterfall.

By · 15 years in web development

Traces are trees of spans

A trace represents one end-to-end operation, usually a user request, identified by a 128-bit trace ID. It is made of spans. A span is one timed unit of work: an HTTP handler, a database query, a call to another service. Each span records a span ID, its parent span ID (empty for the root), a name, start and end timestamps, and a status (unset, ok or error).

Spans carry attributes, key-value details like http.request.method, http.response.status_code or db.system, and timestamped events such as "retry started" or an exception with its stack trace. The span kind says what role it plays: SERVER (handling an incoming request), CLIENT (making an outgoing call), PRODUCER/CONSUMER (messaging), or INTERNAL.

Because every span knows its parent, the backend can rebuild a tree and draw a waterfall: horizontal bars on a timeline. You can see at a glance which calls ran in sequence, which ran in parallel, and where the time went. Gaps between child spans often point to uninstrumented work.

Spans across services

One call chain from a trace: a user request enters the API gateway span, which calls the order service, which calls the payment service, which in turn calls an external UPI gateway. Each arrow is a parent-child link carried by the same trace ID. (Sibling calls, such as the order service's database query, hang off the same parent in the full tree.)

Five stages in order: a user request, the API gateway span, the order service, the payment service, and the external UPI gateway, each a child of the one before and all sharing one trace ID.
Figure 6.1: request -> API gateway -> order service -> payment service -> UPI gateway

Reading a waterfall

A text rendering of what a trace UI shows. The payment span accounts for most of the 410 ms and ends in error because its child call to the UPI gateway timed out. Database and external calls are CLIENT spans recorded by the caller, so their service column shows the peer they called.

trace 4bf92f3577b34da6a3ce929d0e0e4736                   0ms                    410ms
POST /checkout            api-gateway     [server]       |===========================|   410ms
  auth.verify             auth-service    [server]       |=|                              32ms
  orders.create           order-service   [server]          |=======================|    350ms
    SELECT stock          postgres        [client]          |===|                         48ms
    payments.charge       payment-service [server]              |==================|     290ms  ERROR
      POST /v1/charge     upi-gateway     [client]              |==================|     280ms  timeout

Tracking a courier parcel

A courier tracking page lists every hop of one parcel: picked up in Pune, sorted at the Mumbai hub, out for delivery in Thane, each with a timestamp. The AWB number ties all hops together like a trace ID, and each hop is a span. If the parcel sat at the hub for two days, the timeline shows exactly where the delay happened.

Quick check: What links a child span to the operation that caused it?

  • Its parent span ID, within the same trace ID
  • The hostname of the server it ran on
  • Its start timestamp being later than the parent's
  • A shared log level
Answer

Its parent span ID, within the same trace ID — Each span stores the ID of its parent span, and all spans of a request share the trace ID. Timestamps alone cannot reliably reconstruct the tree.