# Anatomy of a Distributed Trace — Logging, Metrics, Tracing & Alerting

Source: https://www.skillbyai.com/en/observability/ob-trace-anatomy

> Identify traces, spans, parent-child links, attributes and span status, and read a trace waterfall.

## Traces are trees of spans

A **trace** represents one end-to-end operation, usually a user request, identified by a 128-bit **trace ID**. It is made of **spans**. A span is one timed unit of work: an HTTP handler, a database query, a call to another service. Each span records a **span ID**, its **parent span ID** (empty for the root), a name, start and end timestamps, and a **status** (unset, ok or error).

Spans carry **attributes**, key-value details like `http.request.method`, `http.response.status_code` or `db.system`, and timestamped **events** such as "retry started" or an exception with its stack trace. The **span kind** says what role it plays: `SERVER` (handling an incoming request), `CLIENT` (making an outgoing call), `PRODUCER`/`CONSUMER` (messaging), or `INTERNAL`.

Because every span knows its parent, the backend can rebuild a tree and draw a **waterfall**: horizontal bars on a timeline. You can see at a glance which calls ran in sequence, which ran in parallel, and where the time went. Gaps between child spans often point to uninstrumented work.

## Spans across services

One call chain from a trace: a user **request** enters the **API gateway** span, which calls the **order service**, which calls the **payment service**, which in turn calls an external **UPI gateway**. Each arrow is a parent-child link carried by the same trace ID. (Sibling calls, such as the order service's database query, hang off the same parent in the full tree.)

![Five stages in order: a user request, the API gateway span, the order service, the payment service, and the external UPI gateway, each a child of the one before and all sharing one trace ID.](assets/figures/observability/section-6-map.svg) — Figure 6.1: request -> API gateway -> order service -> payment service -> UPI gateway

## Reading a waterfall

A text rendering of what a trace UI shows. The payment span accounts for most of the 410 ms and ends in error because its child call to the UPI gateway timed out. Database and external calls are CLIENT spans recorded by the caller, so their service column shows the peer they called.

```text
trace 4bf92f3577b34da6a3ce929d0e0e4736                   0ms                    410ms
POST /checkout            api-gateway     [server]       |===========================|   410ms
  auth.verify             auth-service    [server]       |=|                              32ms
  orders.create           order-service   [server]          |=======================|    350ms
    SELECT stock          postgres        [client]          |===|                         48ms
    payments.charge       payment-service [server]              |==================|     290ms  ERROR
      POST /v1/charge     upi-gateway     [client]              |==================|     280ms  timeout
```

## Tracking a courier parcel

A courier tracking page lists every hop of one parcel: picked up in Pune, sorted at the Mumbai hub, out for delivery in Thane, each with a timestamp. The AWB number ties all hops together like a trace ID, and each hop is a span. If the parcel sat at the hub for two days, the timeline shows exactly where the delay happened.

**Quiz:** What links a child span to the operation that caused it?

- [x] Its parent span ID, within the same trace ID
- [ ] The hostname of the server it ran on
- [ ] Its start timestamp being later than the parent's
- [ ] A shared log level

*Answer:* Its parent span ID, within the same trace ID. Each span stores the ID of its parent span, and all spans of a request share the trace ID. Timestamps alone cannot reliably reconstruct the tree.
