Instrumenting an Application

Measure requests, errors and duration

Official and community client libraries exist for Go, Java, Python, Ruby, .NET, Node.js and more. Define metrics once at start-up, update them in code paths, and expose them on /metrics. For every service, start with the RED metrics: request Rate, Errors and Duration, labelled by a bounded route template (/orders/{id}, not the actual ID). Many frameworks have middleware that does this automatically, and OpenTelemetry can export metrics to Prometheus too.

RED metrics in Python

Using the prometheus_client library (a sketch; adapt to your framework).

from prometheus_client import Counter, Histogram, start_http_server
import time

REQUESTS = Counter("http_requests_total", "HTTP requests", ["route", "method", "status"])
LATENCY = Histogram("http_request_duration_seconds", "Request latency", ["route"],
                    buckets=[0.05, 0.1, 0.25, 0.3, 0.5, 1, 2.5])

def handle(route_template, method, handler):
    start = time.perf_counter()
    status = "500"
    try:
        status = str(handler())
        return status
    finally:
        REQUESTS.labels(route_template, method, status).inc()
        LATENCY.labels(route_template).observe(time.perf_counter() - start)

start_http_server(8000)   # serves /metrics on port 8000

Use route templates, never raw paths

Labelling by /orders/12345 creates a new series per order and can overwhelm Prometheus.

Quick check: What do the RED metrics stand for?

• Requests, Exports, Disks
• Reads, Edits, Deletes
• Red, Easy, Done
• Rate, Errors, Duration