Common Pitfalls

Ways event-driven systems go wrong

Event soup: hundreds of loosely named events with no owners or catalogue, so nobody knows what triggers what. Distributed monolith: services that must be deployed together because they share event schemas they all change at once, or that wait synchronously on each other's events. Lost events: dual writes without an outbox. Duplicate side effects: non-idempotent consumers under at-least-once delivery. Ordering bugs: wrong partition keys or retries that reorder updates. Unbounded retries: a poison message blocking a partition forever. Hidden coupling through payloads: consumers depending on fields the producer considers internal. Using events for queries: request-reply over a broker where a simple API call would do. Missing observability: no correlation IDs, no lag alerts. Most of these are prevented by the practices in this course: clear contracts, outbox, idempotency, keys, DLQs, tracing and a few well-chosen synchronous calls.

A pre-production review checklist

Run through it for every new event flow.

[ ] event names are past-tense facts with an owner and a schema in the registry
[ ] events are published via outbox or CDC, not dual writes
[ ] every consumer is idempotent (dedupe table / natural idempotency / keys)
[ ] partition key chosen for the ordering we need
[ ] retries with backoff, max attempts, DLQ with alerting and a replay tool
[ ] correlation id + traceparent propagated; lag and DLQ dashboards exist
[ ] UI handles eventual consistency explicitly
[ ] compensations defined and idempotent for every saga step
[ ] personal data minimised in payloads; retention set on topics
[ ] AsyncAPI / catalogue entry updated

A busy railway junction

Without a timetable (catalogue), signals (monitoring) and rules about which track each train uses (keys and ordering), more trains just mean more collisions.

Quick check: Several services must always be deployed together because they change shared event schemas simultaneously. What is this called?

• Event sourcing
• A distributed monolith
• CQRS level 1
• Choreography