Lesson 11 / 32

Idempotency and At-Least-Once Delivery

Explain why serverless events can arrive twice and make side-effecting handlers idempotent with keys and conditional writes.

By · 15 years in web development

At-least-once means 'sometimes twice'

Most serverless event sources promise at-least-once delivery: an event is not silently lost, but it may be delivered more than once. Retries after timeouts, expired queue visibility and network glitches all create duplicates. A handler is idempotent if processing the same event twice has the same effect as processing it once. Some operations are naturally idempotent, such as setting an order's status to shipped or a PUT that overwrites a record. Others are not: charging a card, sending an SMS, incrementing a counter. The standard fix is an idempotency key: a unique ID for each logical operation, such as an order ID, a key generated by the client, or the message ID. Before acting, the function records the key in a durable store using a conditional write that fails if the key already exists; on a duplicate it returns the saved result instead of acting again. The check and the store must be atomic, or two parallel retries can both slip through.

Making a side effect happen once

Every event, including a duplicate, first passes an idempotency check against a durable store. Only the first event with a given key reaches the side effect.

Three connected stages from left to right: an incoming event that may be a duplicate, an idempotency check against stored keys, and a side effect that runs once.
Figure 4.1: event (maybe duplicate) -> idempotency check -> side effect runs once

Replay the saved result instead of charging twice

The Map stands in for a durable table. Notice that "charging" is printed only once even though the handler ran twice.

// Idempotent payment handler: the same key never charges twice.
const processed = new Map(); // in production: a DynamoDB table or similar

async function charge(amount) {
  console.log(`charging Rs ${amount}`);
  return { chargeId: `ch_${amount}` };
}

export async function handler(event) {
  const key = event.idempotencyKey;
  if (processed.has(key)) return processed.get(key); // replay saved result
  const result = await charge(event.amount);
  processed.set(key, result);
  return result;
}

const evt = { idempotencyKey: 'order-42', amount: 999 };
console.log(await handler(evt));
console.log(await handler(evt)); // retry of the same event

Output:

charging Rs 999
{ chargeId: 'ch_999' }
{ chargeId: 'ch_999' }

Do it properly in production

Store keys with a conditional write, for example DynamoDB PutItem with ConditionExpression: 'attribute_not_exists(pk)', and give records a TTL so the table does not grow forever. Libraries such as Powertools for AWS Lambda include an idempotency utility that handles in-progress states and expiry for you.

Quick check: Which operation is naturally idempotent?

  • Sending a welcome email
  • Adding 100 reward points to a user
  • Setting a ticket's status to "closed"
  • Appending a row to an audit log
Answer

Setting a ticket's status to "closed" — Setting a value to a fixed state gives the same result however many times it runs. The others repeat their effect on every execution.