# Queues and Streams as Triggers — Serverless & Edge

Source: https://www.skillbyai.com/en/serverless-edge/se-queues-streams

> Use a queue to decouple producers from functions, process batches safely, and report partial batch failures.

## Queues decouple; streams preserve order

Putting a **queue** such as Amazon SQS between a producer and a function decouples them: the producer finishes quickly, the queue absorbs bursts, and failed messages are retried later. With an event source mapping, Lambda polls the queue and invokes your function with a **batch** of messages. A received message is hidden from other consumers for the **visibility timeout**; if processing does not succeed in time, it reappears and is retried. That is why AWS recommends a visibility timeout of at least six times the function timeout. If one message in a batch fails and the handler throws, the *whole* batch returns to the queue and the successful messages are processed again. Enabling **ReportBatchItemFailures** and returning only the failed message IDs fixes that. **Streams** (Kinesis, DynamoDB Streams, Kafka) keep order within a shard or partition and let several consumers replay data, at the cost of head-of-line blocking when one record keeps failing.

## From event source to downstream work

A producer emits an event, a trigger or queue holds it, the function processes it, and the result lands in a downstream store or service.

![Four connected stages from left to right: an event source, a trigger or queue, the function, and a downstream service.](assets/figures/serverless-edge/section-3-map.svg) — Figure 3.1: event source -> trigger or queue -> function -> downstream service

## Reporting partial batch failures

Only message `m2` is retried; `m1` and `m3` are deleted from the queue. The first output line is the error logged to stderr.

```js
// SQS-triggered handler with partial batch failure reporting
// (needs ReportBatchItemFailures enabled on the event source mapping)
async function processOrder(order) {
  if (order.amount <= 0) throw new Error(`bad amount for ${order.id}`);
}

export const handler = async (event) => {
  const batchItemFailures = [];
  for (const record of event.Records) {
    try {
      await processOrder(JSON.parse(record.body));
    } catch (err) {
      console.error(err.message);
      batchItemFailures.push({ itemIdentifier: record.messageId });
    }
  }
  return { batchItemFailures };
};

const event = { Records: [
  { messageId: 'm1', body: '{"id":"o1","amount":499}' },
  { messageId: 'm2', body: '{"id":"o2","amount":0}' },
  { messageId: 'm3', body: '{"id":"o3","amount":120}' },
] };
console.log(JSON.stringify(await handler(event)));
```

Output:

```
bad amount for o2
{"batchItemFailures":[{"itemIdentifier":"m2"}]}
```

## The OPD token counter

A hospital OPD gives patients tokens so doctors are not mobbed at the door. Doctors call patients at their own pace, and a patient sent for a test rejoins the queue instead of blocking everyone. A message queue does the same for your function during a traffic spike.

**Quiz:** Without partial batch reporting, what happens when one message in an SQS batch of ten makes the handler throw?

- [ ] Only the failed message is retried
- [x] All ten messages become visible again and are retried
- [ ] The failed message is deleted and the other nine are retried
- [ ] The queue is paused until an operator intervenes

*Answer:* All ten messages become visible again and are retried. A thrown error fails the whole batch, so successful messages are reprocessed too. Returning `batchItemFailures` limits the retry to the failed ones.
