पाठ 8 / 32
Queues and Streams as Triggers
यह पाठ अभी हिंदी में अनुवादित नहीं है, इसलिए अंग्रेज़ी में दिखाया जा रहा है।
Use a queue to decouple producers from functions, process batches safely, and report partial batch failures.
लेखक: Rahul Agarwal · 15 साल से वेब डेवलपर
Queues decouple; streams preserve order
Putting a queue such as Amazon SQS between a producer and a function decouples them: the producer finishes quickly, the queue absorbs bursts, and failed messages are retried later. With an event source mapping, Lambda polls the queue and invokes your function with a batch of messages. A received message is hidden from other consumers for the visibility timeout; if processing does not succeed in time, it reappears and is retried. That is why AWS recommends a visibility timeout of at least six times the function timeout. If one message in a batch fails and the handler throws, the whole batch returns to the queue and the successful messages are processed again. Enabling ReportBatchItemFailures and returning only the failed message IDs fixes that. Streams (Kinesis, DynamoDB Streams, Kafka) keep order within a shard or partition and let several consumers replay data, at the cost of head-of-line blocking when one record keeps failing.
From event source to downstream work
A producer emits an event, a trigger or queue holds it, the function processes it, and the result lands in a downstream store or service.
Reporting partial batch failures
Only message m2 is retried; m1 and m3 are deleted from the queue. The first output line is the error logged to stderr.
// SQS-triggered handler with partial batch failure reporting
// (needs ReportBatchItemFailures enabled on the event source mapping)
async function processOrder(order) {
if (order.amount <= 0) throw new Error(`bad amount for ${order.id}`);
}
export const handler = async (event) => {
const batchItemFailures = [];
for (const record of event.Records) {
try {
await processOrder(JSON.parse(record.body));
} catch (err) {
console.error(err.message);
batchItemFailures.push({ itemIdentifier: record.messageId });
}
}
return { batchItemFailures };
};
const event = { Records: [
{ messageId: 'm1', body: '{"id":"o1","amount":499}' },
{ messageId: 'm2', body: '{"id":"o2","amount":0}' },
{ messageId: 'm3', body: '{"id":"o3","amount":120}' },
] };
console.log(JSON.stringify(await handler(event)));
Output:
bad amount for o2
{"batchItemFailures":[{"itemIdentifier":"m2"}]}The OPD token counter
A hospital OPD gives patients tokens so doctors are not mobbed at the door. Doctors call patients at their own pace, and a patient sent for a test rejoins the queue instead of blocking everyone. A message queue does the same for your function during a traffic spike.
त्वरित जाँच: Without partial batch reporting, what happens when one message in an SQS batch of ten makes the handler throw?
- Only the failed message is retried
- All ten messages become visible again and are retried
- The failed message is deleted and the other nine are retried
- The queue is paused until an operator intervenes
Answer
All ten messages become visible again and are retried — A thrown error fails the whole batch, so successful messages are reprocessed too. Returning `batchItemFailures` limits the retry to the failed ones.