A Lambda that forwards DynamoDB Streams records to Kafka started failing in production.
We had not deployed any application or infrastructure changes. Some invocations were still completing successfully, but enough were failing to trigger an alert:
KafkaTimeoutError: Failed to update metadata after 60.0 secs.Before sending a record, the Kafka producer needs information about the topic and its partitions to know where the record should go. This call was failing while waiting for that information.
The next step was to check how widely this was happening.
Several invocations were timing out while connecting to the same broker, but other invocations were still writing successfully. Comparing the failed invocations with the successful ones helped narrow the investigation. We were looking at a problem affecting part of the traffic rather than a complete Kafka outage.
We traced the timeouts to a Kafka maintenance operation. During that process, leadership for the affected partitions moved to another broker. Some invocations were already writing successfully after the transition, while the affected producers were still reconnecting and refreshing their metadata.
The recovery timing was useful too. The logs showed affected producers recovering only a few seconds after logging the timeout error.
Together, those observations explained why waiting longer was a reasonable response. We had found a temporary interruption during maintenance, and we could see recovery happening. There was no network rule or routing configuration to fix for this incident.
The application was simply giving up before the producer finished recovering.
Two possible approaches
We considered two options.
1. Fail the invocation and allow Lambda to retry the batch
Since our handler created a producer for each invocation, the next attempt would start with a new producer.
That sounded attractive: fail, start again, and get fresh client state.
With whole-batch retries, consider a batch of 50 records. If record 20 failed, records 1 through 19 might already have been written to Kafka. Reprocessing the batch could cause those records to be written again.
Handling duplicate deliveries would require broader changes than adjusting the metadata wait. That concern already existed: DynamoDB Streams event source mappings provide at-least-once processing. A longer wait would not remove the need to handle duplicates.
2. Keep the invocation running and try the write again
The other proposal was to give Kafka some time to finish the transition and then call send() again:
try:
producer.send(...)
except KafkaTimeoutError:
time.sleep(SLEEP_TIME)
producer.send(...)That also sounded reasonable, but it raised a few practical questions.
How long should we sleep? Across many invocations, this would add execution time and cost. It also leaves less time to process the rest of each batch.
We also needed to distinguish a temporary failure worth waiting for from an error that should be raised. The investigation had already shown that the producer could recover. Before adding another retry loop, we needed to understand what it was already doing during those first 60 seconds.
What does retry mean in this case?
The Lambda creates one kafka-python KafkaProducer for each invocation and uses it for every record in the batch. That producer manages the connections to Kafka in a background thread and keeps track of the metadata needed to route records to the right broker.
Calling send() reuses the same producer. It does not create a new client or force a fresh connection. Reconnecting and refreshing metadata remain the responsibility of the producer.
Before send() can queue a record for delivery, the producer needs metadata for its topic. If that information is unavailable, the call waits while the producer tries to obtain it. Calling send() again would wait for that same recovery process. The application would effectively do this:
send(record)
↓
wait up to 60 seconds for metadata
↓
timeout
sleep X seconds
send(record)
↓
wait up to 60 seconds for metadataA second send() could succeed because the producer had more time to reconnect or obtain updated metadata. Both calls would rely on the same producer to manage recovery. The main benefit was more time for the same recovery process. Instead of catching the timeout, sleeping, and calling send() again, we could extend the existing metadata wait:
KafkaProducer(
max_block_ms=120000,
# existing configuration
)We increased max_block_ms from 60 to 120 seconds. This allowed the metadata wait to continue for up to 120 seconds without forcing every call to wait that long. If metadata became available after 63 seconds, the call could continue then.
This setting bounds how long send() can block waiting for metadata or buffer space; it is not a delivery acknowledgment timeout. The handler still needs to check delivery results before reporting success. The longer wait also needs to fit within the Lambda invocation timeout, with time left to finish the batch and confirm pending sends.
That matched what we had observed: affected producers were recovering shortly after the original 60-second boundary.
The difference may look small, but it better represented what the function needed: to give the recovery process more time to finish.
Retry after understanding recovery
Retrying was a reasonable response to this incident. Kafka was going through maintenance, and the logs showed that the affected producers could recover.
The decision was where and how to allow more time.
A few specific questions helped us make that decision:
-
Where had the operation stopped?
-
How much of the system was affected?
-
What recovery was already happening?
-
What would actually be different on the next attempt?
The solution itself was simple: wait longer.
But even this simple solution leaves an important design question for the system:
What is the right way to wait?