The consumer that wouldn't stay up

A consumer group starts crashing. It restarts, processes for a few seconds, and crashes again. Lag climbs. Restarting the pods doesn't help — which is the first genuinely useful piece of information, because it rules out most of what you'd normally suspect.

This is a note about what that failure actually is, and why the fix isn't where people usually reach first.

The shape of the problem

A Kafka consumer commits offsets to record how far it has read. On restart, it resumes from the last committed offset. That's the whole point — it's what makes consumers restartable.

Now suppose a message arrives that the consumer can't deserialise. Maybe a producer shipped a schema change. Maybe a field that was never null is null. The deserialiser throws, the exception propagates, the consumer dies before committing.

It restarts, resumes from the last committed offset, reads the same message, and dies again.

That's the loop. Nothing is flapping, nothing is under-resourced, and no amount of restarting helps — the consumer is behaving exactly as designed. It's just that "resume where you left off" and "you cannot get past this message" combine into a system that makes no progress forever.

Why auto.offset.reset gets blamed

Somebody always suggests changing auto.offset.reset. It's worth being precise about what that setting does, because the confusion is the interesting part.

auto.offset.reset only applies when there is no valid committed offset — a brand new consumer group, or a committed offset that has aged out of the topic's retention. It has two useful values:

• earliest — start from the beginning of the topic
• latest — start from the next message produced

Set it to earliest and a consumer group with no committed offset replays the whole topic from the start. If a poison message sits anywhere in that history, you've guaranteed you hit it — and if the crash also prevents the offset commit, the group never establishes a valid offset, so the next restart takes the reset path again and replays from the beginning again.

That's the version of this bug that looks like auto.offset.reset is at fault. It isn't. It's amplifying the real problem, which is that a deserialisation failure is being treated as fatal.

The actual fix

Three changes, in order of how much they matter.

1. Make deserialisation failures survivable. A message the consumer can't parse should be routed somewhere and skipped, not allowed to kill the poll loop. In Kafka this is what an ErrorHandlingDeserializer and a dead-letter topic are for:

@Bean
public DefaultErrorHandler errorHandler(KafkaTemplate<String, String> template) {
    // after retries are exhausted, publish to <topic>.DLT and move on
    var recoverer = new DeadLetterPublishingRecoverer(template);
    var handler = new DefaultErrorHandler(recoverer, new FixedBackOff(1_000L, 2));

    // never retry what will never succeed — a bad payload is bad on attempt three too
    handler.addNotRetryableExceptions(DeserializationException.class);
    return handler;
}

The distinction that matters: retryable failures (the database was briefly unavailable) should back off and retry. Non-retryable failures (this payload will never parse) should go straight to the dead-letter topic. Retrying a malformed message is just a slower crash loop.

2. Commit deliberately. Auto-commit makes the failure boundary fuzzy — you can't easily reason about what was processed versus what was acknowledged. Manual acknowledgement after successful processing makes "did this message count?" a question with an answer.

3. Then think about auto.offset.reset. With the first two in place, this becomes what it should have been all along: a policy decision about cold starts, not a lever for incident response. For a consumer where missing historical data is worse than reprocessing it, earliest is right. For one where stale events are useless, latest is right. Neither choice can produce a crash loop once bad payloads have somewhere to go.

What to keep

Two things worth carrying forward.

A poison message is a design gap, not an anomaly. Any consumer without an explicit answer for "what happens when this message can't be parsed" has a crash loop waiting for the right input.

Watch consumer lag, not consumer health. A crash-looping consumer often looks fine to a liveness probe — it starts, it's alive, it serves the health endpoint, it dies, it starts again. Lag is what tells you it isn't actually getting anywhere.