[{"data":1,"prerenderedAt":350},["ShallowReactive",2],{"post-\u002Fwriting\u002Fkafka-consumer-crash-loop":3,"toc-\u002Fwriting\u002Fkafka-consumer-crash-loop":345},{"id":4,"title":5,"body":6,"date":331,"description":332,"draft":333,"extension":334,"meta":335,"navigation":234,"path":336,"readingTime":337,"seo":338,"stem":339,"tags":340,"__hash__":344},"writing\u002Fwriting\u002Fkafka-consumer-crash-loop.md","The consumer that wouldn't stay up",{"type":7,"value":8,"toc":324},"minimark",[9,13,16,21,24,27,30,33,42,48,58,74,85,94,98,101,111,269,280,286,301,305,308,314,320],[10,11,12],"p",{},"A consumer group starts crashing. It restarts, processes for a few seconds, and crashes again.\nLag climbs. Restarting the pods doesn't help — which is the first genuinely useful piece of\ninformation, because it rules out most of what you'd normally suspect.",[10,14,15],{},"This is a note about what that failure actually is, and why the fix isn't where people usually\nreach first.",[17,18,20],"h2",{"id":19},"the-shape-of-the-problem","The shape of the problem",[10,22,23],{},"A Kafka consumer commits offsets to record how far it has read. On restart, it resumes from the\nlast committed offset. That's the whole point — it's what makes consumers restartable.",[10,25,26],{},"Now suppose a message arrives that the consumer can't deserialise. Maybe a producer shipped a\nschema change. Maybe a field that was never null is null. The deserialiser throws, the exception\npropagates, the consumer dies before committing.",[10,28,29],{},"It restarts, resumes from the last committed offset, reads the same message, and dies again.",[10,31,32],{},"That's the loop. Nothing is flapping, nothing is under-resourced, and no amount of restarting\nhelps — the consumer is behaving exactly as designed. It's just that \"resume where you left off\"\nand \"you cannot get past this message\" combine into a system that makes no progress forever.",[17,34,36,37,41],{"id":35},"why-autooffsetreset-gets-blamed","Why ",[38,39,40],"code",{},"auto.offset.reset"," gets blamed",[10,43,44,45,47],{},"Somebody always suggests changing ",[38,46,40],{},". It's worth being precise about what that\nsetting does, because the confusion is the interesting part.",[10,49,50,52,53,57],{},[38,51,40],{}," only applies when there is ",[54,55,56],"strong",{},"no valid committed offset"," — a brand new consumer\ngroup, or a committed offset that has aged out of the topic's retention. It has two useful values:",[59,60,61,68],"ul",{},[62,63,64,67],"li",{},[38,65,66],{},"earliest"," — start from the beginning of the topic",[62,69,70,73],{},[38,71,72],{},"latest"," — start from the next message produced",[10,75,76,77,79,80,84],{},"Set it to ",[38,78,66],{}," and a consumer group with no committed offset replays the whole topic from the\nstart. If a poison message sits anywhere in that history, you've guaranteed you hit it — and if the\ncrash also prevents the offset commit, the group never establishes a valid offset, so the ",[81,82,83],"em",{},"next","\nrestart takes the reset path again and replays from the beginning again.",[10,86,87,88,90,91],{},"That's the version of this bug that looks like ",[38,89,40],{}," is at fault. It isn't. It's\namplifying the real problem, which is that ",[54,92,93],{},"a deserialisation failure is being treated as fatal.",[17,95,97],{"id":96},"the-actual-fix","The actual fix",[10,99,100],{},"Three changes, in order of how much they matter.",[10,102,103,106,107,110],{},[54,104,105],{},"1. Make deserialisation failures survivable."," A message the consumer can't parse should be\nrouted somewhere and skipped, not allowed to kill the poll loop. In Kafka this is what an\n",[38,108,109],{},"ErrorHandlingDeserializer"," and a dead-letter topic are for:",[112,113,118],"pre",{"className":114,"code":115,"language":116,"meta":117,"style":117},"language-java shiki shiki-themes github-dark-dimmed","@Bean\npublic DefaultErrorHandler errorHandler(KafkaTemplate\u003CString, String> template) {\n    \u002F\u002F after retries are exhausted, publish to \u003Ctopic>.DLT and move on\n    var recoverer = new DeadLetterPublishingRecoverer(template);\n    var handler = new DefaultErrorHandler(recoverer, new FixedBackOff(1_000L, 2));\n\n    \u002F\u002F never retry what will never succeed — a bad payload is bad on attempt three too\n    handler.addNotRetryableExceptions(DeserializationException.class);\n    return handler;\n}\n","java","",[38,119,120,133,161,168,189,229,236,242,254,263],{"__ignoreMap":117},[121,122,125,129],"span",{"class":123,"line":124},"line",1,[121,126,128],{"class":127},"s1ain","@",[121,130,132],{"class":131},"scLw-","Bean\n",[121,134,136,139,142,146,149,152,155,158],{"class":123,"line":135},2,[121,137,138],{"class":131},"public",[121,140,141],{"class":127}," DefaultErrorHandler ",[121,143,145],{"class":144},"sLyBy","errorHandler",[121,147,148],{"class":127},"(KafkaTemplate",[121,150,151],{"class":131},"\u003C",[121,153,154],{"class":127},"String, String",[121,156,157],{"class":131},">",[121,159,160],{"class":127}," template) {\n",[121,162,164],{"class":123,"line":163},3,[121,165,167],{"class":166},"sVrTk","    \u002F\u002F after retries are exhausted, publish to \u003Ctopic>.DLT and move on\n",[121,169,171,174,177,180,183,186],{"class":123,"line":170},4,[121,172,173],{"class":131},"    var",[121,175,176],{"class":127}," recoverer",[121,178,179],{"class":131}," =",[121,181,182],{"class":131}," new",[121,184,185],{"class":144}," DeadLetterPublishingRecoverer",[121,187,188],{"class":127},"(template);\n",[121,190,192,194,197,199,201,204,207,210,213,216,220,223,226],{"class":123,"line":191},5,[121,193,173],{"class":131},[121,195,196],{"class":127}," handler",[121,198,179],{"class":131},[121,200,182],{"class":131},[121,202,203],{"class":144}," DefaultErrorHandler",[121,205,206],{"class":127},"(recoverer, ",[121,208,209],{"class":131},"new",[121,211,212],{"class":144}," FixedBackOff",[121,214,215],{"class":127},"(",[121,217,219],{"class":218},"sDGpf","1_000L",[121,221,222],{"class":127},", ",[121,224,225],{"class":218},"2",[121,227,228],{"class":127},"));\n",[121,230,232],{"class":123,"line":231},6,[121,233,235],{"emptyLinePlaceholder":234},true,"\n",[121,237,239],{"class":123,"line":238},7,[121,240,241],{"class":166},"    \u002F\u002F never retry what will never succeed — a bad payload is bad on attempt three too\n",[121,243,245,248,251],{"class":123,"line":244},8,[121,246,247],{"class":127},"    handler.",[121,249,250],{"class":144},"addNotRetryableExceptions",[121,252,253],{"class":127},"(DeserializationException.class);\n",[121,255,257,260],{"class":123,"line":256},9,[121,258,259],{"class":131},"    return",[121,261,262],{"class":127}," handler;\n",[121,264,266],{"class":123,"line":265},10,[121,267,268],{"class":127},"}\n",[10,270,271,272,275,276,279],{},"The distinction that matters: ",[54,273,274],{},"retryable"," failures (the database was briefly unavailable) should\nback off and retry. ",[54,277,278],{},"Non-retryable"," failures (this payload will never parse) should go straight\nto the dead-letter topic. Retrying a malformed message is just a slower crash loop.",[10,281,282,285],{},[54,283,284],{},"2. Commit deliberately."," Auto-commit makes the failure boundary fuzzy — you can't easily reason\nabout what was processed versus what was acknowledged. Manual acknowledgement after successful\nprocessing makes \"did this message count?\" a question with an answer.",[10,287,288,294,295,297,298,300],{},[54,289,290,291,293],{},"3. Then think about ",[38,292,40],{},"."," With the first two in place, this becomes what it\nshould have been all along: a policy decision about cold starts, not a lever for incident response.\nFor a consumer where missing historical data is worse than reprocessing it, ",[38,296,66],{}," is right.\nFor one where stale events are useless, ",[38,299,72],{}," is right. Neither choice can produce a crash loop\nonce bad payloads have somewhere to go.",[17,302,304],{"id":303},"what-to-keep","What to keep",[10,306,307],{},"Two things worth carrying forward.",[10,309,310,313],{},[54,311,312],{},"A poison message is a design gap, not an anomaly."," Any consumer without an explicit answer for\n\"what happens when this message can't be parsed\" has a crash loop waiting for the right input.",[10,315,316,319],{},[54,317,318],{},"Watch consumer lag, not consumer health."," A crash-looping consumer often looks fine to a\nliveness probe — it starts, it's alive, it serves the health endpoint, it dies, it starts again.\nLag is what tells you it isn't actually getting anywhere.",[321,322,323],"style",{},"html pre.shiki code .s1ain, html code.shiki .s1ain{--shiki-default:#ADBAC7}html pre.shiki code .scLw-, html code.shiki .scLw-{--shiki-default:#F47067}html pre.shiki code .sLyBy, html code.shiki .sLyBy{--shiki-default:#DCBDFB}html pre.shiki code .sVrTk, html code.shiki .sVrTk{--shiki-default:#768390}html pre.shiki code .sDGpf, html code.shiki .sDGpf{--shiki-default:#6CB6FF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}",{"title":117,"searchDepth":135,"depth":135,"links":325},[326,327,329,330],{"id":19,"depth":135,"text":20},{"id":35,"depth":135,"text":328},"Why auto.offset.reset gets blamed",{"id":96,"depth":135,"text":97},{"id":303,"depth":135,"text":304},"2026-05-18","A Kafka consumer group stuck in a restart loop, one malformed message, and why auto.offset.reset is the wrong place to look for a fix.",false,"md",{},"\u002Fwriting\u002Fkafka-consumer-crash-loop","6 min read",{"title":5,"description":332},"writing\u002Fkafka-consumer-crash-loop",[341,342,343],"Kafka","Debugging","Streaming","76xbGl3Lhx5ldvYgIRWRhc0RstGKhpWBf2nyQWrdd8E",[346,347,348,349],{"id":19,"depth":135,"text":20},{"id":35,"depth":135,"text":328},{"id":96,"depth":135,"text":97},{"id":303,"depth":135,"text":304},1785591010577]