[{"data":1,"prerenderedAt":346},["ShallowReactive",2],{"writing-all":3},[4],{"id":5,"title":6,"body":7,"date":332,"description":333,"draft":334,"extension":335,"meta":336,"navigation":235,"path":337,"readingTime":338,"seo":339,"stem":340,"tags":341,"__hash__":345},"writing\u002Fwriting\u002Fkafka-consumer-crash-loop.md","The consumer that wouldn't stay up",{"type":8,"value":9,"toc":325},"minimark",[10,14,17,22,25,28,31,34,43,49,59,75,86,95,99,102,112,270,281,287,302,306,309,315,321],[11,12,13],"p",{},"A consumer group starts crashing. It restarts, processes for a few seconds, and crashes again.\nLag climbs. Restarting the pods doesn't help — which is the first genuinely useful piece of\ninformation, because it rules out most of what you'd normally suspect.",[11,15,16],{},"This is a note about what that failure actually is, and why the fix isn't where people usually\nreach first.",[18,19,21],"h2",{"id":20},"the-shape-of-the-problem","The shape of the problem",[11,23,24],{},"A Kafka consumer commits offsets to record how far it has read. On restart, it resumes from the\nlast committed offset. That's the whole point — it's what makes consumers restartable.",[11,26,27],{},"Now suppose a message arrives that the consumer can't deserialise. Maybe a producer shipped a\nschema change. Maybe a field that was never null is null. The deserialiser throws, the exception\npropagates, the consumer dies before committing.",[11,29,30],{},"It restarts, resumes from the last committed offset, reads the same message, and dies again.",[11,32,33],{},"That's the loop. Nothing is flapping, nothing is under-resourced, and no amount of restarting\nhelps — the consumer is behaving exactly as designed. It's just that \"resume where you left off\"\nand \"you cannot get past this message\" combine into a system that makes no progress forever.",[18,35,37,38,42],{"id":36},"why-autooffsetreset-gets-blamed","Why ",[39,40,41],"code",{},"auto.offset.reset"," gets blamed",[11,44,45,46,48],{},"Somebody always suggests changing ",[39,47,41],{},". It's worth being precise about what that\nsetting does, because the confusion is the interesting part.",[11,50,51,53,54,58],{},[39,52,41],{}," only applies when there is ",[55,56,57],"strong",{},"no valid committed offset"," — a brand new consumer\ngroup, or a committed offset that has aged out of the topic's retention. It has two useful values:",[60,61,62,69],"ul",{},[63,64,65,68],"li",{},[39,66,67],{},"earliest"," — start from the beginning of the topic",[63,70,71,74],{},[39,72,73],{},"latest"," — start from the next message produced",[11,76,77,78,80,81,85],{},"Set it to ",[39,79,67],{}," and a consumer group with no committed offset replays the whole topic from the\nstart. If a poison message sits anywhere in that history, you've guaranteed you hit it — and if the\ncrash also prevents the offset commit, the group never establishes a valid offset, so the ",[82,83,84],"em",{},"next","\nrestart takes the reset path again and replays from the beginning again.",[11,87,88,89,91,92],{},"That's the version of this bug that looks like ",[39,90,41],{}," is at fault. It isn't. It's\namplifying the real problem, which is that ",[55,93,94],{},"a deserialisation failure is being treated as fatal.",[18,96,98],{"id":97},"the-actual-fix","The actual fix",[11,100,101],{},"Three changes, in order of how much they matter.",[11,103,104,107,108,111],{},[55,105,106],{},"1. Make deserialisation failures survivable."," A message the consumer can't parse should be\nrouted somewhere and skipped, not allowed to kill the poll loop. In Kafka this is what an\n",[39,109,110],{},"ErrorHandlingDeserializer"," and a dead-letter topic are for:",[113,114,119],"pre",{"className":115,"code":116,"language":117,"meta":118,"style":118},"language-java shiki shiki-themes github-dark-dimmed","@Bean\npublic DefaultErrorHandler errorHandler(KafkaTemplate\u003CString, String> template) {\n    \u002F\u002F after retries are exhausted, publish to \u003Ctopic>.DLT and move on\n    var recoverer = new DeadLetterPublishingRecoverer(template);\n    var handler = new DefaultErrorHandler(recoverer, new FixedBackOff(1_000L, 2));\n\n    \u002F\u002F never retry what will never succeed — a bad payload is bad on attempt three too\n    handler.addNotRetryableExceptions(DeserializationException.class);\n    return handler;\n}\n","java","",[39,120,121,134,162,169,190,230,237,243,255,264],{"__ignoreMap":118},[122,123,126,130],"span",{"class":124,"line":125},"line",1,[122,127,129],{"class":128},"s1ain","@",[122,131,133],{"class":132},"scLw-","Bean\n",[122,135,137,140,143,147,150,153,156,159],{"class":124,"line":136},2,[122,138,139],{"class":132},"public",[122,141,142],{"class":128}," DefaultErrorHandler ",[122,144,146],{"class":145},"sLyBy","errorHandler",[122,148,149],{"class":128},"(KafkaTemplate",[122,151,152],{"class":132},"\u003C",[122,154,155],{"class":128},"String, String",[122,157,158],{"class":132},">",[122,160,161],{"class":128}," template) {\n",[122,163,165],{"class":124,"line":164},3,[122,166,168],{"class":167},"sVrTk","    \u002F\u002F after retries are exhausted, publish to \u003Ctopic>.DLT and move on\n",[122,170,172,175,178,181,184,187],{"class":124,"line":171},4,[122,173,174],{"class":132},"    var",[122,176,177],{"class":128}," recoverer",[122,179,180],{"class":132}," =",[122,182,183],{"class":132}," new",[122,185,186],{"class":145}," DeadLetterPublishingRecoverer",[122,188,189],{"class":128},"(template);\n",[122,191,193,195,198,200,202,205,208,211,214,217,221,224,227],{"class":124,"line":192},5,[122,194,174],{"class":132},[122,196,197],{"class":128}," handler",[122,199,180],{"class":132},[122,201,183],{"class":132},[122,203,204],{"class":145}," DefaultErrorHandler",[122,206,207],{"class":128},"(recoverer, ",[122,209,210],{"class":132},"new",[122,212,213],{"class":145}," FixedBackOff",[122,215,216],{"class":128},"(",[122,218,220],{"class":219},"sDGpf","1_000L",[122,222,223],{"class":128},", ",[122,225,226],{"class":219},"2",[122,228,229],{"class":128},"));\n",[122,231,233],{"class":124,"line":232},6,[122,234,236],{"emptyLinePlaceholder":235},true,"\n",[122,238,240],{"class":124,"line":239},7,[122,241,242],{"class":167},"    \u002F\u002F never retry what will never succeed — a bad payload is bad on attempt three too\n",[122,244,246,249,252],{"class":124,"line":245},8,[122,247,248],{"class":128},"    handler.",[122,250,251],{"class":145},"addNotRetryableExceptions",[122,253,254],{"class":128},"(DeserializationException.class);\n",[122,256,258,261],{"class":124,"line":257},9,[122,259,260],{"class":132},"    return",[122,262,263],{"class":128}," handler;\n",[122,265,267],{"class":124,"line":266},10,[122,268,269],{"class":128},"}\n",[11,271,272,273,276,277,280],{},"The distinction that matters: ",[55,274,275],{},"retryable"," failures (the database was briefly unavailable) should\nback off and retry. ",[55,278,279],{},"Non-retryable"," failures (this payload will never parse) should go straight\nto the dead-letter topic. Retrying a malformed message is just a slower crash loop.",[11,282,283,286],{},[55,284,285],{},"2. Commit deliberately."," Auto-commit makes the failure boundary fuzzy — you can't easily reason\nabout what was processed versus what was acknowledged. Manual acknowledgement after successful\nprocessing makes \"did this message count?\" a question with an answer.",[11,288,289,295,296,298,299,301],{},[55,290,291,292,294],{},"3. Then think about ",[39,293,41],{},"."," With the first two in place, this becomes what it\nshould have been all along: a policy decision about cold starts, not a lever for incident response.\nFor a consumer where missing historical data is worse than reprocessing it, ",[39,297,67],{}," is right.\nFor one where stale events are useless, ",[39,300,73],{}," is right. Neither choice can produce a crash loop\nonce bad payloads have somewhere to go.",[18,303,305],{"id":304},"what-to-keep","What to keep",[11,307,308],{},"Two things worth carrying forward.",[11,310,311,314],{},[55,312,313],{},"A poison message is a design gap, not an anomaly."," Any consumer without an explicit answer for\n\"what happens when this message can't be parsed\" has a crash loop waiting for the right input.",[11,316,317,320],{},[55,318,319],{},"Watch consumer lag, not consumer health."," A crash-looping consumer often looks fine to a\nliveness probe — it starts, it's alive, it serves the health endpoint, it dies, it starts again.\nLag is what tells you it isn't actually getting anywhere.",[322,323,324],"style",{},"html pre.shiki code .s1ain, html code.shiki .s1ain{--shiki-default:#ADBAC7}html pre.shiki code .scLw-, html code.shiki .scLw-{--shiki-default:#F47067}html pre.shiki code .sLyBy, html code.shiki .sLyBy{--shiki-default:#DCBDFB}html pre.shiki code .sVrTk, html code.shiki .sVrTk{--shiki-default:#768390}html pre.shiki code .sDGpf, html code.shiki .sDGpf{--shiki-default:#6CB6FF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}",{"title":118,"searchDepth":136,"depth":136,"links":326},[327,328,330,331],{"id":20,"depth":136,"text":21},{"id":36,"depth":136,"text":329},"Why auto.offset.reset gets blamed",{"id":97,"depth":136,"text":98},{"id":304,"depth":136,"text":305},"2026-05-18","A Kafka consumer group stuck in a restart loop, one malformed message, and why auto.offset.reset is the wrong place to look for a fix.",false,"md",{},"\u002Fwriting\u002Fkafka-consumer-crash-loop","6 min read",{"title":6,"description":333},"writing\u002Fkafka-consumer-crash-loop",[342,343,344],"Kafka","Debugging","Streaming","76xbGl3Lhx5ldvYgIRWRhc0RstGKhpWBf2nyQWrdd8E",1785591010234]