DEV Community

Containers die whenever; two hours of progress should not die with them

dreamyman on November 13, 2026

Three posts in: tasks are isolated from each other, and the wait went from 45 seconds to a fraction of one. But one thing has not changed since the...
Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

@softverseadmin Hard cap, and make it stop loudly. Summarizing at page boundaries drops the exact detail the resume needs, and it fails silently. A cap that refuses the write and says so is a limit the agent cannot talk its way past.

Collapse
 
softverseadmin profile image
Rabin Adhikari • • Edited

So I actually ran this before replying. Paged job, checkpoint budget of 256 bytes, and the resume needs one exact page token to restart from.

The summarize version lost it. The summary said something like 'processed up to the low forties' and the exact token the resume needed was just gone. Nothing flagged the loss either. Resume would've been guessing.

Yours held up. Checkpoint only carried the state: task id, cursor, last verified output, plus a pointer into the append-only log (offset + hash). 138 bytes. Resume followed the pointer and pulled the exact token back out. And when I stuffed the whole transcript in there — 1,376 bytes — the write refused. Nothing written, run stops. Loud, like you said.

So the working setup is this. Checkpoint holds state only, under a hard cap that refuses the write. The detail never goes in the checkpoint at all — it lives in an append-only log, and all the checkpoint carries is a pointer to it, offset and hash. Resume follows the pointer and pulls the exact detail back out. The cap keeps the checkpoint honest. The log keeps the detail alive.

Collapse
 
mike-johnstone profile image
Mike Johnstone •

I'd add crashes around the two writes to that test. The offset + hash only helps if its log entry survives too. After killing the worker during a log append or checkpoint update, I'd want resume to find either the old complete checkpoint and its log data or the new pair, never a checkpoint pointing at data that didn't survive. That's a separate check from refusing an oversized write.

Thread Thread
 
softverseadmin profile image
Rabin Adhikari •

Yeah, that's the hole in it. I ran the kill between the two writes and it did exactly what you said — checkpoint went out pointing at a log entry that didn't survive, so resume came back with a pointer to nothing.

What held was boringly ordered: log first, fsync it, then the checkpoint with the hash, and resume doesn't trust the pointer unless the hash matches. If it doesn't, it falls back to the last good checkpoint instead of chasing a dead one. So it's not just fsyncing both, it's the order — and the hash is what makes the fallback real instead of hopeful.

Thread Thread
 
mike-johnstone profile image
Mike Johnstone •

Thanks for actually running it. That's a useful distinction: two fsyncs aren't enough if the checkpoint can get ahead of the log. Keeping the last good pair makes a failed hash check recoverable instead of just detectable.

Thread Thread
 
softverseadmin profile image
Rabin Adhikari •

Yeah, that's the better way to say it — detectable just tells you something broke, recoverable is what the next container actually needs. The fallback to the last good pair was the part that made the test pass instead of just fail loudly.

I think that's the pair that matters then: log first, checkpoint second, and never trust a pointer without the hash. The rest is just making the failure boring.

Collapse
 
ascendra_ventures_92c21f2 profile image
Ascendra Ventures •

Deleting the working pod on purpose is the honest test here, and the log line resuming from page 17 says more than the code does. Rabin's point is worth keeping next to it: checkpointing decides what survives, not what the next container is allowed to touch. Paul's split between the state of the work and the full transcript is the other half, since every resume pays to read the checkpoint back.

We rebuilt a candidate-enrichment pipeline at Ascendra Ventures around the same idea, for a seed-stage talent platform where any slow step could leave a person in an unclear state. Each person now runs as a durable workflow in Temporal. It claims the person, fetches profile data, processes urgent profiles straight away and batches the rest, waits durably on OpenAI batch jobs, and falls back to polling when webhooks go quiet. The rule we held to was that every candidate either finishes or visibly fails, never sits in the middle. A checkpoint file gets you a long way, and what a workflow engine adds on top is that the waiting itself survives a restart.

What happens today if two containers pick up the same task id after a re-dispatch? Is anything stopping both from writing to the same checkpoint?

Collapse
 
softverseadmin profile image
Rabin Adhikari •

Temporal's the honest version of this if you're already running it — the waiting surviving a restart is the part a file on a shared disk just can't do on its own.

On the duplicate task question, nothing in my setup stops it. Two containers pick up the same task id, both write the same checkpoint, last writer wins and you don't even know it happened. The thing that would stop it is a lease — dispatcher hands the unit out with a token, checkpoint write has to carry it, stale token gets refused. Without that the idempotency key saves the side effect but not the checkpoint itself.

Collapse
 
delayrun profile image
delay.run •

Deleting the container on purpose is the only test that really counts here, nice to see it done instead of argued about.

One thing I'd watch with the JSON file on a shared disk: if the write isn't atomic (temp file, then rename), a container killed mid-write leaves a half-written checkpoint the next shift can't parse. Looking forward to the watcher post.

Collapse
 
softverseadmin profile image
Rabin Adhikari •

Good catch, same thing bit me in the test — killed mid-write on the JSON file directly and resume couldn't even parse it. Temp file plus rename fixed it, resume just saw the old checkpoint and carried on. That's the one I'd actually ship first, it's the cheapest of the three fixes.

Collapse
 
izgorodin profile image
Edward Izgorodin •

The resume from page 17 also says when a page runs twice. The checkpoint is written after the page is done, so a kill that lands after the page's work and before the json.dump repeats that page on the next shift. For a crawl that only reads, the repeat costs a second fetch. For an agent step that writes somewhere else, such as opening a pull request, posting a message or charging a card, the next container does it again, and the checkpoint cannot tell, because it records how far the work got, not what the last step already did outside the disk.

The usual fix sits on the side effect rather than on the checkpoint: a key made of the task id and the unit number, which the external write checks before it acts. The same delete-the-pod test would show it, with the kill placed right after a step that writes and a count of the duplicates. Once the next posts make the dispatcher remember outstanding work and re-send a stalled task by itself, will the unit that was in flight run again?

Collapse
 
softverseadmin profile image
Rabin Adhikari •

That's the one that actually bites in practice. Checkpoint tells you how far you got, not what you already did outside the disk. I ran the kill right after a page that writes out — in my test it was opening a PR — and resume ran the page again, second PR, because nothing on the other side knew it'd already happened.

The fix wasn't in the checkpoint at all. It was the key — task id plus unit number, checked by the thing receiving the write before it acts. Same kill, same repeat, one PR.

And on your dispatcher question, yeah — if it re-sends a stalled unit on its own, the one still in flight runs too unless there's a lease on it. Without something fencing the checkpoint write you've just got two containers racing the same unit, and the idempotency key is the only thing stopping the double write.

Collapse
 
izgorodin profile image
Edward Izgorodin •

Rabin, your test is the useful version of the argument, since one PR instead of two is a result and not a claim. The lease raises one more case worth putting in the same test. A container that stalls past its lease without dying, through a long pause or a lost network, can wake up and finish its unit after the dispatcher has given that unit to someone else. The idempotency key stops the second PR, but the late container then writes its checkpoint over the newer one and the progress counter moves backwards. Writing the checkpoint as compare-and-set, so a unit number can only go up, closes that, and pausing a container past its lease instead of killing it would show whether it is needed.

Thread Thread
 
softverseadmin profile image
Rabin Adhikari • • Edited

That's the case I hadn't covered — the stall, not the kill. I paused one instead of killing it, let the dispatcher hand the unit to the other container, then woke the first one up after the second had moved on.

You're right about the split. The key stopped the second PR, same as before. But the late one still wrote its checkpoint over the newer one and the counter went backwards — 18 back to 17. So the key saves the side effect and the checkpoint still lies.

The compare-and-set held. Same run, a write only lands if the unit number is higher than what's already there — the stale write just gets refused and the counter stays at 18. That's the piece I'd actually ship with the lease: the lease decides who runs, the monotonic write decides what counts as progress.

One thing I'm still not sure about: does the lease need a heartbeat from the worker, or is the dispatcher's own timeout enough? If the worker's clock is the one that's wrong, the timeout fires on stale info either way.

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

The checkpoint decides what the next container has to read back before it can act, and that read is paid input. So what goes on the shared disk is a cost line as well as a recovery one. The checkpoint should hold the state of the work, not the whole transcript.

Collapse
 
softverseadmin profile image
Rabin Adhikari •

That's the cost line most checkpoint designs hide, Paul — the read-back is paid input, and for an agent it's paid in tokens as well as time. A checkpoint that holds the whole transcript turns every resume into re-reading the entire job.

The split I'd draw is state vs transcript: the checkpoint holds only the minimal resumable state — task id, page cursor, last verified output — and the transcript stays in an append-only log the next container fetches only when it actually needs it. Resume reads a few hundred bytes, not the whole history.

How do you bound it in practice — a hard size cap on the checkpoint, summarise-and-drop at page boundaries, or something else? The cap is where the recovery guarantee and the cost line actually meet.

Collapse
 
softverseadmin profile image
Rabin Adhikari •

The delete-the-pod test is the right judge — most resilience claims die exactly there. One half I'd add: checkpointing answers what survives the container. It doesn't answer what the next container is allowed to touch. A resumed agent mounting the same PVC with the same identity inherits the full blast radius of the one that crashed — including when the crash was the agent's own doing.

When I set up agent sandboxes I ended up pairing the two: state on a volume so the work survives, but the workload itself inside a gVisor sandbox with a tightly scoped identity, so delete-and-resume is safe to automate, not just possible. Is the PVC fenced per task in your pool, or shared across tasks? A shared blackboard is also a shared contamination path — curious how you fenced it.

The isolation half, for reference: devopsenginer.com/blog/sandbox-ai-...