Full-stack developer in Denver. Mostly TypeScript and Go, plus Postgres, and I like shipping small tools that solve one problem well. Learning Rust; building a small Postgres queue.
The one-droplet constraint is what makes this interesting. We did something similar with a second port and an nginx switch, and the flakiest part was in-flight requests during the restart. A short grace period and keeping the old worker alive for a few seconds fixed it.
Full-stack dev shipping Django/React SaaS and Shopify stores since 2018. 37 client projects, 11 live Shopify builds. I write production war stories so you don’t repeat my outages.
Same pattern, different tools. Your nginx port switch and Traefik picking up healthy containers both solve the cutover. Like you said, the in-flight part is what bites. What fixed it for me was draining first: fail the old container’s healthcheck so Traefik stops routing to it, then send SIGTERM. Did you add a delay before the switch, or just keep the old worker alive afterward?
The critical bottleneck for single-droplet zero-downtime deploys isn't reverse proxy cutover; it's peak memory contention during the worker handoff. Spinning up overlapping application instances—whether via dual Gunicorn processes or parallel containers—on a resource-constrained box routinely trips the Linux OOM killer under real traffic. Decoupling backward-compatible schema migrations from the deployment phase and utilizing systemd socket passing or staggered worker reloads keeps your footprint strictly bounded without dropping in-flight connections.
Full-stack dev shipping Django/React SaaS and Shopify stores since 2018. 37 client projects, 11 live Shopify builds. I write production war stories so you don’t repeat my outages.
Fair point, and on 2 GB it’s the real constraint. The overlap is about a minute per deploy with two Django containers. I didn’t include peak memory numbers in the post, and I should have. That’s going in the next round of measurements. Celery and Flower restart after the old container is gone, so the Django overlap is the only one. Gunicorn’s in-place reload (USR2/HUP) is the lower-memory alternative. The trade-off is giving up the “new image proves itself healthy before taking traffic” property, and that check is what caught the broken image in rehearsal. Expand/contract migrations are covered at the end.
Full-stack developer in Stockholm. I ship side projects end to end: Rails or TypeScript, Postgres, just enough frontend. Learning SQLite internals; building a CLI for tracking side projects.
Full-stack dev shipping Django/React SaaS and Shopify stores since 2018. 37 client projects, 11 live Shopify builds. I write production war stories so you don’t repeat my outages.
90 seconds, matching gunicorn’s --timeout, because some requests wait on an LLM. In practice the drain step does most of the work. By the time SIGTERM arrives, Traefik has stopped sending new requests, so the grace period only has to cover what’s already in flight. That’s usually a few seconds. The 90 is a ceiling, not a typical wait.
Custom software & AI development company. We build production-ready AI integrations, mobile apps, and web platforms for businesses that want real results. lycore.com
The control request to google.com in your uptime loop is a smart detail; most people skip that and blame the server for their own network blips. One thing I'd look at on a single droplet is whether the new container's healthcheck verifies the DB connection and not just that gunicorn is listening, since Traefik will route traffic as soon as it reports healthy. Also curious how you handle migrations that drop or rename columns while the old container is still serving.
Full-stack dev shipping Django/React SaaS and Shopify stores since 2018. 37 client projects, 11 live Shopify builds. I write production war stories so you don’t repeat my outages.
Good catch on the control. My own Wi-Fi dropped twice that day. On the DB check, there’s a trade-off on a single box: Traefik routes on this healthcheck, so if it queries the DB, a DB blip marks every container unhealthy at once. You’d get 503s either way, but it changes what the check means. The middle ground is a DB check in the deploy script’s readiness gate, and a lighter check for routing. Migrations: expand/contract, covered in the “What 99.99% still needs” section. Renames and drops take two deploys.
Good point. A healthcheck that only checks that gunicorn is listening can pass while the database is still unreachable, so I would add a simple database query to it.
For migrations, I would keep them backward compatible during the rollout, adding columns first and removing or renaming them only in a later deploy.
That way the old container can keep serving until the new one is fully healthy.
Custom software & AI development company. We build production-ready AI integrations, mobile apps, and web platforms for businesses that want real results. lycore.com
Thanks Rishabh, agreed on both points. For the healthcheck I'd keep the query as cheap as SELECT 1 with a short timeout, so a slow database fails the check quickly instead of hanging it. And splitting schema changes expand/contract, adding first and dropping or renaming in a later deploy, means old and new code both work during the switch.
SELECT 1 is the right call — it proves the connection without touching real tables.
If the check passes but the app still serves stale data, a second probe on the read path catches that split.
Thanks for sharing the detail, that distinction is worth keeping in the checklist.
Full-stack dev shipping Django/React SaaS and Shopify stores since 2018. 37 client projects, 11 live Shopify builds. I write production war stories so you don’t repeat my outages.
+1 on SELECT 1 with a short timeout. The one caveat is the one above: if the proxy routes on that same check, a slow DB pulls every container out of rotation together. A separate readiness probe and routing probe keeps the two jobs apart.
The drain-before-stop step is the part I would keep. A healthcheck that only tells the deploy script the new container is ready does not help if Traefik still sends traffic to the old one until it stops.
The one-droplet constraint is what makes this interesting. We did something similar with a second port and an nginx switch, and the flakiest part was in-flight requests during the restart. A short grace period and keeping the old worker alive for a few seconds fixed it.
Same pattern, different tools. Your nginx port switch and Traefik picking up healthy containers both solve the cutover. Like you said, the in-flight part is what bites. What fixed it for me was draining first: fail the old container’s healthcheck so Traefik stops routing to it, then send SIGTERM. Did you add a delay before the switch, or just keep the old worker alive afterward?
The critical bottleneck for single-droplet zero-downtime deploys isn't reverse proxy cutover; it's peak memory contention during the worker handoff. Spinning up overlapping application instances—whether via dual Gunicorn processes or parallel containers—on a resource-constrained box routinely trips the Linux OOM killer under real traffic. Decoupling backward-compatible schema migrations from the deployment phase and utilizing systemd socket passing or staggered worker reloads keeps your footprint strictly bounded without dropping in-flight connections.
Bohdan Hordii // bohdanhordii.dev
Fair point, and on 2 GB it’s the real constraint. The overlap is about a minute per deploy with two Django containers. I didn’t include peak memory numbers in the post, and I should have. That’s going in the next round of measurements. Celery and Flower restart after the old container is gone, so the Django overlap is the only one. Gunicorn’s in-place reload (USR2/HUP) is the lower-memory alternative. The trade-off is giving up the “new image proves itself healthy before taking traffic” property, and that check is what caught the broken image in rehearsal. Expand/contract migrations are covered at the end.
The grace period for in-flight requests is the detail people miss. How long did you end up using?
90 seconds, matching gunicorn’s --timeout, because some requests wait on an LLM. In practice the drain step does most of the work. By the time SIGTERM arrives, Traefik has stopped sending new requests, so the grace period only has to cover what’s already in flight. That’s usually a few seconds. The 90 is a ceiling, not a typical wait.
The locking section is the one I keep re-reading.
The control request to google.com in your uptime loop is a smart detail; most people skip that and blame the server for their own network blips. One thing I'd look at on a single droplet is whether the new container's healthcheck verifies the DB connection and not just that gunicorn is listening, since Traefik will route traffic as soon as it reports healthy. Also curious how you handle migrations that drop or rename columns while the old container is still serving.
Good catch on the control. My own Wi-Fi dropped twice that day. On the DB check, there’s a trade-off on a single box: Traefik routes on this healthcheck, so if it queries the DB, a DB blip marks every container unhealthy at once. You’d get 503s either way, but it changes what the check means. The middle ground is a DB check in the deploy script’s readiness gate, and a lighter check for routing. Migrations: expand/contract, covered in the “What 99.99% still needs” section. Renames and drops take two deploys.
Good point. A healthcheck that only checks that gunicorn is listening can pass while the database is still unreachable, so I would add a simple database query to it.
For migrations, I would keep them backward compatible during the rollout, adding columns first and removing or renaming them only in a later deploy.
That way the old container can keep serving until the new one is fully healthy.
Thanks Rishabh, agreed on both points. For the healthcheck I'd keep the query as cheap as SELECT 1 with a short timeout, so a slow database fails the check quickly instead of hanging it. And splitting schema changes expand/contract, adding first and dropping or renaming in a later deploy, means old and new code both work during the switch.
SELECT 1 is the right call — it proves the connection without touching real tables.
If the check passes but the app still serves stale data, a second probe on the read path catches that split.
Thanks for sharing the detail, that distinction is worth keeping in the checklist.
+1 on SELECT 1 with a short timeout. The one caveat is the one above: if the proxy routes on that same check, a slow DB pulls every container out of rotation together. A separate readiness probe and routing probe keeps the two jobs apart.
The drain-before-stop step is the part I would keep. A healthcheck that only tells the deploy script the new container is ready does not help if Traefik still sends traffic to the old one until it stops.
Cool