Celery doesn't promise exactly-once execution. In many setups, it's at-least-once, so your tasks need to survive running twice. Here's a practical breakdown.
Why a task runs twice
-
acks_late=True+ worker crash: the broker redelivers the task, possibly after partial work. -
Redis visibility timeout: unacknowledged messages become visible again (1 hour by default). Long tasks or long
countdown/etavalues can be redelivered while still running. - Retries: a task that fails after a side effect repeats that side effect.
- Double submissions: double-clicks, duplicate webhooks, client retries.
- Celery Beat: overlapping runs, or more than one Beat process.
- Queuing inside a transaction: the worker can start before the data is committed.
1. Make tasks idempotent
# Not idempotent
@shared_task
def give_bonus(user_id):
Wallet.objects.filter(user_id=user_id).update(balance=F("balance") + 10)
# Idempotent
@shared_task
def recalculate_total(order_id):
order = Order.objects.get(pk=order_id)
order.total = calculate_total(order)
order.save(update_fields=["total"])
Compute from source data, prefer update_or_create(), and check state before acting.
2. Use a unique constraint as a safety net
Check-then-act fails when two copies run at the same moment. Let the database decide:
from django.db import IntegrityError, transaction
@shared_task
def create_payout(order_id):
order = Order.objects.get(pk=order_id)
try:
with transaction.atomic():
Payout.objects.create(order=order, amount=order.total)
except IntegrityError:
return # duplicate run
For concurrent updates to one row, use select_for_update() inside transaction.atomic().
3. Use a lock for non-overlapping work
@shared_task(bind=True)
def generate_invoice(self, order_id):
lock = redis_client.lock(f"lock:invoice:{order_id}", timeout=300)
if not lock.acquire(blocking=False):
return "skipped: already running"
try:
build_invoice(order_id)
finally:
try:
lock.release()
except redis.exceptions.LockError:
pass
Rules of thumb:
- Always set a timeout, or a crashed worker blocks the task forever
- Make the timeout longer than the task's normal runtime
- A lock is not idempotency: a duplicate arriving after the lock is released still runs
-
cache.add()works as a simple lock only on shared backends like Redis, not local-memory cache
4. Idempotency keys for webhooks
class ProcessedEvent(models.Model):
event_id = models.CharField(max_length=200, unique=True)
processed_at = models.DateTimeField(auto_now_add=True)
@shared_task
def handle_payment_webhook(event_id, payload):
try:
with transaction.atomic():
ProcessedEvent.objects.create(event_id=event_id)
apply_payment(payload)
except IntegrityError:
return # already handled
The key and the business change share one transaction, so a failed run leaves no key behind and a retry can still succeed.
5. Queue after commit
def place_order(request):
with transaction.atomic():
order = Order.objects.create(...)
transaction.on_commit(lambda: process_order.delay(order.id))
Pass IDs, not model objects.
6. Check your config
CELERY_TASK_ACKS_LATE = True
CELERY_TASK_REJECT_ON_WORKER_LOST = True
CELERY_BROKER_TRANSPORT_OPTIONS = {"visibility_timeout": 43200} # 12 hours
Set visibility_timeout higher than your longest task and longest countdown. For Beat, run exactly one instance and lock long periodic tasks.
Which fix when?
| Situation | Approach |
|---|---|
| Any retryable task | Idempotent design |
| Create a record once | Unique constraint + IntegrityError
|
| Concurrent updates | select_for_update() |
| Expensive, must not overlap | Redis lock with timeout |
| Webhooks, API retries | Idempotency key in the database |
| Periodic tasks | Lock + single Beat instance |
Full guide
The complete post has a pre-ship checklist and an FAQ:
👉 https://blog.reiteachaigean.com/blog/celery-prevent-duplicate-task-execution
More on Celery and large datasets:
How do you handle duplicate tasks in production?
Top comments (0)