Celery, workers, retries & Beat

What are the broker and the result backend, and do you need both?

The broker (Redis, RabbitMQ, SQS) transports task messages from producers to workers — it is required. The result backend (Redis, a database, S3) stores return values and states — it is optional, and you should turn it off unless you actually read results.

app.conf.task_ignore_result = True     # default on: most tasks' return values are never read

Why: every result write costs a round trip and memory, and with a Redis backend an unbounded pile of celery-task-meta-* keys is a classic memory incident. If you need results, set result_expires and prefer writing the outcome to your own database where you can query it.

RabbitMQ vs Redis as broker: RabbitMQ is a real broker with acknowledgements, durable queues and flow control — the safer choice for reliability. Redis is simpler and faster but the visibility-timeout model means a task can be redelivered while still running, and a Redis failover can lose queued messages.

Explain acks_late. What is the trade-off?

By default a worker acknowledges a message as soon as it is received (acks_early): if the worker is killed mid-task, the message is gone and the task never runs. With task_acks_late = True the ack happens after the task completes, so a crashed worker's task is redelivered.

app.conf.task_acks_late = True
app.conf.task_reject_on_worker_lost = True   # requeue if the worker process dies (OOM kill)
app.conf.worker_prefetch_multiplier = 1      # essential with acks_late for long tasks

The trade-off: acks_late gives at-least-once delivery, so a task can run twice — once partially before the crash and again after. That is acceptable only if your task is idempotent. If it is not idempotent, making it so is the actual work.

How do you make a task idempotent?

Give each logical operation a stable key and guard on it in the database, not in code:

@app.task(bind=True, acks_late=True, max_retries=5)
def charge_order(self, order_id: int, idem_key: str):
    with transaction.atomic():
        # unique index on idem_key: a duplicate delivery loses the race and exits
        created = Charge.objects.filter(idem_key=idem_key).exists()
        if created:
            return "already-processed"
        order = Order.objects.select_for_update().get(pk=order_id)
        if order.status != "pending":
            return "not-pending"
        Charge.objects.create(order=order, idem_key=idem_key, amount=order.total)
        order.status = "charged"; order.save(update_fields=["status"])

Techniques: unique constraints, conditional updates (UPDATE … WHERE status='pending'), a processed-messages table, or a Redis SET NX lock with a TTL for cheap deduplication. Pass ids, not objects — a serialised model in the message is stale by the time it runs, and it bloats the queue.

How does retrying work, and how do you avoid retry storms?

@app.task(bind=True, autoretry_for=(RequestException,), retry_backoff=True,
          retry_backoff_max=600, retry_jitter=True, max_retries=5,
          retry_kwargs={"countdown": 5})
def sync_customer(self, customer_id):
    ...
    # or explicitly:
    # raise self.retry(exc=exc, countdown=2 ** self.request.retries)

Points to make: exponential backoff with jitter (without jitter, a downstream outage produces synchronised thundering-herd retries); a bounded max_retries with a terminal handler (on_failure → alert, dead-letter table); and distinguishing retryable errors (timeouts, 5xx, deadlocks) from permanent ones (validation failure, 404) — retrying a ValueError five times just delays the inevitable and hides the bug.

Also set task_time_limit (hard, SIGKILL) and task_soft_time_limit (raises SoftTimeLimitExceeded so you can clean up). A task with no time limit can hang a worker slot forever.

What is prefetching, and why is my queue not draining evenly?

Workers fetch worker_prefetch_multiplier * concurrency messages in advance. With the default of 4 and long tasks, one worker grabs a batch and sits on it while other workers idle — messages are reserved but unstarted, so the queue looks busy and latency is terrible.

For long or variable-duration tasks set worker_prefetch_multiplier = 1. For very short tasks a higher value reduces broker round trips. This is one of the most impactful Celery settings and a strong signal that you have operated it.

How do you route tasks, and why does one queue not scale?

app.conf.task_routes = {
    "billing.*":            {"queue": "critical"},
    "reports.generate_*":   {"queue": "heavy"},
    "emails.*":             {"queue": "default"},
}
# celery -A proj worker -Q critical -c 8 --max-tasks-per-child=200
# celery -A proj worker -Q heavy    -c 2 --max-memory-per-child=500000

One queue means a batch of 50,000 slow report tasks blocks every password-reset email behind them. Split by latency class and resource profile, run a separate worker deployment per queue, and scale them independently. Priorities exist but are unreliable across brokers — separate queues are the robust answer.

What is Celery Beat, and how do you run it safely?

Beat is the scheduler: it reads a schedule and publishes tasks at the right times. It does not execute anything — workers do.

app.conf.beat_schedule = {
    "nightly-invoices": {
        "task": "billing.close_day",
        "schedule": crontab(hour=2, minute=0),
        "options": {"queue": "critical", "expires": 3600},
    },
    "poll-webhooks": {"task": "hooks.poll", "schedule": 30.0},
}

The operational rules:

1. Exactly one Beat process. Two schedulers means every job fires twice. Use a singleton deployment (Kubernetes replicas: 1, a leader lock, or celery-redbeat, which stores the schedule in Redis with a lock).
2. Set expires on periodic tasks. If workers were down for two hours, you do not want 240 stale polls to execute at once when they come back.
3. timezone — set app.conf.timezone explicitly and think about DST: crontab(hour=2) runs twice or zero times on the transition day. For anything financial, schedule in UTC.
4. Beat's schedule file (celerybeat-schedule) is local state — on a container it must be a volume or you must use RedBeat, otherwise a restart can re-fire or skip jobs.
5. Make periodic tasks idempotent and overlap-safe — a job that takes 90 seconds on a 60-second schedule will overlap. Guard with a Redis lock.

Design a task that must not run twice concurrently.

@contextmanager
def redis_lock(key, ttl=600):
    token = uuid4().hex
    got = redis.set(key, token, nx=True, ex=ttl)
    try:
        yield got
    finally:
        if got:   # compare-and-delete via Lua, so we never delete someone else's lock
            redis.eval("if redis.call('get',KEYS[1])==ARGV[1] then return redis.call('del',KEYS[1]) end",
                       1, key, token)

@app.task
def rebuild_search_index():
    with redis_lock("lock:rebuild-index", ttl=1800) as acquired:
        if not acquired:
            return "skipped: already running"
        do_rebuild()

Note the TTL must exceed the worst-case runtime, or the lock expires mid-run and a second copy starts. If correctness depends on it, add a fencing token or a database-level guard as well.

What are chains, groups and chords — and where do they go wrong?

chain(fetch.s(url), parse.s(), store.s())          # sequential, each result feeds the next
group(process.s(i) for i in ids)                   # parallel fan-out
chord(group(part.s(i) for i in ids))(combine.s())  # fan-out then a callback on all results

Where they go wrong: chords require a result backend and the callback holds state while waiting; a group of 100,000 tasks creates 100,000 messages at once (chunk it with group(...).skew() or task.chunks(items, 100)); a failure mid-chain leaves you with a partially completed workflow and no rollback. For anything with real business steps and compensation, a workflow engine (Temporal, Airflow, or a state machine in your own database) is a better answer than a deep Celery canvas — say so.

How do you monitor and debug Celery in production?

Metrics that matter: queue depth and age of the oldest message (the real latency signal), task success/failure rate, runtime p50/p95 per task, worker liveness, and retries per minute. Tools: Flower for a live view, celery -A proj inspect active/reserved/stats, Prometheus exporters, and Sentry for exceptions with task arguments.

Common production failures to be able to name: memory leaks in workers (fix with --max-tasks-per-child / --max-memory-per-child), tasks enqueued before the transaction commits (fix with transaction.on_commit), a task holding a database connection for a long external call, large payloads in messages (pass ids, store blobs in S3), losing all queued work when Redis restarts without persistence, and a deploy where new task code meets old messages — always add new parameters with defaults, and never rename a task without a deprecation period.