The incident round

Every weekday at 09:00 your p99 goes from 180 ms to 4 s for about ninety seconds. p50 never moves. Where do you look?

Start by naming what the shape rules out. If p50 were also rising, you would be looking at global saturation: CPU, connection pool, a slow dependency on the common path. It is not moving, so the vast majority of requests are unaffected. Something is hurting a minority of requests, hard, at a fixed time.

Candidates that fit "small subset, fixed time, self-healing":

• Cache expiry avalanche. Keys populated during yesterday's 09:00 traffic ramp with an identical TTL all expire together. Every miss goes to Postgres, the misses collide on the same rows, and the pile-up resolves as soon as the cache refills. The tell: a Redis miss-rate spike exactly coincident with the latency spike, and database QPS jumping while cache QPS drops.
• A cron or scheduled job — a nightly report, an ETL, a backup, a VACUUM, an analytics query — competing for database I/O or taking locks. The tell: database CPU or checkpoint activity spikes with no corresponding application traffic increase.
• Cold start / autoscaler. Traffic ramps at 09:00, the HPA adds pods, and new pods serve traffic before their connection pools, JIT, or in-process caches are warm. The tell: the spike correlates with pod-count changes, and per-pod latency shows the new pods are the slow ones.
• Connection pool exhaustion at the ramp. A brief surge queues requests waiting for a connection; those requests are the p99 and everything else is fine.

Customers are occasionally charged twice. The charge task takes a Redis lock. Explain how a double charge is still possible.

There are at least four independent ways to double charge here, and a strong answer names several:

1. Lease expiry under a pause. The worker takes the lock with a 60-second TTL, then stalls — a GC pause, CPU steal on a noisy host, a slow Stripe call, a network partition. The lock expires while the work is still in flight. A second worker acquires it legitimately and charges again. Nothing is broken; the lock did exactly what it promised. Locks with timeouts cannot provide mutual exclusion in an asynchronous system, because you cannot distinguish a dead process from a slow one.
2. Redelivery with acks_late. The message is acknowledged only after completion. If the worker dies after Stripe succeeded but before the ack, the broker redelivers and the whole task runs again. This is at-least-once delivery working as designed.
3. Redis failover. Redis replication is asynchronous. If the primary fails over after granting the lock but before replicating it, the new primary has no record and grants the same lock to someone else.
4. The release is not compare-and-delete. If the task deletes the key unconditionally at the end, a worker whose lease already expired can delete a different worker's lock, cascading the problem.

The correct fix is not a better lock. It is to make the operation idempotent at the point of truth:

# 1. The client of this task generates a stable key per logical charge attempt.
# 2. The database enforces uniqueness; the lock becomes an optimisation, not a guarantee.
with transaction.atomic():
    charge, created = Charge.objects.get_or_create(
        idempotency_key=key,                       # UNIQUE index — the real mutual exclusion
        defaults={"order_id": order_id, "status": "pending", "amount": amount},
    )
    if not created:
        return charge.result                       # someone else owns this; do nothing
# 3. Pass the same key to the provider so THEY dedupe too.
result = stripe.PaymentIntent.create(..., idempotency_key=key)

Note the layering: your database prevents duplicate rows, and the provider's idempotency key prevents duplicate charges even if you crash between the row and the API call. The unique index is what actually enforces correctness, because it is the one place where two concurrent actors are forced through a single serialisation point.

A dependency starts returning 500s. Your service, which retries three times, goes down completely. Why?

Three things are compounding:

Retry amplification. If A retries 3 times and B retries 3 times, one user request becomes up to 9 calls to C. C is degraded, so its error rate rises, so more retries fire. The retry traffic is now larger than the organic traffic, and C cannot recover because you are DDoSing it with your own retries. This is why retries must be budgeted (a global cap like "retries may not exceed 10% of requests"), why only one layer in a call chain should retry, and why every retry needs exponential backoff with jitter.

Thread/connection pool exhaustion — the reason unrelated endpoints fail. C is slow, so calls to it hold their worker threads or connections much longer. Those come from a pool shared with every other endpoint. The pool drains, and requests that never touch C are now queuing behind ones that do. The fix is a bulkhead: a separate, bounded pool per downstream dependency, so a slow dependency can only consume its own budget.

Metastable failure — the reason A does not recover when C does. Once the queue is deep, every request that reaches a worker has already been waiting longer than the client's timeout; the client has given up and retried, so the work is wasted, and completing it produces no value while consuming the capacity needed to drain the queue. The system is now sustaining its own overload with no external trigger. Recovery requires shedding load: drop the queue, reject early, or restart.

Memory on your web pods climbs steadily and they OOM every 40 hours. Restarting fixes it. How do you find the leak?

Linear growth independent of traffic is itself a strong clue: it points at something accumulating per unit time (a background thread, a scheduler, a metrics registry) rather than per request. If it scaled with traffic, I would look at per-request accumulation instead.

The method:

1. Confirm it is Python heap, not RSS from another cause. Fragmentation, a C extension's allocations, and unbounded thread stacks all raise RSS without showing in Python's heap. Compare tracemalloc totals with process RSS: if the heap is flat and RSS climbs, the leak is in native code or fragmentation, and the answer is different (arena tuning, a leaking C library, or jemalloc).
2. Snapshot and diff. tracemalloc.take_snapshot() on a timer, compare_to(previous, "lineno"), log the top 10 growers. This names the allocating line in production with acceptable overhead. memray gives better output if you can run it.
3. Look at the usual accumulators, and check them explicitly rather than guessing: an unbounded module-level dict or list used as a cache; functools.lru_cache on a method (it retains self, so no instance is ever collected); a logging handler or metrics library creating a new label combination per request (a classic: putting a user id or URL path in a Prometheus label, which creates unbounded cardinality); an event listener or callback registry appended to and never removed; a gc-uncollectable cycle involving __del__; or a session/connection object retained in a request-scoped context that is never torn down.
4. Correlate with what changed three weeks ago. git log on the deploy that precedes the onset. A leak with a start date is a code change, and finding the commit is often faster than the profiler.

Your database CPU is at 95% but query latency is fine and throughput has not increased. What is going on?

Work through the candidates in order of likelihood:

• A new query, or an old query with a new plan. The planner may have flipped to a sequential scan after statistics drifted or after a data-distribution change (a tenant grew 100x). Check pg_stat_statements ordered by total_exec_time and compare against last week. A plan flip is invisible in the code and is the most common cause of "nothing changed but everything is slower".
• Autovacuum working hard, because a large delete or update created millions of dead tuples. It is CPU and I/O intensive, and it is doing necessary work. The tell: pg_stat_progress_vacuum has active entries.
• Connection churn. Postgres forks a process per connection; an application without pooling that opens and closes connections per request burns CPU on process creation. The tell: high numbackends variance and a low ratio of transactions to connections. PgBouncer fixes it.
• Index maintenance overhead on write. Someone added four indexes to a hot table; every insert now updates five structures.
• Fewer rows cached. If the working set outgrew shared_buffers, the same query does more I/O and more CPU decompressing/reading pages. Cache hit ratio falls before latency does.

Then the important framing: 95% CPU with acceptable latency is not necessarily an incident — it may be efficient use of a resource. The real question is headroom: at 95% you have no capacity for a traffic spike, a failover, or a rebuild. So the urgency is about risk, not current pain, and that distinction is what you want to demonstrate.

A deploy goes out at 14:00. Error rate rises at 14:20, twenty minutes later. Why the delay, and what do you do first?

Do first: roll back. Not diagnose. The single most common mistake in real incidents is debugging while users are affected, because the engineer wants to understand it. Mitigate, then investigate, from a clean system, with the evidence you captured before rolling back (logs, a heap dump, a sample of failing requests).

Why the twenty-minute delay is plausible, and worth naming so the interviewer knows you are not assuming coincidence:

• A rolling deploy finished at 14:20, so the last pods only just took traffic; or the canary was small and only reached full traffic then.
• A cache with a 20-minute TTL masked the change until entries expired.
• A connection pool slowly cycled from old connections to new ones with different behaviour.
• A scheduled or batch job that runs every 20 minutes first exercised the new code path.
• A lazily-initialised code path — the first request to a rarely-used endpoint, or a migration-dependent query that only fires on a specific input.
• It is unrelated to the deploy. Correlation is not causation; check whether a dependency changed at 14:20 too. But roll back anyway: rollback is cheap and reversible, and if the error rate persists you have learned something valuable — namely that it was not you.