The staff pressure round

Requirements say this must be strongly consistent. Making it eventually consistent would cut cost by 70% and latency by half. What do you do?

The wrong moves are symmetrical: implementing it as stated without asking, and quietly deciding the PM is wrong.

The right move is to decompose the requirement, because "strongly consistent" is almost never uniformly true across a system. For inventory specifically:

• Browsing a product page: showing "12 in stock" that is three seconds stale is harmless. Nobody has been hurt.
• Adding to a cart: still fine — carts are not reservations, and everyone accepts that items can become unavailable.
• Checkout / decrementing stock: this must be strongly consistent, or you oversell. But it is a tiny fraction of traffic.

So the answer is not "strong" or "eventual", it is strong where it is load-bearing, eventual everywhere else — which typically captures most of the 70% saving because the read path is 99% of the volume. Reads go to regional replicas; the decrement is a conditional write against the primary (UPDATE ... WHERE qty >= n), which is a single atomic statement and needs no distributed transaction.

Then go and find out what the requirement was protecting. Ask the PM: "what happens if a customer sees an item as available and it turns out not to be at checkout?" The answer might be "nothing, we show a message" — or it might be "we have a contractual SLA with sellers about overselling and it costs us $X per incident". That second answer would change the design, and it is exactly the kind of context that never makes it into a requirements document.

Design the migration from the old system to the new one, with no downtime, while it is serving 4,000 requests per second.

The pattern is expand / migrate / contract, and the parts people forget are the shadow-read verification and the ability to abort at every stage.

Stage 0 — stop the bleeding. No new direct readers. Put the existing access behind a client library or an API in the monolith so that later stages have one place to change, not twelve.

Stage 1 — dual write. Every write goes to both old and new stores. The safe way is a transactional outbox: write the row and an event in one local transaction, and a consumer applies it to the new store. Dual-writing directly from application code to two systems is not atomic — a crash between them silently diverges the data, and you will not know until a customer complains.

Stage 2 — backfill. Batched, resumable, idempotent, throttled by replica lag. It must be safe to run concurrently with live dual writes, which means upserts keyed on the primary key and a rule that live writes always win over backfill writes (compare timestamps or use ON CONFLICT DO NOTHING for backfill).

Stage 3 — verify with shadow reads. This is the stage that distinguishes people who have done this. Read from both stores on every request, return the old result, and log any divergence with enough context to debug. Run it for at least a full business cycle — a week, ideally a month-end. You will find divergences, and every one is a bug you would otherwise have shipped: timezone handling, null-vs-empty-string, truncation, a field the backfill derived differently from the live path.

Stage 4 — cut over reads, gradually. 1% → 10% → 50% → 100%, per-service or per-tenant, with a flag you can flip back in seconds. Watch error rate and latency at each step, and hold at each level long enough to see a full traffic pattern.

Stage 5 — stop writing to the old store, but keep it for a defined period. Then contract: delete the old code, then the old data, in separate releases weeks apart.

You have three weeks and the design you would want takes three months. What do you cut?

The framework: sort every element of the design by how expensive it is to change later, then spend your three weeks on the expensive ones and defer everything else.

Do not cut (expensive or impossible to change later):

• The data model and its identifiers. Getting a primary key or a tenancy boundary wrong is a migration for every future engineer. Put tenant_id on every table today even if you are single-tenant, and use a sortable id.
• The API contract, if external clients will consume it. You can change internals forever; you cannot un-ship a field that a mobile app in the wild depends on.
• Anything that touches money, permissions or personal data. Correctness and auditability here are not features you retrofit; they are constraints that shape the schema.
• Basic operability: structured logs with a request id, a health check, and one dashboard. Three weeks of work you cannot observe is not three weeks of progress.

Cut freely (cheap to change later):

• Horizontal scale. One process and one database until proven otherwise.
• Caching. Add it when a measurement demands it; a cache added early is a consistency bug added early.
• The queue, unless the work genuinely cannot happen in the request. A after_response hook or a cron is fine at first.
• Microservice boundaries. Start as a module inside the monolith with a clean interface; extracting later is straightforward, merging back is not.
• Admin tooling, dashboards, the second auth method, the perfect test pyramid. Write the tests for the money path and the permission path; skip the rest for now.

Then the part that is not technical: say the cuts out loud, in writing, with what would trigger revisiting them. "No cache; we will add one when p95 exceeds 400 ms" is a decision. Silently skipping the cache is a landmine, and the difference is entirely in whether it was written down.

I disagree with your database choice. Convince me, or change your mind.

The failure modes first, because they are what the interviewer is watching for. Folding immediately ("oh, sure, Cassandra then") says you will not defend a design in a real meeting and your technical opinions cannot be relied on. Digging in ("no, Postgres is fine") without engaging says you cannot be updated by new information, which is worse in a colleague than being wrong.

The move is to convert the disagreement into a question about the requirement, because "which database" is downstream of numbers you may not both have:

"That might be right — it depends on numbers I do not have yet. What write rate are we designing for, and does the access pattern have a natural partition key? My reasoning for Postgres was: at 5,000 writes per second with a relational access pattern and a team of six, a single primary with replicas handles it comfortably, and I get transactions, joins and ad-hoc queries — which we will need because the product is still changing. Cassandra buys linear write scaling and no single point of failure, but it costs me joins, ad-hoc queries and multi-row transactions, and it requires the query patterns to be fixed at table-design time. If we are at 200,000 writes per second, or we need multi-region active-active, that trade is worth it and I would switch. Which are we?"

That answer does three things: it states the reasoning rather than the conclusion, it names what would change your mind specifically and falsifiably, and it hands the interviewer a genuine question rather than a defence.

Then actually update if they give you the number. If they say 200,000 writes per second with a natural partition key, say so plainly: "That changes it — at that rate a single primary is not viable, and if the access pattern is partitionable, Cassandra or Scylla is the right call. What I would still want is..." Changing your mind with a stated reason is the strongest possible signal in this exchange.

Your service is the bottleneck for six other teams. They all want different things. What do you do?

Treat it as a product problem, because it is one.

1. Find out what they actually need, not what they asked for. Six feature requests frequently decompose into two underlying needs plus four workarounds for a missing capability. The teams are describing solutions; your job is to extract the problems.
2. Make the current state visible. Publish what the service does, its limits, its latency, and its roadmap. A large fraction of "requests" are actually questions, and a good docs page removes them permanently.
3. Prioritise by aggregate impact, transparently. A public queue with the reasoning attached means teams argue with each other about relative priority rather than lobbying you individually — which is both fairer and much less exhausting.
4. Enable rather than serve, where you can. If four teams need custom filtering, one extension point beats four bespoke endpoints and eliminates you from their critical path. The goal for a platform team is to be unblockable, not to be responsive.
5. Say no clearly and early. "Not this quarter, here is why, here is what would change it" is far kinder than an indefinite maybe. Teams can plan around a no; they cannot plan around silence.
6. Escalate the systemic problem. If six teams are blocked on one service, the constraint is probably staffing or architecture, and that is a decision for someone with budget. Present it with data — blocked team-weeks, not adjectives.

Draw the architecture you would need if traffic grew 100x tomorrow. Then tell me why you would not build it.

Do the first half properly, because you must demonstrate you can: CDN and edge caching for static and cacheable responses; a stateless, horizontally-scaled application tier behind an L7 load balancer; read replicas and then sharding by tenant with a lookup directory; a partitioned event log (Kafka) for asynchronous work; a distributed cache with jittered TTLs and stampede protection; search and analytics in dedicated stores fed by change data capture; multi-region for latency and failover; the operational layer of tracing, per-tenant quotas and load shedding.

Then the second half, which is the actual question:

• The traffic will not grow 100x tomorrow. If it grows over two years, you will get to make each decision with information you do not have now — and you will make better decisions than the ones you would make today.
• Every element above has a running cost in engineering time, not just money. Sharding means every future feature considers cross-shard behaviour. Kafka means someone is on call for Kafka. Multi-region means every consistency decision gets made twice.
• It slows down the thing that determines whether you ever need it. Premature distribution is the most reliable way to make a small team slow, and a slow team ships fewer features, and fewer features means less growth.
• Most of it can be added later without a rewrite, provided you preserve the options: keep the app tier stateless, put a tenant key on every table, keep write paths idempotent, and do not let business logic leak into the database. Those four choices cost nothing now and preserve every scaling path.