Docker, CI/CD, Linux & observability

Write a production Dockerfile for a Python service and explain each decision.

FROM python:3.12-slim AS builder
WORKDIR /app
RUN pip install --no-cache-dir uv
COPY pyproject.toml uv.lock ./
RUN uv sync --frozen --no-dev            # dependencies before source: this layer caches

FROM python:3.12-slim
RUN useradd --create-home --uid 10001 app     # never run as root
WORKDIR /app
COPY --from=builder /app/.venv /app/.venv     # only the artefact, not the toolchain
COPY --chown=app:app . .
ENV PATH="/app/.venv/bin:$PATH" \
    PYTHONUNBUFFERED=1 PYTHONDONTWRITEBYTECODE=1
USER app
EXPOSE 8000
HEALTHCHECK --interval=30s --timeout=3s CMD python -c "import urllib.request;urllib.request.urlopen('http://localhost:8000/healthz')"
CMD ["gunicorn", "-k", "uvicorn.workers.UvicornWorker", "-w", "4", "-b", "0.0.0.0:8000", "app:app"]

The reasoning, which is the actual answer: layer ordering (dependencies change rarely, source changes constantly — copying source first invalidates every cached layer); multi-stage so compilers and dev dependencies never ship; slim/distroless base for a smaller attack surface (alpine's musl can break wheels and is often slower for Python); non-root user; PYTHONUNBUFFERED so logs appear immediately; and a .dockerignore excluding .git, .venv, tests and secrets — without it you copy your entire history into the image.

What is the difference between an image and a container, and between CMD and ENTRYPOINT?

An image is an immutable stack of read-only layers plus metadata; a container is a running instance with a thin writable layer on top. Deleting a container discards that layer — hence volumes for anything that must persist.

ENTRYPOINT is the executable; CMD provides default arguments (and is replaced by anything passed on the command line). The idiomatic pair is ENTRYPOINT ["gunicorn"] + CMD ["app:app"]. Use exec form (["a","b"]) not shell form, or your process runs as a child of /bin/sh and never receives SIGTERM — which is why containers that "won't stop gracefully" take 10 seconds to be killed.

How do you reduce image size and build time?

Multi-stage builds; a slim base; --no-cache-dir and cleaning apt lists in the same RUN (a separate RUN rm does not shrink the earlier layer); a strict .dockerignore; BuildKit cache mounts (RUN --mount=type=cache,target=/root/.cache/pip); combining related RUN commands; and copying only what you need. Verify with docker history and dive. Typical result: a 1.2 GB naive Python image becomes 150-250 MB.

Explain container networking and volumes at the level you would debug them.

Each container gets a network namespace. On the default bridge, containers reach each other by IP; on a user-defined network they resolve each other by service name via Docker's embedded DNS — that is why docker compose services can talk using postgres:5432. -p 8080:80 maps a host port to a container port via NAT. host networking skips the namespace (Linux only).

Volumes: named volumes are Docker-managed and the right choice for databases; bind mounts map a host path (great for development, a permissions minefield in production); tmpfs is memory-only. The classic debugging chain: docker compose logs -f svc, docker exec -it svc sh, docker inspect svc, docker network inspect.

Design a CI/CD pipeline.

push → lint + typecheck → unit tests → integration tests (service containers)
     → build image, tag with the git SHA → scan (trivy/grype) → push to registry
     → deploy to staging → smoke tests → manual approval → canary 5% → monitor → 100%

Principles worth stating: build once, promote the same artefact through environments (never rebuild per environment — you would be deploying something you did not test); tag by immutable git SHA, not latest; keep secrets in the CI provider's store, never in the repo; make the pipeline fast enough that people do not batch changes; and ensure every step is reproducible locally.

Compare deployment strategies.

• Rolling — replace instances gradually. Default, no extra cost, but two versions run at once, so the database schema and API must be compatible across both.
• Blue-green — a full parallel environment, switch traffic at the load balancer, instant rollback. Doubles infrastructure cost briefly; the database is still shared, which is the hard part.
• Canary — 1% → 5% → 25% → 100%, watching error rate and latency at each step, automatic rollback on regression. The safest for high-traffic services; needs good metrics to be meaningful.
• Feature flags — deploy the code dark and release the behaviour separately, per user segment. Decouples deploy risk from release risk, and gives you an instant kill switch. The strongest answer usually combines rolling deploys with flags.

Always mention that database migrations must be backwards-compatible (expand/contract) because during any of these, old and new code run simultaneously.

Which Linux commands do you use to debug a production issue?

top / htop            # load, CPU, memory at a glance
ps aux --sort=-%mem   # who is using memory
df -h / du -sh *      # disk full is the single most common outage cause
free -h               # memory, and how much is cache
journalctl -u svc -f --since "10 min ago"   # service logs
ss -tulpn             # listening sockets and their processes (netstat's replacement)
curl -sv -o /dev/null -w '%{time_total}\n' http://localhost:8000/healthz
dig +short api.example.com                  # DNS is always a suspect
strace -p PID -f -e trace=network           # what syscalls is it stuck on
lsof -p PID           # open files/sockets — file descriptor exhaustion
tail -f app.log | grep -i error
iostat -x 1 / vmstat 1                      # disk saturation, context switches, steal time

A sensible triage order to describe: is it up (process/health check)? → is it resourced (CPU, memory, disk, file descriptors)? → is it reachable (DNS, port, TLS, firewall)? → what changed (deploy, config, traffic, a dependency)? "What changed?" resolves most incidents.

What are the Kubernetes concepts you must know?

Pod (one or more containers sharing a network namespace, the unit of scheduling) · Deployment (declarative replica management and rolling updates) · Service (a stable virtual IP and DNS name load-balancing to pods) · Ingress/Gateway (HTTP routing and TLS) · ConfigMap/Secret · StatefulSet (stable identity and storage, for databases) · DaemonSet (one per node, for agents) · Job/CronJob · HPA (autoscaling on metrics) · PDB (how many pods may be down during maintenance).

The two things that actually cause incidents: probes — liveness restarts a pod (a wrong one causes restart loops; never make it check a dependency), readiness removes it from the Service (this is the one that matters for zero-downtime deploys), startup covers slow boots — and resource requests/limits: requests drive scheduling, exceeding a memory limit gets you OOMKilled, and CPU limits cause throttling that looks like mysterious latency.

What do you monitor, and what is the difference between metrics, logs and traces?

Metrics are cheap aggregated numbers over time (rate, latency histograms, saturation) — for dashboards and alerts. Logs are discrete events with detail — for investigating a specific request; make them structured JSON with a request id. Traces follow one request across services with timing per span — for finding where latency lives in a distributed call graph. Together they are the "three pillars", correlated by trace/request id.

What to alert on: the four golden signals — latency (p50/p95/p99, never the mean), traffic, errors, saturation — or the RED method (Rate, Errors, Duration) for services and USE (Utilisation, Saturation, Errors) for resources. Alert on symptoms users feel (error rate, p99 latency, queue age) rather than causes (CPU at 80% is not an incident). Every alert must be actionable and have a runbook; a paging alert nobody acts on trains people to ignore the pager.

What are SLIs, SLOs and error budgets?

An SLI is a measured indicator (the fraction of requests served successfully under 300 ms). An SLO is the target (99.9% over 30 days). The error budget is the allowed shortfall — 0.1%, or about 43 minutes a month.

The value is that it turns reliability into a shared, quantitative decision: while the budget has room, ship features; when it is exhausted, the team stops feature work and spends it on reliability. It also stops the "everything must be 100%" conversation — chasing an extra nine costs real money, so the business chooses the target, not the engineers alone.