What a container actually is

What is a container, precisely?

A normal Linux process, started with some kernel features switched on. There is no container object in the kernel. docker run is: create namespaces, configure cgroups, set up a root filesystem, drop capabilities, apply a seccomp filter, then execve your binary. Three primitives do the work:

Everything else — capabilities, seccomp, AppArmor/SELinux, user namespaces — is security hardening on top. Because it is just a process, a container shares the host kernel: there is no guest OS, startup is milliseconds, overhead is near zero, and the isolation boundary is the kernel's syscall interface rather than a hypervisor. That last point is the security trade-off, and it is why gVisor, Kata Containers and Firecracker exist for genuinely untrusted workloads.

What do the namespaces actually do?

Seven of them, each virtualising one global kernel resource:

• PID — your process becomes PID 1 in its own tree. It cannot see or signal host processes.
• Mount (mnt) — its own filesystem view; this is what makes the container's root look like a whole system.
• Network (net) — its own interfaces, routing table, iptables rules, and port space. This is why two containers can both bind :8080.
• UTS — its own hostname.
• IPC — its own shared memory and semaphores.
• User — UID mapping, so root inside (UID 0) maps to an unprivileged UID outside. The strongest isolation feature, and still not on by default in many setups.
• Cgroup — hides the host's cgroup hierarchy.

unshare --pid --fork --mount-proc /bin/bash    # a "container" in one command
ls /proc/$$/ns/                                # every process's namespace links

PID 1 is where this bites in practice. In Linux, PID 1 has special semantics: signals without an explicit handler are not applied to it — the kernel's default actions do not apply — and it is responsible for reaping orphaned children. Consequences:

1. If your app does not explicitly handle SIGTERM, it ignores it, so docker stop and Kubernetes graceful termination wait the full grace period and then SIGKILL. Every in-flight request dies. This is a very common cause of dropped requests on every deploy.
2. CMD npm start in shell form runs /bin/sh -c npm start, so the shell is PID 1 and does not forward signals to your app at all. Use exec form: CMD ["node", "server.js"].
3. Processes that spawn children need an init to reap zombies — docker run --init, or tini.

How do cgroups produce OOMKilled and CPU throttling?

cgroups v2 organises processes into a hierarchy with controllers per resource.

Memory. memory.max is a hard limit on the cgroup's total charge — anonymous memory, page cache, kernel memory, tmpfs. Exceeding it invokes the cgroup OOM killer, which kills the largest process: exit code 137 (128 + SIGKILL). Two things follow:

• Your runtime does not know about the limit. /proc/meminfo still reports the host's memory, because there is no memory namespace. So a JVM without MaxRAMPercentage, Node without --max-old-space-size, or Go without GOMEMLIMIT sizes its heap for the host and gets killed before it ever runs a full GC. Modern JVMs and Go are container-aware; Python and Node are not by default.
• Page cache counts. Writing large files to an emptyDir or doing heavy I/O charges the cgroup, so a process with a small heap can still be OOMKilled. Under pressure the kernel reclaims cache first, so this usually shows up as mysterious I/O slowness before it shows up as a kill.

CPU works completely differently, and conflating the two is the classic mistake. cpu.max is a quota per period — typically 100 ms — enforced by the CFS scheduler. Exceeding the quota does not kill anything; the cgroup is throttled: every runnable thread is stopped until the next period begins.

That produces a signature pathology. A container limited to 0.5 CPU with a multi-threaded runtime can burn its entire 50 ms quota in the first 10 ms of a period, then sit frozen for 90 ms — so average utilisation looks like 30% while p99 latency is dreadful. Check container_cpu_cfs_throttled_seconds_total, not average CPU.

• Requests vs limits in Kubernetes map onto this: requests becomes cpu.weight (a share used for scheduling and for contention) while limits becomes cpu.max (a hard ceiling). Memory requests drive scheduling; memory limits kill.
• Many teams set CPU requests but no CPU limits, precisely to avoid throttling a latency-sensitive service that is otherwise well-behaved. Memory limits, by contrast, you almost always want, because memory is incompressible.
• Runtimes read the host CPU count unless told otherwise: Go's GOMAXPROCS and thread pools sized by nproc will create far too many threads for a 0.5-CPU container, causing context-switch overhead and throttling. automaxprocs or explicit configuration fixes it.

How do image layers and the union filesystem work?

An image is an ordered stack of read-only layers, each a tarball of filesystem changes, addressed by content hash. At runtime, overlayfs presents them as one filesystem with a thin writable layer on top.

• Reading finds the file in the topmost layer that has it.
• Writing to a file from a lower layer triggers copy-up: the whole file is copied into the writable layer first. Writing one byte to a 2 GB file copies 2 GB. This is why databases in containers must use volumes, and why heavy write workloads on the container filesystem are slow.
• Deleting a lower-layer file writes a whiteout marker. The data is still in the image. So RUN rm secrets.txt in a later layer does not remove it — anyone with the image can extract it from the earlier layer. Secrets must never be in any layer; use build secrets (RUN --mount=type=secret) or runtime injection.

The layer model explains the build rules people follow by rote:

COPY package*.json ./          # changes rarely  → this layer stays cached
RUN npm ci                     # expensive       → cached with it
COPY . .                       # changes always  → only this layer and below rebuild

Copying source before installing dependencies invalidates the dependency layer on every code change, which is the single most common cause of slow builds. Layers are also shared: ten images on the same base share one copy on disk and one pull.

And the reason RUN apt-get install … must clean up in the same RUN: each instruction is a layer, so a separate RUN rm -rf /var/lib/apt/lists/* adds a whiteout without shrinking the earlier layer.

What is the security boundary, and how do you harden it?

The boundary is the syscall interface. A kernel vulnerability reachable from a container is a host compromise, which is the fundamental difference from a VM.

Layers of defence, roughly in order of value:

1. Do not run as root. USER 10001. Root in a container is root on the host for anything that escapes namespace protection, and it makes several escape techniques trivial.
2. User namespaces — map container UID 0 to an unprivileged host UID, so even "root" inside is nobody outside. This is what rootless Podman does by default.
3. Drop capabilities. Docker already drops most, keeping ~14 including NET_BIND_SERVICE and CHOWN. --cap-drop=ALL and add back only what you need. CAP_SYS_ADMIN is effectively root — never grant it.
4. --privileged disables essentially all of this. It is the answer to "how did they escape the container?" often enough to be worth flagging in review every time you see it.
5. seccomp filters syscalls. Docker's default profile blocks ~44 dangerous ones. Custom profiles are better; the default is already a big win, and --security-opt seccomp=unconfined throws it away.
6. Read-only root filesystem (--read-only plus a tmpfs for scratch), no new privileges (no-new-privileges), and drop the Docker socket — mounting /var/run/docker.sock into a container is equivalent to giving it root on the host.
7. Scan images (Trivy, Grype), use minimal bases (distroless, Alpine, or a scratch image with a static binary — less code means fewer CVEs), and pin base images by digest so a rebuild is reproducible.

Why does "works on my machine" still happen?

Containers eliminate several classes of it and not others. The remaining causes, all of which are worth being able to name:

• CPU architecture. An image built on an Apple Silicon laptop is linux/arm64; the cluster is amd64. Symptom: exec format error or no matching manifest. Fix: docker buildx --platform linux/amd64,linux/arm64.
• The kernel is the host's. Anything depending on kernel version, modules, or /proc and /sys details differs between your laptop's kernel and the node's. Docker Desktop on macOS runs a Linux VM, so your "local Linux" is not the production Linux.
• Mutable tags. :latest or :v1 pulled at different times gives different code. Deploy by digest or immutable version tag.
• Build-time versus run-time configuration. Environment differences, mounted secrets, ConfigMaps, and network policy exist only in the cluster.
• Resource limits. Locally it has 16 GB and 8 cores; in production it has 512 MiB and 0.5 CPU — and the runtime may still believe it has the host's resources, as above.
• Filesystem semantics. Case sensitivity (macOS is case-insensitive by default, Linux is not), file ownership through bind mounts, and permission differences after copy-up.
• Time and locale, and a container with no tzdata behaving differently from your machine.

What should you actually know about container networking?

Each container gets a net namespace with a veth pair — one end inside, one on a host bridge. Docker's default bridge network NATs outbound traffic and uses -p 8080:80 to DNAT inbound via iptables. On a user-defined network, containers resolve each other by name through an embedded DNS server, which is why Compose services can talk using postgres:5432.

In Kubernetes the model is different and worth stating clearly: every pod gets its own IP, and every pod can reach every other pod without NAT. Containers within a pod share one network namespace, so they communicate over localhost and cannot both bind the same port — that is what makes sidecars work. A CNI plugin (Calico, Cilium) implements the flat network, and kube-proxy (iptables, IPVS, or eBPF via Cilium) turns virtual Service IPs into pod IPs.

Debugging follows directly: nothing listens on a Service ClusterIP (it is a virtual IP rewritten by packet-filtering rules), so ping to it fails harmlessly; an app bound to 127.0.0.1 inside the container is unreachable from outside its own namespace and must bind 0.0.0.0; and a NetworkPolicy is enforced by the CNI, so a default-deny namespace silently drops traffic your application never sees.