Some links on this page are affiliate links: if you sign up through them, this site earns a commission at no extra cost to you. Prices and recommendations are not affected.
A post called “Docker is weird” hit the front page of Hacker News last week: 880 points, 369 comments. The author is writing their own container runtime and found out that containers are not a thing the kernel has, images are just folders, and runc needs a C module because Go has no usable fork. It is a good read and all of it is true.
This post is the other half of that story. Not why Docker is weird on the inside, but where that weirdness surfaces on a small production server at two in the morning. Seven behaviours, each one reproduced below on a live box running Docker 28.3.1 with the actual command output, not from memory. If you want the hardware this kind of box runs on, we compared the options in Best VPS for self-hosting in 2026; the stack itself is in Caddy and Docker in production.
1. A restart policy does not care that your container is unhealthy
This is the one that costs the most downtime, because it looks like you have covered yourself twice: a healthcheck and restart: always.
docker run -d --name dw1 --restart always \
--health-cmd 'exit 1' --health-interval 2s --health-retries 1 \
alpine:3 sleep 600
sleep 12
docker inspect -f 'status={{ .State.Status }} health={{ .State.Health.Status }} failing={{ .State.Health.FailingStreak }} restarts={{ .RestartCount }}' dw1
status=running health=unhealthy failing=5 restarts=0
Five consecutive failed checks, the container openly marked unhealthy, and zero restarts. A restart policy reacts to the main process exiting. A healthcheck only writes a status. Nothing in plain Docker connects the two.
So the application that has deadlocked its own worker pool, kept the port open and stopped answering will stay exactly where it is, with a green-ish docker ps line that says Up 3 days (unhealthy) in a column nobody reads. Swarm and Kubernetes do act on health. Compose on one box does not.
What to do instead: make the process exit when it is broken. If the app cannot do that, put the check outside Docker. A systemd timer that polls the endpoint and runs docker restart is five lines and it actually works:
curl -fsS --max-time 5 http://127.0.0.1:3000/health >/dev/null || docker restart app
Use the healthcheck for what it is genuinely good at: depends_on: condition: service_healthy, so a database is ready before the app connects.
2. A published port walks straight past ufw
You configured ufw. ufw status says only 22, 80 and 443 are allowed. Then you publish a port in Compose for a quick look at a dashboard, and that port is on the public internet.
Here is why, on the same box:
docker run -d --name pub -p 8099:80 nginx:alpine
iptables -t nat -S | grep 8099
iptables -S FORWARD | head -8
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 8099 -j DNAT --to-destination 172.17.0.5:80
-P FORWARD DROP
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-FORWARD
-A FORWARD -j ufw-before-logging-forward
-A FORWARD -j ufw-before-forward
...
Read the order. Traffic to a published port is DNAT-ed before any filtering decision you made, then handed to DOCKER-USER and DOCKER-FORWARD before the first ufw chain is reached. Docker accepts it, the packet is forwarded, and the ufw chains further down never see it. On this box DOCKER-USER is what it is on most boxes:
-N DOCKER-USER
Empty. That chain exists precisely so that you can put your rules somewhere Docker will not overwrite, and almost nobody uses it.
Three fixes, in order of how much they are worth:
- Bind to localhost.
-p 127.0.0.1:8099:80instead of-p 8099:80. Then reach it over an SSH tunnel. This is the right default for every admin interface. - Do not publish at all. Put the container on the reverse proxy’s network and let the proxy reach it by name on the internal port. Nothing in the stack behind our Caddy publishes a port.
- Write rules in DOCKER-USER if a port genuinely must be public and restricted:
iptables -I DOCKER-USER -i eth0 ! -s 203.0.113.0/24 -p tcp --dport 8099 -j DROP.
If you have just taken over a machine, this is item one on the sweep, and the rest of that sweep is in the audit of an inherited server.
3. Logs have no rotation until you add it
The default logging driver is json-file, and the default configuration is nothing:
docker inspect -f '{{ json .HostConfig.LogConfig }}' dw2
{"Type":"json-file","Config":{}}
No max-size, no max-file. The file under /var/lib/docker/containers/<id>/<id>-json.log grows until the disk is full. How fast? One container printing a short line in a loop, left running for twenty seconds:
docker run -d --name chat alpine:3 sh -c \
'i=0; while [ $i -lt 200000 ]; do echo "line $i some payload here"; i=$((i+1)); done; sleep 60'
# twenty seconds later
du -sh /var/lib/docker/containers/9ee39f60*/
19.0M /var/lib/docker/containers/9ee39f60.../
19 MB in twenty seconds from one container writing one short line per iteration. An application that logs every request at debug level, left alone over a long weekend, is a full root filesystem, and a full root filesystem on a single-box setup means the database stops accepting writes, the proxy cannot write its access log and TLS renewal fails.
Fix it globally, once, before you need it:
{
"log-driver": "json-file",
"log-opts": { "max-size": "10m", "max-file": "3" }
}
That goes in /etc/docker/daemon.json, followed by systemctl restart docker. It applies to containers created after the restart, not to existing ones, so recreate them. Per service in Compose:
logging:
driver: json-file
options:
max-size: "10m"
max-file: "3"
Thirty megabytes per container, bounded, and docker logs still works.
4. PID 1 is a shell, and your SIGTERM goes nowhere
Two containers, the same sleep, stopped the same way:
docker run -d --name dw2 alpine:3 sh -c 'trap "echo caught; exit 0" TERM; sleep 600'
time docker stop dw2
docker run -d --name dw3 --init alpine:3 sleep 600
time docker stop dw3
dw2 10.11s
dw3 0.09s
Ten seconds versus nine hundredths. And note that the slow one has a TERM trap: shells queue the trap until the foreground command finishes, so the handler is useless while sleep is running. docker stop sends SIGTERM to PID 1, waits out the grace period, then sends SIGKILL.
A hundred times out of a hundred this is a latency annoyance. The hundred and first time it is data loss, because SIGKILL is what your database, queue consumer or file writer gets instead of a shutdown sequence. It also makes every deploy ten seconds slower per container, which you will feel in a rsync-and-restart pipeline.
Three ways out, all cheap:
- Exec form in the Dockerfile:
CMD ["nginx", "-g", "daemon off;"], neverCMD nginx -g 'daemon off;'. execas the last line of an entrypoint script, so your process replaces the shell instead of being its child.init: truein Compose, which puts a tiny init as PID 1 to forward signals and reap zombies.
Then verify, because this is the kind of thing everyone believes they already did: docker exec <name> ps -o pid,comm and check what PID 1 actually is.
5. Container DNS only exists on networks you created
Same engine, two containers, two different resolvers. On the default bridge:
nameserver 192.168.178.1
nameserver 8.8.8.8
On a network created with docker network create:
nameserver 127.0.0.11
127.0.0.11 is Docker’s embedded DNS server, and it is the thing that makes postgres://db:5432 work. On the default bridge it is not there: the host’s /etc/resolv.conf is copied in and container names resolve to nothing. This is why docker run examples from a blog post cannot find each other while the same images in Compose can - Compose creates a project network for you.
It also means your service names are not a stable API. Legacy --link is gone, and the default bridge gives you IP addresses only, which brings us to the next one.
6. Compose hands you a different subnet than the one you whitelisted
docker compose up -d
docker network inspect app_default -f '{{ (index .IPAM.Config 0).Subnet }}'
# 172.19.0.0/16
docker compose down
docker network create decoy1 && docker network create decoy2
docker compose up -d
docker network inspect app_default -f '{{ (index .IPAM.Config 0).Subnet }}'
# 172.21.0.0/16
docker compose down deletes the project network. The next up creates a new one and takes the next free block, which is not the one you had. Two unrelated networks created in between were enough to move the project from 172.19 to 172.21.
Anything you pinned to the old subnet is now silently wrong: a Postgres pg_hba.conf line, an application allowlist, a firewall rule, an API that compares X-Forwarded-For against a CIDR. The failure arrives as “connection refused from unknown host” after a routine redeploy, which is the worst possible time to be reading pg_hba.conf.
Either stop pinning subnets and use container names on a user-defined network, or declare the subnet so it cannot drift:
networks:
default:
ipam:
config:
- subnet: 172.28.0.0/24
7. An OOM kill does not look like a crash
Give a container 32 MB and let it allocate:
docker run --rm -m 32m alpine:3 sh -c 'a=""; while true; do a="$a$(head -c 1000000 /dev/zero | tr "\0" x)"; done'
echo $?
137
Exit code 137 is 128 + 9: SIGKILL. Not your code, not an exception, no stack trace, nothing in the application log. The cgroup hit its limit and the kernel killed the biggest process inside it.
And here is where the Hacker News post and this one meet. The kernel is not killing “a container”, it is killing an ordinary process it can see the whole time:
hp=$(docker inspect -f '{{ .State.Pid }}' dw4)
ps -o pid,cmd -p $hp
grep NSpid /proc/$hp/status
PID CMD
1085777 sleep 600
NSpid: 1085777 1
One process, two numbers: 1085777 on the host, 1 inside. “Containers are not real” is not a provocation, it is the thing you are looking at when you debug an OOM kill. The container’s PID 1 is in your host’s process table, and that is where you find it when docker logs has nothing because the process never got to write a line.
So when a container disappears without explanation: read the exit code first (docker inspect -f '{{ .State.ExitCode }} {{ .State.OOMKilled }}'), then dmesg -T | grep -i oom, then set a memory limit on every service so the kernel kills the container that misbehaved instead of whatever your host had running at the time.
The checklist
Everything above, as the diff you can apply to a stack you already run:
services:
app:
image: app:1.4.2 # not :latest
init: true # PID 1 forwards signals
restart: unless-stopped
mem_limit: 512m # the kernel kills this, not your database
ports:
- "127.0.0.1:3000:3000" # never a bare host port
logging:
driver: json-file
options:
max-size: "10m"
max-file: "3"
healthcheck:
test: ["CMD", "wget", "-qO-", "http://localhost:3000/health"]
interval: 30s
depends_on:
db:
condition: service_healthy
networks:
default:
ipam:
config:
- subnet: 172.28.0.0/24 # so a redeploy cannot move it
Plus two things that live outside the file: max-size in /etc/docker/daemon.json for anything you start by hand, and an external watchdog for the health status Docker will not act on.
None of this is exotic. It is all default behaviour, documented, reproducible in twenty seconds each, and every item on the list has taken a small production server down for somebody this month. The reason it keeps happening is that the defaults are tuned for a laptop, where a 19 MB log file and a ten-second stop are free.
If you want the rest of the stack that these settings belong to, the full config of the box serving this page is in Caddy and Docker in production, and the measurement side is in self-hosted analytics with Umami.