In April I wrote a whole post defending s6 as the init system for daemonless images. I called it the most efficient, C-based babysitter in the business, and ended with a challenge:

s6 is the right init system for FreeBSD containers.

If you disagree — different init system, cleaner pattern, something I missed — make your case on GitHub Discussions. I’m genuinely open to being convinced.

Seven months later, one app image in the fleet still runs on s6. Kubernetes made the case, and a test I had already run in that post made it worse.

s6 Did Nothing Wrong

Nothing broke. s6 ran the fleet for most of a year, and it did what I said it did. If you are running containers under podman on one box, a supervisor inside the container that restarts a crashed app is a perfectly good feature.

The trouble starts when something outside the container wants the same job. Doug Rabson has been building Kubernetes on FreeBSD (CRI-O, kubelet and ocijail in kubernetes-demo), so I stood up a test node and ran our s6 images on it. Two results settled it.

s6 hid crashes. I ran postgres in a pod without the SysV IPC permission it needs, so it could not start. Under s6 the pod showed Running and Ready while postgres crash-looped inside it. s6-supervise was doing exactly what I had praised it for, catching the crash and restarting the app. Kubernetes saw a healthy container, because the process it was watching was s6, and s6 was fine.

s6 could not run as a normal user. Kubernetes likes runAsUser, and so do I. Here is the image as it still ships today:

$ podman run --rm --user 1000:1000 ghcr.io/daemonless/base:15
[init] Starting container initialization...
mkdir: /run/s6/container-environment: Permission denied

To be fair to s6, that mkdir is in my /init, not in s6. The s6 side could be made to work too, and I spent a while trying. I added a DAEMONLESS_SUPERVISE=orchestrator knob that tore down the supervision tree when a service died, and it worked. But by then I was writing code to turn off the thing I had installed s6 for, in the one mode I cared most about.

Same Container, Then and Now

The first post opened with the process tree of a live MariaDB container, so here is that comparison. April:

  PID  PPID USER COMMAND
38492 38490 root s6-svscan /run/s6/services
38706 38492 root s6-supervise mariadb
38707 38492 root s6-supervise mariadb/log
38708 38706 bsd  /usr/local/libexec/mariadbd --datadir=/config/databases ...
38709 38707 bsd  s6-log T n10 s1048576 /config/logs/daemonless/mariadb 1

Today:

  PID  PPID USER COMMAND
66567 66565 bsd  /usr/local/libexec/mariadbd --defaults-extra-file=/config/custom.cnf --datadir=/config/databases ...

I was proud of “four processes” in April. One is better.

If you are wondering who reaps zombies with no init left, a jail has no PID 1 of its own. Orphans reparent to conmon, which reaps them. I killed the parents of five children to check and got zero zombies. Nothing in the image needs to do that job.

base-exec

The new image is base-exec. /init is still a shell script and still does the setup: loopback, PUID/PGID remapping, an image’s init scripts. Then, instead of exec s6-svscan, it ends like this:

if [ "$uid" = 0 ]; then
	exec dsu "${DAEMONLESS_USER:-bsd}" "$@"
fi
exec "$@"

The app replaces the script. Signals from the runtime go straight to the app, and the app’s exit status becomes the container’s. Started as root, it drops to bsd with dsu, a small C replacement for s6-setuidgid. Started as another user, it skips the root-only steps and runs the command as that user.

The app’s start script is now the image’s CMD, and a crash is finally visible from outside:

$ podman exec postgres sh -c 'kill -9 <postmaster pid>'
$ podman inspect postgres --format '{{.State.Status}} {{.State.ExitCode}}'
exited 137

I Was Wrong About Postgres

The centerpiece of the April post was postgres shutdown. podman stop sends a signal to PID 1, postgres needs SIGINT to do a fast shutdown, and s6-svscan sat in the middle. The fix took two files: STOPSIGNAL SIGINT in the Containerfile, and a down-signal file telling s6-supervise to pass INT along. I called it “the level of precision you get from raw s6 that you simply cannot achieve with a naive shell entrypoint.”

Read that again with s6 removed. The only reason down-signal existed was that s6 was standing between podman and postgres. The naive entrypoint I was warning about had a problem only because it did not set STOPSIGNAL, and the s6 version had to set it too.

So here is the current image, one line in the Containerfile, with a client sitting in a SELECT:

$ podman exec -d postgres psql -U postgres -h 127.0.0.1 -c 'select pg_sleep(300)'
$ time podman stop postgres      # 0.97s
$ podman inspect postgres --format '{{.State.ExitCode}}'
0

Under a second, exit 0, no supervisor. I also wrote that a SIGKILL leaves you with a corrupted database, which overstated it. Postgres does crash recovery from the WAL on the next start. It’s still not how you want to stop a database, but it isn’t corruption.

Some Containers Really Are Several Things

One process per container is the right model, but not every image fits it. Vaultwarden behind nginx, or an appliance bundling its own database, wants a service manager. Splitting the easy ones into stacks helped, and the rest still needed something.

In April I had a section called “Why Not rc.d?” The objections were that rc.d daemonizes and that nothing restarts a crashed service. Baptiste Daroussin’s rcd(8) answers both. It is a new FreeBSD service manager with declared units, dependencies and restart policies, and base-rcd is built around it:

  PID  PPID USER COMMAND
68160 68158 root /sbin/rcd -F
68265 68160 bsd  /usr/local/bin/vaultwarden
68266 68160 root nginx: master process /usr/local/sbin/nginx -c /config/nginx/nginx.conf (nginx)
68267 68266 bsd  nginx: worker process (nginx)
...

A service is a small UCL file instead of a directory of magic files:

name = "vaultwarden";
provides = ["vaultwarden"];
type = "simple";
enable = true;
command = "/daemonless/libexec/vaultwarden";
process { env_file = "/run/rcd.env"; }
restart { policy = "always"; max_retries = 0; delay = 5000; backoff = "none"; }

This one comes with an asterisk. rcd is not in a FreeBSD release, so base-rcd builds it from a pinned commit of his branch and carries a patch of ours, including the -F foreground flag you can see above. It also has no way yet to exit when a service fails for good, which is the same crash-hiding problem I just blamed s6 for. Those images are honest about being appliances, but I would like that fixed upstream. I am betting on where FreeBSD service management is going, and I would rather carry a small patch on that than keep building on something outside the base system.

What It Cost

Moving on was not free, and you may notice some of these.

  • No restart in place. A crashed app now stops the container, and it comes back only if you set a restart policy. Set one.

  • No more file logs. /config/logs/daemonless/<service>/ and those TAI64N filenames are gone from migrated images. Logs go to stdout. podman logs is the interface now, and it resets when the container is recreated. I will not miss decoding @4000000069bbb7ad38b6544e, but persistent logs were a real feature.

  • Hung is not crashed. If an app wedges without exiting, nothing notices. Kubernetes has liveness probes. Podman’s scheduled healthchecks depend on systemd timers, so on FreeBSD they never fire by themselves: when I tested in September, a failing check sat at starting forever unless something ran podman healthcheck run. This is the hole I most want to close.

  • Non-root is not done. --user 1000 works on what I’ve tried. An arbitrary UID gets through /init fine, but images can still trip over their own setup. Redis as --user 4242, today, with a volume it owns:

    [init] Running: /daemonless/init.d/20-init-redis
chown: /config/redis.conf: Operation not permitted

    That is a leftover chown in one script, the kind of thing that hides in a fleet this size, and it isn’t the last one.

The images did shrink, but not by anything you’d brag about: base is 231 MB, base-exec is 222 MB, and base-rcd is 225 MB. The nicer win is the build. base has to compile skalibs, execline and s6 from ports just to carry a FreeBSD 14 compatibility patch, and base-exec doesn’t. It took 52 seconds against 23 in one run here.

Where It Landed

  • 89 single-process services in the catalog run on base-exec.
  • 18 appliance services in the catalog run on base-rcd, along with a few images that aren’t in it.
  • booklore is the one app image still on s6.

base keeps building with s6, so anything built on it keeps working. App images moved one repo at a time, and their existing tags picked up the change on the next build. If you were reading those log files, that is when they stopped.

The Kubernetes side is what I wanted to know about. On November 1 I ran the fleet through dbuild’s tests on a single-node FreeBSD 15.1 cluster using Doug’s stack: 134 images, 130 passed. One of the two failures is a test fixture the registry refused to serve, the other is a screenshot check that came in a hair under its threshold on a page that matches its baseline, and two images have no test configured. Those runs were as root with PUID/PGID, today’s default contract, so they don’t say anything about the non-root gap above.

The Verdict

The April post ended by saying daemons are welcome as long as they stay inside the jail and listen to their supervisor. It turns out the supervisor should be outside the jail. Podman or Kubernetes already has that job, and a second one inside the container was mostly arguing with it.

So the name is a bit less ironic now. Most daemonless containers are one daemon and nothing else, with no babysitter. The scooters from April are mostly gone. A few appliances still get around on wheels, and those wheels are at least headed for the FreeBSD base system.

Change My Mind, Again

One process per container, and rcd for the images that can’t be.

It worked last time. If you think I’ve got it wrong, make your case on GitHub Discussions.