Production Self-Hosted Infrastructure: 162 Containers Across Three Hosts

One operator, three hosts, 162 containers in 78 compose stacks — Traefik, SSO, monitoring, mail, CI/CD and backups — and the two failures that changed how I verify my own numbers.

The Storage Manager card said 'running' all day and free space hadn't moved. 121 GiB had come back across three separate reclaims — and the afternoon ate 47 GiB of it.

A Seagate IronWolf 110 SSD threw WRITE FPDMA timeouts, failed mkfs with 'superblock magic doesn't match', and went silent — a textbook dying drive. I swapped one SATA cable and it formatted clean with zero errors. How to test before condemning a drive.

Two weeks after a btrfs RAID1 pool corrupted on healthy-looking NVMe drives, the same all-zero checksum reappeared on a different drive stack. The culprit was queued TRIM (NCQ) on Seagate IronWolf 110 SSDs — a firmware bug FreeBSD blacklisted but Linux never did. How I caught it and the fix.

My Unraid array stop hung forever at unmounting disks because an 8 GB btrfs swapfile pinned /mnt/cache. Root cause, the /boot/config/stop trap, and two User Scripts that fix it.

Why dropping CRF by five steps improved SSIM by only 0.00003: discovering the mathematical conflict between -maxrate bitrate ceilings and quality gates in high-throughput video re-encoding.

A btrfs RAID1 pool on two healthy-looking NVMe drives quietly corrupted a Windows VM. How I caught it with a checksum forensic — a zero-block CRC32C on the found side of every failure — and how the root cause finally resolved two weeks later: asynchronous TRIM, not the drives or the riser.

Why using -map 0 silently aborts Matroska muxers, drops subtitle fonts, and corrupts Apple playback: an empirical teardown of container constraints across multi-track media pipelines.

Why configuring an automated mail client for iCloud SMTP on port 465 results in silent connection timeouts (WSAETIMEDOUT), and why Apple's outbound mail architecture requires explicit STARTTLS on port 587.

My authentik forward-auth gate returned a clean 302 to the login page from the host itself, while the public URL still served the app. The hostname had two ingress layers, and I had only changed one.

A year's worth of authentik major-version upgrade pain: storage mount changes, RBAC session cleanup, trusted proxies, and the authorization_flow vs authentication_flow mix-up that broke SSO.

A container on my NAS kept stopping unexpectedly while Docker reported exit 0 and OOMKilled=false. The honest evidence was in dmesg: six cgroup OOM kills.

A self-hosted app whose front page was a Chinese login box — and a client-rendered SPA I couldn't edit. So I added a second container: nginx, a bilingual page, and three injected lines.

Why a self-hosted admin card failed with 'Unexpected token <': an O(n) library audit running inside a GET request, three reverse-proxy hops, and the 499 status no monitor watches — measured from 13.6 s on an idle box to 2,507 s on a busy one.

Loop engineering is 2026's term for designing the system that prompts and checks an agent. I applied it to self-hosted infrastructure, not a codebase: 11 cron jobs, 8 with no AI at all.

Three hosts, one Prometheus, one Grafana. The coverage matrix, the storage total that counted one NAS volume three times, and the VM that exports no CPU frequency at all.

Every target up, data in Prometheus, panel expressions correct — and all 41 panels reading No data. The culprit was a Grafana variable filtering on itself, plus the verification habit that hid it.

3% free sounds like an emergency until you notice it is 500 GB. On multi-terabyte volumes, percentage thresholds fire when nothing is wrong — and miss when something is.

Out of curiosity I mirrored my portfolio and git server onto the darknet. Here's the hardening that held, the parts that silently broke, and whether it's worth offering to clients.

A 1 TB restic seed stalled at zero bytes. The log named a Google project I did not own: rclone's shared OAuth client, quota pooled with every rclone user alive.

CVE-2026-109618 and CVE-2026-109619 are unauthenticated RCE in PHP itself, exploited before disclosure. What I measured across the infrastructure I manage — and the exact fix that closed it.

I migrated from paid RDPGuard 7.8.7 to open-source IPBan 4.1.0 — and hit three undocumented traps, including an uninstaller that would have silently unbanned 12 attackers.