📌 Part of Blueprint: Production Self-Hosted Infrastructure → L3: Observability, Loops & Self-Healing

Tracking Down 14 MB/s Ghost Uploads Across a 2.5GbE NAS Interconnect: The Media Server Trickplay Trap

· 5 min read · DevOps synologynetworkingsmbjellyfinobservabilitylinux

The Ghost Bandwidth Mystery

When a primary storage node reports a continuous 14 MB/s (112 Mbps) outbound transfer rate, an engineer’s first instinct is checking external gateways: rogue data egress, misconfigured cloud replication jobs, or an outbound exfiltration loop. In a multi-homed infrastructure, aggregate traffic meters often mask an internal pipeline saturation that never touches the public internet.

On a production Synology NAS (DS1821+) anchoring storage for compute nodes across the estate, monitoring alarms fired on sustained disk I/O activity. Storage Pool 1 disks were under high demand, with the system 1-minute load average hovering near 4.2.

Checking outbound replication and external transfer services yielded no culprits:

Yet the NAS network graph displayed a relentless, dead-flat 14 MB/s transmit (TX) baseline. To isolate the consumer, I stepped past high-level dashboard summaries into low-level Linux kernel telemetry.

Isolating Multi-Homed Interfaces and Process Sockets

A production NAS with bonded or dedicated interconnects cannot be analyzed using aggregate total_tx_bytes. Traffic must be isolated per-interface, mapped to active TCP sockets, and correlated with filesystem file descriptors.

+-------------------------------------------------------------+
|               Storage Node (Synology DS1821+)              |
|   ovs_eth0 (1GbE WAN Gateway: 192.168.1.1) -> 0.89 MB/s TX  |
|   ovs_eth4 (2.5GbE Link:       192.168.1.2) -> 13.40 MB/s TX |
+-------------------------------------------------------------+
                               |
              Dedicated 2.5GbE SMB Link (:445)
                               |
                               v
+-------------------------------------------------------------+
|             Compute Host (Unraid Docker: 192.168.1.6)       |
|    Jellyfin Container -> Task: "Generate Trickplay Images"  |
|    Sequential SMB Read: 17,370 Videos -> 35 GB Extracted     |
+-------------------------------------------------------------+

1. Per-Interface Delta Sampling

Sampling /proc/net/dev across a 3-second window cleanly decoupled external traffic from internal fabric communication:

📄 /proc/net/dev 3-Second Rate Delta (18 lines)
=== Network Interface Delta (3s) ===
  ovs_eth0        RX:   2.34 MB/s | TX (Upload):   0.89 MB/s
  eth0            RX:   2.40 MB/s | TX (Upload):   0.89 MB/s
  ovs_eth4        RX:   0.12 MB/s | TX (Upload):  12.84 MB/s
  eth4            RX:   0.13 MB/s | TX (Upload):  13.40 MB/s
  docker629ef00   RX:   0.67 MB/s | TX (Upload):   0.67 MB/s
  docker-284d72bd RX:   0.68 MB/s | TX (Upload):   0.68 MB/s

The diagnosis was immediate:

The upload was not leaking into the outside world; it was flooding the private 2.5GbE point-to-point path connecting the storage NAS directly to the secondary compute server (192.168.1.6).

2. Sockets to PIDs: Attributing the SMB Stream

Querying socket states on 192.168.1.2 isolated the exact process owning the file stream:

sudo netstat -ntup | grep 192.168.1.2

This revealed an active, high-throughput SMB connection on port 445 streaming to the compute host (192.168.1.6), evidenced by the non-zero Send-Q buffer (38240 bytes queued in the kernel socket buffer):

tcp 0 38240 192.168.1.2:445 192.168.1.6:41882 ESTABLISHED 31727/smbd

The consumer was smbd (PID 31727). By inspecting the open file descriptors of that process via /proc/31727/fd, the active files became visible:

sudo ls -l /proc/31727/fd | grep -E '/volume[0-9]+'

Every two seconds, the active file descriptor rotated across different high-bitrate video assets in /volume1/Media/:

Something on the compute host was sequentially reading through video masters across the network at a rate of 100+ megabits per second.

Root Cause, Resolution, and Isolated Monitoring

Connecting to the compute host (192.168.1.6) explained the mystery: a newly deployed Jellyfin container had recently taken over media serving duties.

The Culprit: Unthrottled Trickplay Keyframe Extraction

By default, modern Jellyfin instances enable Trickplay Image Generation—a background worker that invokes ffmpeg to extract video keyframes (I-frames) at 10-second intervals and stitches them into sprite sheets (320px tile grids) for timeline scrubbing previews.

While benign on local NVMe storage, running this unthrottled across an SMB network mount triggered a severe storage anti-pattern:

The Immediate Remediation

Using the Jellyfin Scheduled Tasks API, I queried the running tasks, resolved the task ID GUID, and canceled the active worker:

import json
import urllib.request

JELLYFIN_URL = "http://192.168.1.6:8096"
API_KEY = "REDACTED_API_KEY"
HEADERS = {"X-Emby-Token": API_KEY}

# 1. Resolve the Scheduled Task GUID by its Key/Name
req_list = urllib.request.Request(f"{JELLYFIN_URL}/ScheduledTasks", headers=HEADERS)
with urllib.request.urlopen(req_list) as resp:
    tasks = json.loads(resp.read().decode())

trickplay_task_id = next(
    t["Id"] for t in tasks if t.get("Key") == "RefreshTrickplayImages" or "Trickplay" in t.get("Name", "")
)

# 2. Terminate the active running task instance by Task ID
req_stop = urllib.request.Request(
    f"{JELLYFIN_URL}/ScheduledTasks/Running/{trickplay_task_id}",
    headers=HEADERS,
    method="DELETE",
)
with urllib.request.urlopen(req_stop) as resp:
    print(f"Terminated task {trickplay_task_id} — HTTP status: {resp.status}")

Within 2 seconds of task cancellation:

In Jellyfin’s administrative library configuration, I set:

<EnableTrickplayImageExtraction>false</EnableTrickplayImageExtraction>
<ExtractTrickplayImagesDuringLibraryScan>false</ExtractTrickplayImagesDuringLibraryScan>

Architectural Takeaways

  1. Decoupled Media Architecture Requires Explicit I/O Fencing: When running compute daemons separate from storage nodes, never allow workers to run unthrottled index passes over network mounts. Either restrict thumbnail extraction to local staging drives or cap concurrency strictly.
  2. Segment Multi-NIC Observability in Prometheus: Aggregate host bandwidth rules (sum by (instance) (rate(node_network_transmit_bytes_total[5m]))) are an anti-pattern on multi-homed storage servers. Separate WAN-facing interfaces from storage backplanes explicitly:
# WAN / Edge Interface Egress Alert (Low Threshold: > 8 MB/s for 5m)
rate(node_network_transmit_bytes_total{device="ovs_eth0"}[5m]) > 8 * 1024 * 1024

# Dedicated 2.5GbE Storage Backplane Alert (High Threshold: > 180 MB/s for 15m)
rate(node_network_transmit_bytes_total{device="ovs_eth4"}[15m]) > 180 * 1024 * 1024
🛡️ Production Infrastructure & SRE

Dealing with flaky containers, cloud cost bloat, or unexplained outages?

Real production reliability comes from understanding the kernel and the protocol layers, not just restarting containers. I manage 162 containers across 3 hosts running 24/7 client workloads.

Lee Teong Hoe

Lee Teong Hoe · Mr Hoelee

Full-stack developer & DevOps engineer based in Malaysia. I run my own production infrastructure and build web systems for small businesses — this blog is the living record.

Hiring? Profile & stack · Malaysian SME? Website design & development ↗ · All posts