Tracking Down 14 MB/s Ghost Uploads Across a 2.5GbE NAS Interconnect: The Media Server Trickplay Trap
The Ghost Bandwidth Mystery
When a primary storage node reports a continuous 14 MB/s (112 Mbps) outbound transfer rate, an engineer’s first instinct is checking external gateways: rogue data egress, misconfigured cloud replication jobs, or an outbound exfiltration loop. In a multi-homed infrastructure, aggregate traffic meters often mask an internal pipeline saturation that never touches the public internet.
On a production Synology NAS (DS1821+) anchoring storage for compute nodes across the estate, monitoring alarms fired on sustained disk I/O activity. Storage Pool 1 disks were under high demand, with the system 1-minute load average hovering near 4.2.
Checking outbound replication and external transfer services yielded no culprits:
- Off-site backup and
rclone/ CloudSync engines were completely idle. - Background container registry syncs and scheduled transfer queues were paused.
- Edge router telemetry confirmed public WAN egress was sitting well below 1.2 MB/s.
Yet the NAS network graph displayed a relentless, dead-flat 14 MB/s transmit (TX) baseline. To isolate the consumer, I stepped past high-level dashboard summaries into low-level Linux kernel telemetry.
Isolating Multi-Homed Interfaces and Process Sockets
A production NAS with bonded or dedicated interconnects cannot be analyzed using aggregate total_tx_bytes. Traffic must be isolated per-interface, mapped to active TCP sockets, and correlated with filesystem file descriptors.
+-------------------------------------------------------------+
| Storage Node (Synology DS1821+) |
| ovs_eth0 (1GbE WAN Gateway: 192.168.1.1) -> 0.89 MB/s TX |
| ovs_eth4 (2.5GbE Link: 192.168.1.2) -> 13.40 MB/s TX |
+-------------------------------------------------------------+
|
Dedicated 2.5GbE SMB Link (:445)
|
v
+-------------------------------------------------------------+
| Compute Host (Unraid Docker: 192.168.1.6) |
| Jellyfin Container -> Task: "Generate Trickplay Images" |
| Sequential SMB Read: 17,370 Videos -> 35 GB Extracted |
+-------------------------------------------------------------+
1. Per-Interface Delta Sampling
Sampling /proc/net/dev across a 3-second window cleanly decoupled external traffic from internal fabric communication:
📄 /proc/net/dev 3-Second Rate Delta (18 lines)
=== Network Interface Delta (3s) ===
ovs_eth0 RX: 2.34 MB/s | TX (Upload): 0.89 MB/s
eth0 RX: 2.40 MB/s | TX (Upload): 0.89 MB/s
ovs_eth4 RX: 0.12 MB/s | TX (Upload): 12.84 MB/s
eth4 RX: 0.13 MB/s | TX (Upload): 13.40 MB/s
docker629ef00 RX: 0.67 MB/s | TX (Upload): 0.67 MB/s
docker-284d72bd RX: 0.68 MB/s | TX (Upload): 0.68 MB/s
The diagnosis was immediate:
ovs_eth0(the standard 1GbE interface on192.168.1.1, servicing normal LAN clients and the default internet gateway via192.168.1.254) was transmitting a modest 0.89 MB/s.ovs_eth4/eth4(the dedicated 2.5GbE PCIe NIC on192.168.1.2) was consuming 13.40 MB/s of continuous outbound throughput.
The upload was not leaking into the outside world; it was flooding the private 2.5GbE point-to-point path connecting the storage NAS directly to the secondary compute server (192.168.1.6).
2. Sockets to PIDs: Attributing the SMB Stream
Querying socket states on 192.168.1.2 isolated the exact process owning the file stream:
sudo netstat -ntup | grep 192.168.1.2
This revealed an active, high-throughput SMB connection on port 445 streaming to the compute host (192.168.1.6), evidenced by the non-zero Send-Q buffer (38240 bytes queued in the kernel socket buffer):
tcp 0 38240 192.168.1.2:445 192.168.1.6:41882 ESTABLISHED 31727/smbd
The consumer was smbd (PID 31727). By inspecting the open file descriptors of that process via /proc/31727/fd, the active files became visible:
sudo ls -l /proc/31727/fd | grep -E '/volume[0-9]+'
Every two seconds, the active file descriptor rotated across different high-bitrate video assets in /volume1/Media/:
.../Production_Archive/Project_A/Scene_12.mkv.../Reference_Library/Render_Master_04.mp4.../Transcode_Staging/Source_Stream_09.mkv
Something on the compute host was sequentially reading through video masters across the network at a rate of 100+ megabits per second.
Root Cause, Resolution, and Isolated Monitoring
Connecting to the compute host (192.168.1.6) explained the mystery: a newly deployed Jellyfin container had recently taken over media serving duties.
The Culprit: Unthrottled Trickplay Keyframe Extraction
By default, modern Jellyfin instances enable Trickplay Image Generation—a background worker that invokes ffmpeg to extract video keyframes (I-frames) at 10-second intervals and stitches them into sprite sheets (320px tile grids) for timeline scrubbing previews.
While benign on local NVMe storage, running this unthrottled across an SMB network mount triggered a severe storage anti-pattern:
- Network-to-Disk Amplification: An unthrottled
ffmpegpipeline on the compute host was issuing continuous byte-range SMB reads across the 2.5GbE interconnect. - 17,370 Assets Scanned: The worker had already traversed 17,370 video files, writing 35 GB of sprite tiles to the compute node’s cache.
- Mechanical Seek Thrashing (
DState Inflation): Although 13.40 MB/s represents less than 5% of a 2.5GbE link’s capacity (~295 MB/s), seeking across thousands of large media files forced the NAS mechanical RAID array into relentless random actuator movement. Because Linux Load Average includes processes blocked in uninterruptible I/O sleep (TASK_UNINTERRUPTIBLE), this seek contention drove system load to 4.22 and starved co-located transactional services.
The Immediate Remediation
Using the Jellyfin Scheduled Tasks API, I queried the running tasks, resolved the task ID GUID, and canceled the active worker:
import json
import urllib.request
JELLYFIN_URL = "http://192.168.1.6:8096"
API_KEY = "REDACTED_API_KEY"
HEADERS = {"X-Emby-Token": API_KEY}
# 1. Resolve the Scheduled Task GUID by its Key/Name
req_list = urllib.request.Request(f"{JELLYFIN_URL}/ScheduledTasks", headers=HEADERS)
with urllib.request.urlopen(req_list) as resp:
tasks = json.loads(resp.read().decode())
trickplay_task_id = next(
t["Id"] for t in tasks if t.get("Key") == "RefreshTrickplayImages" or "Trickplay" in t.get("Name", "")
)
# 2. Terminate the active running task instance by Task ID
req_stop = urllib.request.Request(
f"{JELLYFIN_URL}/ScheduledTasks/Running/{trickplay_task_id}",
headers=HEADERS,
method="DELETE",
)
with urllib.request.urlopen(req_stop) as resp:
print(f"Terminated task {trickplay_task_id} — HTTP status: {resp.status}")
Within 2 seconds of task cancellation:
- Network throughput on
ovs_eth4plummeted from 13.40 MB/s to 0.00 MB/s. - System load average on the storage node dropped from 4.22 down to 1.15.
- Disk I/O wait on Storage Pool 1 normalized completely.
In Jellyfin’s administrative library configuration, I set:
<EnableTrickplayImageExtraction>false</EnableTrickplayImageExtraction>
<ExtractTrickplayImagesDuringLibraryScan>false</ExtractTrickplayImagesDuringLibraryScan>
Architectural Takeaways
- Decoupled Media Architecture Requires Explicit I/O Fencing: When running compute daemons separate from storage nodes, never allow workers to run unthrottled index passes over network mounts. Either restrict thumbnail extraction to local staging drives or cap concurrency strictly.
- Segment Multi-NIC Observability in Prometheus: Aggregate host bandwidth rules (
sum by (instance) (rate(node_network_transmit_bytes_total[5m]))) are an anti-pattern on multi-homed storage servers. Separate WAN-facing interfaces from storage backplanes explicitly:
# WAN / Edge Interface Egress Alert (Low Threshold: > 8 MB/s for 5m)
rate(node_network_transmit_bytes_total{device="ovs_eth0"}[5m]) > 8 * 1024 * 1024
# Dedicated 2.5GbE Storage Backplane Alert (High Threshold: > 180 MB/s for 15m)
rate(node_network_transmit_bytes_total{device="ovs_eth4"}[15m]) > 180 * 1024 * 1024 Dealing with flaky containers, cloud cost bloat, or unexplained outages?
Real production reliability comes from understanding the kernel and the protocol layers, not just restarting containers. I manage 162 containers across 3 hosts running 24/7 client workloads.