Uploading 100MB+ Videos to YouTube via CDP: The Docker Archive Pattern

Automating web uploads via headless browser or Chrome DevTools Protocol (CDP) works seamlessly for small JSON payloads, forms, and occasional screenshots. But when you attempt to upload a 100MB+ 1080p video file to YouTube through a remote browser session, standard browser automation approaches quickly collapse.

Here is the exact trap that breaks remote CDP uploads, why browser memory limits fight you, and the lightweight Docker Archive pattern that makes large-file uploads rock-solid.

TL;DR — Do not push megabytes through the CDP WebSocket. PUT /containers/{id}/archive?path=… on the Docker Engine API (here via Portainer) tars the file straight into the Chrome container’s filesystem, then DOM.setFileInputFiles hands the container path to the page input. 130 MB staged in 1.89 s on a gigabit LAN. This is the staging half of our music-video pipeline; the rendering and visual transition tuning is covered in debugging Remotion video jitter.

1. When to Use Which File-Upload Strategy

Before jumping into implementation, it helps to understand where each upload technique belongs:

Why CDP Over YouTube Data API?

YouTube’s official Data API imposes a 10,000-unit daily default quota limit. A single video upload consumes 1,600 units, meaning you hit quota exhaustion after just 6 videos a day. Furthermore, uploading multi-language .srt tracks and custom video localization through the Data API requires complex OAuth2 consent scopes that expire frequently.

By driving an already-authenticated Chrome instance in Docker via CDP, we operate under standard browser sessions with zero quota walls.

2. The 50MB Remote Browser Pitfall

When controlling a local Chrome instance, Playwright or Puppeteer attaches files using page.setInputFiles("input[type=file]", local_path). The browser reads the file from your local disk and passes the handle directly to the OS file picker.

However, in a self-hosted on-premise or production setup, you frequently run Chrome inside an isolated Docker container on a NAS or server to keep your authenticated session alive 24/7 without exposing your daily desktop.

When your automation script runs on a different workstation or runner and tries to push a video over a remote CDP connection:

1. Playwright attempts to base64-encode or stream the file buffer over the WebSocket protocol.
2. The WebSocket frame exceeds protocol buffers or consumes massive memory on both ends.
3. The remote browser crashes with Target closed, or CDP drops the socket due to frame size timeouts.

Even if you increase the WebSocket chunk size, pushing a 130MB file over CDP is excruciatingly slow and fragile.

3. The Reframe: Local File Injection

The key realization is simple: Chrome inside the container doesn’t need the file pushed over the wire if the file already exists on its internal filesystem.

If the file /config/Downloads/video.mp4 already sits inside the container, CDP provides a native, zero-memory command that requires no file transfer at all:

cdp = context.new_cdp_session(page)
doc = cdp.send("DOM.getDocument")
file_node = cdp.send("DOM.querySelector", {
    "nodeId": doc["root"]["nodeId"],
    "selector": "input[type='file']",
})

# Takes 2 milliseconds and transfers ZERO bytes over the browser socket:
cdp.send("DOM.setFileInputFiles", {
    "files": ["/config/Downloads/video.mp4"],
    "nodeId": file_node["nodeId"],
})

The real question becomes: How do you get a 130MB file into the container filesystem without SSH keys, volume mounts, or slow SFTP?

4. The Portainer Docker Archive API (Hardened Implementation)

Every Docker daemon exposes a container archive endpoint: PUT /containers/{id}/archive?path=/path/in/container

If you manage your Docker host with Portainer (a common control plane on self-hosted Synology and Unraid nodes), Portainer proxies this Docker socket endpoint behind an API key:

PUT http://<host>:9000/api/endpoints/2/docker/containers/{container_id}/archive?path=/config/Downloads

Here is a tested, production-ready staging script with proper error handling, parameter validation, and secure file modes:

import io
import os
import tarfile
import urllib.error
import urllib.request

PORTAINER_KEY = os.environ.get("PORTAINER_API_KEY", "")
PORTAINER_BASE = "http://192.168.1.1:9000/api/endpoints/2/docker"
CONTAINER_ID = "2dc0fb6a11c7"  # Short ID or container name ("chrome") — prefer the name: a name survives container recreation

def stage_file_to_container(local_path: str, dest_filename: str) -> None:
    if not PORTAINER_KEY:
        raise ValueError("PORTAINER_API_KEY environment variable is not set")
    if not os.path.exists(local_path):
        raise FileNotFoundError(f"Local source file not found: {local_path}")

    size = os.path.getsize(local_path)
    tar_stream = io.BytesIO()

    # Build tar in memory (safe for 100-200MB payloads on workstations with 16GB+ RAM)
    with tarfile.open(fileobj=tar_stream, mode="w") as tar:
        with open(local_path, "rb") as f:
            ti = tarfile.TarInfo(name=dest_filename)
            ti.size = size
            # 0o644: owner read/write, group/others read-only (avoid world-writable 0o666/0o777)
            ti.mode = 0o644
            tar.addfile(ti, f)

    tar_stream.seek(0)
    data = tar_stream.read()

    endpoint = f"{PORTAINER_BASE}/containers/{CONTAINER_ID}/archive?path=/config/Downloads"
    req = urllib.request.Request(
        endpoint,
        data=data,
        headers={
            "X-Api-Key": PORTAINER_KEY,
            "Content-Type": "application/x-tar",
        },
        method="PUT",
    )

    try:
        with urllib.request.urlopen(req, timeout=120) as resp:
            if resp.status not in (200, 204):
                raise RuntimeError(f"Archive PUT failed with HTTP {resp.status}")
        print(f"Staged {dest_filename} ({size / (1024 * 1024):.2f} MB) in container successfully.")
    except urllib.error.HTTPError as e:
        body = e.read().decode("utf-8", errors="replace")
        raise RuntimeError(f"Portainer Archive API error {e.code}: {body}") from e

Operational Caveats & Security Checklist

1. Client RAM Footprint: While Chrome experiences zero memory pressure, io.BytesIO() buffers the 130MB tar archive in the Python process memory. For multi-gigabyte video dumps, write the .tar to a temporary file and stream it using requests or urllib3 chunked uploads.
2. Pre-existing Destination Directory: The Docker Archive API extracts into the directory specified by ?path=. If /config/Downloads does not exist in the container image, the API returns HTTP 404/500 rather than creating the parent folders.
3. API Key & CDP Exposure: Both the Portainer API key and the unauthenticated Chrome DevTools port (default 9222 or 9242) grant root-level execution capability. Never bind these ports to public IPs (0.0.0.0). Keep them strictly on an isolated internal bridge or behind mutual-TLS / Authentik SSO.

Measured Performance

On a local 1 Gbps LAN:

• 128.85 MB video file (.mp4): Transferred and extracted inside the container in 1.89 seconds (single run, warm container, SSD target on Synology NAS).
• Chrome container memory consumption: 0 MB RAM spike across all browser tabs.
• WebSocket overhead: 0 bytes transferred over DevTools Protocol.

5. YouTube Studio Upload Flow Automation

Once the video and custom thumbnail are staged in /config/Downloads/, the Playwright script navigates to https://studio.youtube.com/, triggers the upload modal, and executes the wizard:

1. Attach Video: Call DOM.setFileInputFiles on the primary input[type='file'].
2. Title & Description: Click div[contenteditable="true"][aria-label*="title"] and type the title; use .fill() on the description area.
3. Attach Thumbnail: Refresh the DOM with DOM.getDocument to discover the newly mounted input[type='file']#file-loader and inject /config/Downloads/cover.jpg.
4. Audience Setting: Check VIDEO_MADE_FOR_KIDS_NOT_MFK.
5. Wizard Steps: Click #next-button through Elements and Checks.
6. Visibility: Select PUBLIC and click #done-button.

Because the file was already on disk inside the container, YouTube Studio reads the stream locally, achieving maximum upload bandwidth without dropping a single frame.

6. Summary

• Never stream large binaries over WebSocket CDP. Remote browsers are not designed to buffer multi-hundred-megabyte payloads in memory.
• Stage files out-of-band: Use Docker Archive PUT to drop files onto the container filesystem in seconds.
• Inject via filesystem path: Call DOM.setFileInputFiles with the container path for instantaneous, zero-overhead file attachments.

Need rock-solid browser or media-publishing automation without API quota walls?

I architect and maintain self-hosted browser automation and media pipelines — from persistent Dockerised Chrome fleets driven over CDP to end-to-end video rendering and publishing workflows. If your team has a high-volume upload, portal workflow, or multi-step operation that breaks standard headless scrapers, I can build a deterministic pipeline on your own infrastructure.

Reach me at [email protected] or WhatsApp +60 12-797 2969, or see what I do at hoelee.com.

Lee Teong Hoe · Mr Hoelee

Full-stack developer & DevOps engineer based in Malaysia. I run my own production infrastructure and build web systems for small businesses — this blog is the living record.

LinkedIn GitHub Gitea [email protected]

Hiring? Profile & stack · Malaysian SME? Website design & development ↗ · All posts