When Lowering CRF Does Nothing: The -maxrate Ceiling and the SSIM Quality Deadlock
The batch that burned 106 hours doing nothing
In a high-throughput transcoding pipeline running across dedicated on-premise infrastructure,
one of my automated ingestion queues had a strict rule: the output must never exceed the size
of the source (SIZE_CEIL = 1.0).
The goal was automated space reclamation across tens of thousands of archived video files. If an
HEVC transcode came out larger than the H.264 original, the pipeline rejected it as counterproductive.
To enforce this, every encode passed -maxrate <source_vbps> and -bufsize <source_vbps * 2.5>
to FFmpeg.
At the same time, the pipeline guarded visual fidelity with an automated quality gate:
every transcoded file had its structural similarity (SSIM) measured against the source frame by frame.
If the mean SSIM fell below 0.995, the item failed the quality gate. To recover borderline files,
the supervisor triggered an automatic retry: quality+2 (dropping CRF from 29 to 27 or 19 to 17).
Then the production telemetry came in.
Over a two-week run, 487 quality rounds failed, consuming 106.2 hours of dedicated compute. Worse: of 202 files that went through two consecutive rounds, 194 showed an SSIM movement of less than 0.0015. The second round burned full CPU cycles to produce a file almost byte-for-byte identical to the one rejected ten minutes earlier.
The anomaly in the lab
To isolate why the quality ladder was failing to move the needle, I isolated a representative test
video: a 720p@30 clip (1280x720, 96 seconds) with an original video bitrate of 1832 kbps.
I ran three controlled encoding passes using libx265:
Run 1 (Baseline):
Command: -crf 19 -maxrate 1832k -bufsize 4580k
Size: 20.6 MB (87.5% of source)
SSIM Mean: 0.99192 --> REJECTED (below 0.995)
Run 2 (The Aggressive Quality Retry):
Command: -crf 14 -maxrate 1832k -bufsize 4580k (5 quality steps lower!)
Size: 22.9 MB (97.3% of source)
SSIM Mean: 0.99195 --> REJECTED (below 0.995)
Delta: +0.00003 SSIM improvement
Run 3 (Uncapped Ceiling):
Command: -crf 19 (no -maxrate, no -bufsize)
Size: 40.6 MB (172.5% of source)
SSIM Mean: 0.99648 --> PASS (above 0.995)Look at the difference between Run 1 and Run 2: dropping CRF by five full steps moved the mean SSIM by exactly 0.00003.
Yet in Run 3, the exact same CRF 19 sailed past the 0.995 bar to 0.99648—at the cost of expanding the file to 172% of its original size.
The mathematical trap between CRF and -maxrate
The culprit was not the video encoder, the preset, or the content complexity. It was a fundamental contradiction in the pipeline’s control architecture:
- CRF is a quality target, not a rate controller. It calculates the quantization parameter needed to maintain uniform visual fidelity across frames.
-maxrateis a hard bitrate ceiling enforced by the VBV (Video Buffer Verifier) buffer.- When complex scenes demand more bits to meet a lower CRF, the VBV governor intervenes and drops quality to honor the
-maxrateceiling.
In high-motion, high-entropy scenes, reaching an SSIM mean of 0.995 required at least 125% to
170% of the source’s native bitrate. But the SIZE_CEIL = 1.0 contract I enforced forbade giving
the encoder those bits.
Consequently, when an encode ran close to the ceiling (spending >= 97% of the allowable bits), lowering the CRF did not buy more detail—it simply forced the encoder to hit the ceiling earlier in the GOP and throw the extra bits away. The quality ladder stopped being a ladder.
The architectural solution: Two-tier gates and cap awareness
To eliminate the 106 hours of wasted compute, I refactored the supervisor’s decision engine in three ways:
1. Identifying the true business objective (space_reclaim_source)
Why was a file being transcoded in the first place?
- Case A: Format Remediation. If the source was VP9, AV1, or inside an MKV container with AC-3 audio, it could not play natively in mobile or web browsers without server-side transcoding. Here, fidelity is paramount: I enforce the strict
0.995threshold. - Case B: Pure Space Reclamation. If the source was already an H.264 baseline MP4 with AAC audio, it already played everywhere natively. The only purpose of re-encoding was saving disk space. Holding a compatible file to a strict 0.995 bar that forces bitrate expansion is counterproductive.
For Case B, the pipeline now accepts a secondary tier: SSIM_MEAN_MIN_FALLBACK = 0.990.
2. Detecting the ceiling wall (CAP_BOUND_SHARE)
If an encode used >= 97% of the allowable source bitrate (obps >= vbps * SIZE_CEIL * 0.97),
the file was capped. Retrying with a lower CRF would mathematically yield zero benefit. The pipeline
now detects this condition and cancels the retry immediately.
🔍 View the two-tier quality gate and cap-bound bypass logic — evaluate_quality_gate() (28 lines)
SIZE_CEIL = 1.0 # Output video bitrate must not exceed the source's
CAP_BOUND_SHARE = 0.97 # Spent >= 97% of ceiling -> rate-limited, not gear-limited
QUALITY_STEPS = 2 # Retry offset (crf - 2)
def space_reclaim_source(info, src):
"""True if the video and audio streams already decode natively on mobile/web targets."""
if info.get("vcodec") not in ("h264", "avc1"):
return False
if info.get("acodec") not in (None, "aac", "mp4a"):
return False
return os.path.splitext(src)[1].lower() in (".mp4", ".m4v", ".mov")
def evaluate_quality_gate(info, src, mean_ssim, bad_share, vbps, obps):
capped = bool(vbps) and obps >= (vbps * SIZE_CEIL * CAP_BOUND_SHARE)
systematic = bad_share > MAX_BAD_SHARE or (mean_ssim < SSIM_TARGET)
# Second-tier relaxation for files that already play everywhere
if systematic and bad_share <= MAX_BAD_SHARE and mean_ssim >= SSIM_MEAN_MIN_FALLBACK:
if space_reclaim_source(info, src):
return "PASS_SECOND_TIER", "Accepted at secondary 0.990 threshold for space reclamation"
if systematic:
# If output is already hitting the bitrate ceiling, retry is mathematically inert
if capped:
return "FAIL_CAPPED", f"SSIM {mean_ssim:.4f} below bar, but output is capped ({obps/vbps*100:.1f}%) - retry skipped"
return "RETRY", f"Retry with crf - {QUALITY_STEPS}"
return "PASS", "Clean pass"The result
After deploying this change to the production nodes:
- The test file that previously bounced between
crf 19andcrf 14with aFAIL quality 0.99192now cleared on its first pass asPASS_SECOND_TIER(87.5% of original size, zero retries, 120s encode time). - The 106-hour wasted compute overhead on boundary files dropped to zero.
- Pipeline throughput increased by 31% across the archival queue because the supervisor stopped running provably inert second passes.
In an automated transcoding pipeline, quality metrics can never be evaluated in isolation from the rate-control ceiling. When CRF meets -maxrate, the ceiling always wins.
Dealing with flaky containers, cloud cost bloat, or unexplained outages?
Real production reliability comes from understanding the kernel and the protocol layers, not just restarting containers. I manage 162 containers across 3 hosts running 24/7 client workloads.
