Calibrating Intel Quick Sync HEVC: A Cliff at 28, and a Preset That Does Nothing

The question I could not answer: which number do I ship?

My transcoding queue decides, per file, which encoder does the work: copy when the source is already efficient, NVENC on the GPU, x265 on the CPU — and now Intel Quick Sync. Each tier has two knobs that decide quality, and every tier except QSV had numbers I had measured on my own library.

QSV had none. And the code refused to guess: a shipping default of qsv_quality = 0 / qsv_preset = "" means nobody has calibrated this, and the engine treats that as a hard exclusion — QSV can exist in the code and never be selected on any machine, including the one that has the hardware.

So the task was not “turn on QSV”. It was “produce measured numbers for two knobs, so the encoder becomes selectable honestly”.

Problem one: the hardware is not in the machine where the app runs

The queue runs on Windows 11 in a VM. That VM has an RTX 3060 passed through and no Intel graphics of any kind — which is exactly why QSV had never been measured: the obvious move is to run one test locally and read the numbers off.

The iGPU is on the host instead (unRaid, i9-13900K), idle and unclaimed:

00:02.0 VGA compatible controller: Intel Raptor Lake-S GT1 [UHD Graphics 770]
  driver: i915            refcount: 0
/dev/dri/card0  /dev/dri/renderD128        (present, world-accessible)
no running container declares a dri device
virsh dumpxml Win11: hostdevs = 01:00.0, 01:00.1 (the 3060 + its HDMI audio)

Two conclusions, both useful. The VM’s Error creating a MFX session is expected and not a bug to chase — the VM can never see that iGPU. And the only way to measure QSV is on the Linux side, in a container, on the host.

Measure with the engine’s own gate, not with ffmpeg

This is the part that makes the numbers worth anything. I did not write ffmpeg commands and compute SSIM myself — the calibration runs the engine that will do the work, with its own verification and its own accept/reject gate:

python3 library_run.py --work-dir /work --tier qsv --force-tier qsv \
  --qsv-quality Q --qsv-preset P --path <real source> --keep-local --overwrite

--keep-local means encode, verify, run the gate, and then do not land the file — the verdict is what I am collecting, not a new library. Sources came in read-only over an existing SMB mount, and I hashed each source before and after: unchanged.

The image is linuxserver/ffmpeg (ffmpeg 9.0.2, iHD 26.3.5, oneVPL 2.17) plus python3 — plus numpy, which is not optional. Without it the gate prints flat-frame scan skipped: No module named 'numpy' and quietly changes how near-flat frames are judged, which changes verdicts.

-global_quality maps to RateControlMethod: ICQ on hevc_qsv, which is the knob being calibrated.

The ladder: on a thin source, low ICQ makes the file bigger

First run, on a real 1080p25 source at 951 kbps (0.0184 bits per pixel — a “thin” file, the kind that is already squeezed):

Three things fall out of that table.

The counterintuitive one: low ICQ does not mean safe. ICQ 18–22 produce files larger than the source, so the engine’s size ceiling (100% of the source’s video bitrate) rejects them. QSV is not alone in this — NVENC at a low cq does the same thing on thin inputs — but the boundary sits higher here: the usable range starts at 24, not 18.

Second, there is a cliff between 28 and 30. 30 and 32 fail on quality, and they fail on the first attempt. That phrase matters, because the engine retries a failed quality attempt at global_quality - 2 — lower ICQ, higher quality — and logs the retry. A line reading “ok at quality+2” is a first-attempt failure, not a pass. I read that line backwards in my own notes the first time; the rule is to read the first attempt, not the recovery.

Third, the floor is where the choice gets made: with a 0.995 mean-SSIM floor, q28 passes with 0.0001 of margin and q26 with 0.0010.

Full files confirm the ladder — and one shortcut survives

Excerpts (180-second lossless slices) are how you rank ten candidates without ten full encodes. They are only legitimate if they agree with full-file runs, so I ran the real files end to end:

The 180-second excerpt of the first file measured mean 0.9959; the full file measured 0.9960. That is close enough to trust the shortcut for quality ranking — and I now have the evidence to say so rather than hope so.

The preset that does nothing

The excerpt runs said preset only changes speed: veryfast 30 s / 61.6%, fast 30 s / 61.0%, medium 60 s / 61.0%, slow and veryslow 60 s. Clean story: pick veryfast, save 30 seconds per file.

Then the full files disagreed with it. The same source at ICQ 26, one run with fast and one with medium:

fast:    168,435,266 B   enc 360 s   mean SSIM 0.9960
medium:  168,435,266 B   enc 360 s   mean SSIM 0.9960

Byte-for-byte identical, same encode time, same quality. The 30-vs-60-second difference in the excerpt layer was per-run fixed overhead, not encoding speed — and it is the kind of number that would have gone into a config file and a blog post if I had stopped one step early.

So: excerpts rank quality, never speed. Speed has to be measured on full files, where the fixed overhead disappears.

What I shipped

qsv_quality = 26, qsv_preset = medium — 61.0% of the source’s video bitrate at mean SSIM 0.9960 (min 0.9938) at roughly 5.6× realtime, and 50.4% at mean 0.9977 on the thinnest file in the library. q28’s margin over the floor is 0.0002, and the cost of a quality failure is two full encodes plus a park, so the extra 15 percentage points of savings are not worth the tail risk.

Two supporting measurements worth recording, because they answer “is it QSV or is it the file?”:

• Cross-encoder control. One 720p source with non-uniform cadence scored QSV q24 mean 0.9154, QSV q28 mean 0.9146, and x265 crf26 mean 0.9164 — and the engine’s alignment search reported no alignment explains the mismatch. The rejection belongs to the source, not to Quick Sync.
• Concurrency. The engine refuses a second encoder on a shared work root: another transcode is already running (pid 7, /work/run-qsv.lock, 3 s old). With separate work roots, two jobs run — and take 201 s where one takes 107 s. Twice the wall clock, zero throughput gain. One QSV job at a time.

Four traps, each of which cost a cycle

1. Never mount a shared library at /lib. I mounted the SMB share into the container at /lib — which is where the container’s own dynamic loader lives. All eight jobs died in under a second with exec /bin/bash: no such file or directory, which reads like a broken image, not a bad mount. /library works.
2. Synthetic sources lie about ICQ. hevc_qsv’s rate control is rejected by the driver for rgb24 input (Selected ratecontrol mode is unsupported) — so a testsrc/mandelbrot self-test concludes the opposite of the truth. Use a real file, or add format=nv12.
3. Already-HEVC sources cannot be calibrated at all. The gate answers SKIP already HEVC hvc1 and --overwrite does not lift it. The fat 1080p H.264 originals in this library had all been converted long ago; the remaining fat files are HEVC — so that tier has nothing left to measure on. That is a missing input, not a skipped task.
4. Read the first attempt, not the retry (see the cliff above).

What I would do differently

1. Measure speed on the artifact you will actually run, not on a slice of it. My one wrong number in this whole exercise came from trusting an excerpt’s wall clock.
2. Make the machine prove it has the hardware before you plan the test. Six minutes of lspci and virsh dumpxml retired the “why does QSV fail in the VM” question that had been open for weeks.
3. Keep the gate in the loop. Because the engine’s own verification judged every run, the tables above are decisions, not opinions — the same gate that will reject a bad file in production rejected my ICQ 30.
4. Write the honest scope line into the notes. These numbers are Linux iHD 26.3.5 + oneVPL 2.17 on a UHD 770; the production target is the Windows MSDK path on the same silicon, and I cannot claim they agree. What this establishes is that the command runs on real hardware, where the quality↔size curve sits, and how the lock behaves — leaving a re-measurement on Windows+Intel as one table, instead of another afternoon.

The result

QSV went from “present in the code and never selectable” to a calibrated tier with published numbers, and the safety rule changed with it: the test that used to pin shipping defaults must mean uncalibrated now pins shipping defaults must be measured values, while keeping every assertion that an empty value is a hard exclusion. The doctor output on a machine with no Intel GPU moved from an informational “waiting for hardware” line to a warning — which is its own small lesson about what a calibrated encoder failing self-check should look like, and why hardware the machine does not have belongs in a summary row rather than a warning.

Lee Teong Hoe · Mr Hoelee

Full-stack developer & DevOps engineer based in Malaysia. I run my own production infrastructure and build web systems for small businesses — this blog is the living record.

LinkedIn GitHub Gitea [email protected]

Hiring? Profile & stack · Malaysian SME? Website design & development ↗ · All posts