It Reported Success. Nothing Had Changed.
The batch that looked fine
I run a conversion pipeline over a 582-file video library that lives on a
NAS and is mounted as a Windows drive letter through a third-party SMB
client (the kind that presents the share as a normal Z:\ drive). Small
files, big media files, one writer.
The pipeline does the obvious things: stage the source locally, encode, verify the output, upload it next to the original, replace the original. It logged success on every step I had thought to check.
Then I started auditing the results instead of the log. Over a set of 30
files the pipeline had reported as done, one had not changed at all —
the file on the share was still the original. No error, no warning, no
failed step. The log said ok.
That is the failure mode this post is about: not a crash, not a corrupted file, but a system that reports an outcome it did not achieve. If your pipeline’s success is defined by its own log, you have no idea what is actually on disk.
Lie 1: a write that stops early and returns success
The first one is the scariest because everything looks right at the moment of the call.
FAIL read-back mismatch
local = 3b7cd25d1de1 403,448,100 bytes
remote = 6a7785351031 12,582,912 bytesThe upload of a 403 MB output file wrote 12.5 MB and returned. No exception, no non-zero exit, no short-write return code. The client’s own error — a timeout toast — appeared asynchronously, minutes later, long after my code had moved on and printed its success line.
This is why a “did the write succeed?” check that trusts the API is worth nothing on this class of storage. The write call answered a different question (“did I hand the data to the cache?”) than the one you asked (“is the file on the other side correct?”).
The fix is boring and total: hash what you wrote locally, read the file back from the share, hash that, and compare. On a mismatch, re-upload. Twice, if needed, with a backoff.
# after uploading dst from src
if hash_file(src) != hash_file(_reread(dst)):
retry_upload()Note the deliberate cost: this reads the file over the network a second time. For a 400 MB file that is a few seconds — against a 25-minute encode it is rounding error. The alternative is a library where some fraction of files are silently not what you think.
Lie 2: a rename that does not land, with metadata that says it did
The second one hides behind a filesystem primitive you trust blindly:
os.replace() — atomic, same-filesystem, cannot half-happen. Except
“filesystem” here means a network client with its own metadata cache.
The sequence was: upload show.mp4.new.mp4, verify the hash, os.replace()
it over show.mp4, then probe the result to confirm. The probe said HEVC,
the size looked right, the run logged ok 1085 MB -> 478 MB.
The file on the share was the untouched original: H.264, 921,950,850
bytes. The replace had not been applied; the client served cached attributes
for the path, so getsize() and mtime happily described a file that no
longer existed.
The only check that survives a lying metadata cache is reading the bytes back:
os.replace(tmp, dst) # claim
if hash_file(dst) != hash_file(local): # evidence
raise VerificationError(dst)After I added that step, the same audit that found 1-in-30 failures found zero — and, more importantly, any future silent failure turns into a loud one, because “the bytes at the destination do not match the bytes I produced” is not a thing a cache can fake.
Lie 3: a size check that agrees with a bad copy
The third one is the subtlest, because it fools verification itself, not just the operation.
Staging a source file from the share into local scratch is a copy. The natural check is “did the right number of bytes arrive?” — so the pipeline compared byte counts. They matched.
Then, in an unrelated investigation, the same source file failed to decode (“Error splitting the input into NAL units”), and I went looking for a corrupt file that was not there. The file was fine: 921,950,850 bytes, healthy header, sixty seconds decoded with zero errors. The copy had been truncated in a way that preserved the length, or the read had silently returned short and been padded.
Length is not a checksum. A copy is verified when the hash of what you read matches the hash of what is on the share — which means reading the share twice, or hashing after the read and comparing against a hash taken on a different occasion.
The metric traps that made it worse
While chasing those three, my quality checks were lying too, in ways worth naming because they are not specific to network storage:
- A decode test passes on wrong pictures.
ffmpeg -v error -i out.mp4 -f null -printed nothing on a file with 106 frames decoded from the wrong timestamps. “No errors” means “no errors”, not “correct”. - SSIM is meaningless on flat frames. Two nearly-black frames can score 0.25 while being visually identical. Any threshold that treats a single low frame as proof of a defect will reject good encodes — and any that ignores frames wholesale will miss real ones.
- Seeking into an open-GOP source decodes the wrong pictures. Comparing a file against itself, with one side decoding from a mid-file seek and the other sequentially, measured min SSIM 0.25. A false “systematic defect” produced entirely by my own comparison harness.
- A stream copy is not a lossless slice. Cutting 60 seconds out of an
H.264 file with B-frames using
-ss ... -c copyproduced 1270 frames where 1500 were expected. Fine for scrubbing; fatal as a test fixture. - Two concurrent runs sharing a scratch directory will overwrite each other’s intermediates. Mine had fixed filenames for the decoded raw frames, so two gates running at once compared two different videos and reported 100% of frames below the threshold. Every temporary path now carries the process id.
The discipline
None of this needs cleverness. It needs a rule: a step is done when you have independently observed the outcome, not when the tool reported it.
In practice, for a pipeline that writes to a share:
- Read the source twice. Stage the file, hash it, hash a fresh read of the share, compare. Length is not evidence. 2. Hash the upload before you replace anything. 3. Hash the destination after the replace. Cached metadata cannot fake bytes. 4. Verify the content, not just the container — compare pictures, not frame rates. 5. Make every failure loud. A gate that fails safe costs one wasted encode; a gate that passes a bad file costs a corrupted library and your trust in it. 6. Do not parallelise the writer. Three jobs sharing one share and one scratch directory produced short writes, timeouts and cross-contaminated comparisons. When I dropped to one writer, the error rate went to zero.
The audit line I care about is not “0 errors”. It is:
reported=30 landed=29 not-landed=0 wrong-codec=1 gone=0Every number there was produced by reading the destination, not by trusting the log.
