VMAF Video Tests: Align the Reference Before Trusting the Score

A VMAF score is only meaningful when the reference and encoded video represent the same content at corresponding moments under a controlled comparison. A frame offset, inconsistent scaling, or an unrecorded model change can overwhelm the encoder difference you intended to measure.

Netflix’s official VMAF repository provides the implementation and documentation. Its current README notes a June 2026 model generation alongside documentation for older models. That makes model identity an important part of a reproducible October 2026 test—not an optional footnote.

Define the comparison before encoding

Write down the reference file, exact start and end points, frame rate, dimensions, pixel format, color handling, and audio treatment. Audio quality is a separate concern; a video metric should not be presented as proof that dialogue or synchronization is correct.

Choose a small corpus with different difficulties: motion, low light, fine text, faces, grain, and gradients. These are test-selection categories, not a claim that any one clip represents all video. Preserve permission to use the source material and avoid uploading private customer footage to unrelated tools.

Record which output constraints are fixed. Comparing one encoder at a much larger bitrate answers a different question from comparing both at the same delivery budget.

Align frames and transformations

Check that the first compared reference frame corresponds to the first compared distorted frame. A one-frame shift can compare a moving object against its previous position. If an encoder adds or removes frames, investigate the timing mismatch instead of averaging through it.

Apply the same intended spatial comparison to every candidate. If one output is lower resolution, document the scaling path used for assessment and playback. Otherwise, the test can accidentally reward one preprocessing choice rather than the codec configuration.

Inspect a few matched frame pairs manually at the beginning, middle, and end. This catches trimming mistakes and timebase drift that a successful command exit does not explain. Treat those checks as part of preparing the metric, not as optional review after a surprising score.

Pin the metric configuration

Save the VMAF implementation version, chosen model, command or script, filter options, and any preprocessing. Model generations are not interchangeable labels on the same ruler. Do not compare a historical score produced under one configuration with a new score produced under another and attribute the whole change to your encoder.

Follow the repository’s usage documentation for the installed toolchain. Confirm that your FFmpeg build actually includes the required filter if that is your integration path. A command copied from another machine can fail or use unavailable options.

Keep the machine-readable metric output beside the local test manifest. Duck Cloud’s JSON Diff can help inspect sanitized result structures, but it does not perform video assessment or validate the underlying experiment.

Look below the whole-video mean

Consider an invented two-minute clip with 110 seconds of easy content and ten seconds of fast motion. A strong overall average can coexist with visibly poor quality during those ten seconds. The mean does not tell you when a viewer sees the problem.

Inspect per-frame or per-scene results and review the lowest-quality intervals. Keep scene boundaries consistent across candidates. A brief title card should not carry the same business importance as a product demonstration merely because both produce metric samples.

Report a distribution and specific reviewed intervals rather than inventing a universal passing threshold. A score acceptable for an internal tutorial may not satisfy a high-quality archive, and different content types expose different defects.

Combine metrics with human checks

Watch the candidate at the intended display size and playback conditions. Check small text, edge ringing, blockiness, banding, and motion. Review scenes selected before seeing the scores as well as scenes flagged afterward, so the process does not become a hunt for evidence supporting the favored encoder.

Do not assume that every image enhancement improving a metric also preserves the intended appearance. Sharpening, denoising, and grain removal can change texture or artistic choices. Keep the desired output fidelity explicit.

Also verify file duration, seek behavior, audio alignment, and device compatibility. An output can score well in a controlled metric test and still fail as a delivered video.

Turn the experiment into a release decision

Build a table with candidate settings, bitrate, encoding time, model identity, aggregate metric, worst reviewed intervals, and compatibility results. Use the same source corpus for each row. Mark missing measurements rather than filling them from a different run.

If a candidate saves bandwidth but harms one important scene, try a targeted adjustment and rerun the complete comparison. Preserve the original outputs so reviewers can see what changed. Avoid claiming a percentage improvement in perceived quality from a percentage change in a metric score.

Conclusion

VMAF is useful evidence when the experiment is controlled. Align the reference, pin the model, inspect weak scenes, and combine the score with playback checks before deciding that an encoder setting is ready to ship.