VMAF can’t see what ñü.media does.
so we test it five ways.

what VMAF is good at

VMAF is Netflix’s open quality score. It compares every frame of a video with the original and predicts, from 0 to 100, how viewers would rate it. For ordinary codecs it works well, and we keep publishing it, unmodified, next to everything else. When nu.media sends less detail than the original, as in its low-data mode, a low VMAF score is real lost detail, not a flaw in the test.

why it isn’t enough for ñü.media

• It only measures sameness. VMAF rewards pixels that match the original. When a host or viewer device rebuilds a picture at 4K, or adds frames to reach 240 fps, the new pixels can’t match a source that never had them, so VMAF counts them as error even when people prefer them.
• It has nothing to compare against above the source. A 720p, 30 fps original has no 4K or 240 fps version to score against. VMAF can only score the output shrunk back down, which hides exactly what changed.
• It was trained for television. Its models assume 8-bit SDR, ordinary frame rates and a living-room viewing distance (the 4K model, a large TV up close). It wasn’t trained on neural reconstruction, HDR, frame rates above 60 or a phone held at arm’s length.
• It averages away failures. A few broken frames disappear into the mean, and dropped frames don’t count at all unless the frames are aligned first.
• It can be gamed. Sharpening raises VMAF without making video better, which is why Netflix also publishes VMAF NEG, a version that ignores enhancement.

what we use instead

Every clip is scored in five layers. A result counts as “looks the same as the original” only when every gate passes, never on one average.

1. Fidelity, at the source’s own size and frame rate: VMAF and VMAF NEG, PSNR-Y, MS-SSIM and Netflix’s CAMBI banding score, with the mean, the worst 1% of frames and the worst tile. Large frames are scored in 4K tiles, so 8K and larger sources work.
2. Perception across screens and frame rates:ColorVideoVDP from the University of Cambridge, which models the display’s size, distance and brightness, HDR, and frame rate. Its JOD scale reads 10 for identical.
3. Timing: dropped and repeated frames, the longest stall, the frame rate actually shown, and the peak bitrate in any one-second window, not just the average.
4. Enhancement checks for pictures rebuilt on a host or viewer device. The enhanced output, shrunk back to the source’s size and frame rate, must still match the source, so enhancement can add detail but can’t change what was there. Generated detail and frames are marked as generated in the stream’s provenance record.
5. People. Blind viewer tests to ITU-T P.910 (each clip rated alone, with a hidden original) and ITU-R BT.500pairwise comparisons, on real iPhones at arm’s length. We publish mean scores with 95% confidence intervals and every raw vote. When people and the metrics disagree, people decide. The charts on this site show the metrics.

check it yourself

You don’t need any of our code. Every published result lists the device, operating system and date it was measured on. For each clip we publish the exact bits that crossed the network, the decoded or enhanced output, and a SHA-256 hash of each, so you can confirm you have the same files and score them with public tools:

shasum -a 256 original.mp4 sent.mp4 shown.mp4

# Fidelity at the source's size: VMAF, VMAF NEG, PSNR, MS-SSIM and banding (ffmpeg built with libvmaf)
ffmpeg -i shown.mp4 -i original.mp4 -lavfi \
  "[0:v]scale=720:1280:flags=lanczos,setpts=PTS-STARTPTS[d];[1:v]setpts=PTS-STARTPTS[r];\
   [d][r]libvmaf=model='version=vmaf_v0.6.1\:name=vmaf|version=vmaf_v0.6.1neg\:name=neg':\
feature=name=psnr|name=float_ms_ssim|name=cambi:log_fmt=json:log_path=scores.json" -f null -

# Perception, including display and frame rate (pip install cvvdp)
cvvdp --test shown.mp4 --ref original.mp4 --display standard_phone

# Bits on the wire: average and peak bitrate come from the packet sizes
ffprobe -v error -select_streams v:0 -show_entries packet=pts_time,size -of csv=p=0 sent.mp4

files

what stays private

nu.media’s encoder and enhancement code aren’t published. The tests, the files and the numbers are public, so you can verify every claim without trusting us or seeing the code.