Quality Comparison
Due to the amount of encodes I performed for this article, it's hard to produce every single relevant side-by-side comparison for this section. Instead, I've included what I believe are the most relevant comparisons, and if you want to compare further, you can download this archive of lossless, full-resolution screenshots comparing each encode.
You'll need to look closely to spot the differences in the frames below. As we're looking at different forms of compression, the images generally look pretty similar, except in fine details. Typically you can spot the better encode by looking for sharper detail on faces and fabrics such as shirts; less blocking in blurred backgrounds or other smooth gradient areas; and fewer artefacts.
The frames that follow are 1100 x 600 crops of the full 1920 x 1080 frame. They are not downscaled in any way.
I'll start by comparing the difference in quality between the default H.264 preset in Handbrake (x264 Very Fast) and my custom settings. My custom preset delivers significantly improved fine detail, less background blocking, and better clarity during moving scenes. However, this isn't a surprise as it uses a higher bitrate to achieve this quality, producing slightly larger files.
It should be noted here that my custom H.264 preset comes the closest to the source material of all the encodes I performed for this article. The source does have higher quality throughout, though with a file size around three times larger.
Here you can see the difference between the two best presets in my opinion: x265 Medium and my custom H.264 preset. There is very little difference in visual quality, with a slight advantage to H.264 in very fine detail and background clarity. When playing back the video, the differences between the two are practically indistinguishable.
This is a great result for HEVC, as it exhibits nearly identical quality in a file less than half the size. Encoding times were increased by 45% on average to achieve this quality.
Interestingly, there is almost no visual difference between the Faster and Medium HEVC presets. If anything, Medium is slightly better quality when displaying facial detail, at a smaller bitrate.
And again, Medium isn't any different to Slow when it comes to visual quality. For all intents and purposes, these two presets are identical in quality, and with a very similar file size and significantly slower encodes, the Slow preset isn't worth using.
There is no point using Nvidia's HEVC encoder for low bitrate encodes: the quality is horrible in comparison to an x265 Medium encode, with reduced detail across the entirety of the frame. Even though the dedicated Nvidia hardware encodes the file much faster, the quality is so poor from this identical-bitrate comparison that it's not worth using.
There is a slight difference between Nvidia's HEVC encoder set to CQP mode compared to the x265 Medium preset. Medium has a small advantage in fine still detail here, and slightly less blocking, at a significantly smaller file size. Nvidia has a slight advantage in detail during high motion scenes, although this isn't surprising considering its superior bitrate.
Nvidia's encoder has a small quality advantage over the x265 Ultrafast preset in most circumstances, however Ultrafast encodes end up being significantly smaller.
By extension, Ultrafast encodes are also inferior to Medium encodes, although they take less than half the time to encode.
And just in case you were wondering how the HEVC Medium encode compares to the original, these are the comparisons for you. Note that the HEVC Medium preset delivers a file just one fifth the size of the original Blu-ray rip (525 MB versus 2.5 GB).
Viewing these cropped and (slightly) compressed comparisons on a computer monitor isn't the best way to directly compare the image quality. For the best comparisons, check out the archive that includes every scene I used above for every encode in high-quality images.