x264: Adaptive Quantization and You!

[Dark Shikari]

There's been quite a lot of improvements to x264 lately, mostly speed-oriented. Indeed, performance has gotten 10-20% faster in just the past 3 weeks, depending on what settings you use. The latest builds can be found at x264.nl.

But there has recently been one very major change that will certainly affect anime encoding with x264: adaptive quantization. Now in the official x264 and enabled by default, variance-based AQ (VAQ) attempts to improve quality in flatter areas of the image at the cost of more complex areas of the image. Generally, this helps resolve the classic problem x264 has of overflattening and blurring out fine detail in an image. Even in anime, which it wasn't designed for, the result can be pretty beneficial:



On the other hand, especially for those groups that love to release subs at absurdly low bitrates, AQ might not always be beneficial, since the only way to get acceptable quality at stupidly low bitrates is to just accept the blurring of fine details.

tl;dr: if you're one of those people who loves to make 40MB x264 rips of animes, you might want to try turning AQ off and seeing if its better or not to your eyes. As a general warning, AQ redistributes bits between frames, so some scenes will lose bits and others gain bits as AQ sees fit.

[checkers]

Or if it's sort-of breaking stuff, you can reduce the strength with --aq-strength. My tests with low-detail low-bitrate anime have basically shown there's very little difference either way. Obviously it produces magic & stars at high bitrates though :-)

[martino]

And how does it compare to Haali's AQ then? I remember Shikari mentioning somewhere that it wasn't as good for anime as for live action footage...

[Dark Shikari]

Originally Posted by martino

And how does it compare to Haali's AQ then? I remember Shikari mentioning somewhere that it wasn't as good for anime as for live action footage...

That seems to be somewhat questionable also at this point. Haali's AQ is good at avoiding blocking in "flat but not completely flat" areas, but does absolutely nothing for the kind of blurring shown in the first post.

Overall I think VAQ is probably better than Haali's, even for anime.

[guest0815]

Is VAQ supposed to save that much bits, without a noteworthy hit on encoding speed to boot? Those are the numbers for a minute of mixed material with no VAQ and default VAQ 1 there:

Spoiler:

[Dark Shikari]

Originally Posted by guest0815

Is VAQ supposed to save that much bits, without a noteworthy hit on encoding speed to boot? Those are the numbers for a minute of mixed material with no VAQ and default VAQ 1 there:

CRF values with very different AQ values are not comparable in terms of quality. Remember, CRF X with Y settings is not necessarily exactly the same quality as CRF X with Z settings, especially if Y and Z are very very different. That is not the purpose of CRF.

[guest0815]

How would I go about evaluating its use? I compared a few before/after frames and all I noticed was a slight image degradation but flat areas (provided I understand what those are correctly. I used backgrounds of a cloudy sky, a wooden table and a marbled floor) looked pretty much the same. Pixel may have been shuffled around a little bit but it didn't seem like an improvement to me. The raws could have been too smooth or maybe x264 is just so awesome that there is no detail lost to begin with . If I can't make out a visual difference should I have it turned on or off then?

[Soichiro]

The easiest way to compare would be to compare PSNR and SSIM values (which means getting rid of the --no-psnr and --no-ssim flags). Of course, this isn't the best way (which would be using your eyes), but it is simple and generally effective, as long as Psy optimizations are not being compared.

[Dark Shikari]

Originally Posted by Soichiro

The easiest way to compare would be to compare PSNR and SSIM values (which means getting rid of the --no-psnr and --no-ssim flags). Of course, this isn't the best way (which would be using your eyes), but it is simple and generally effective, as long as Psy optimizations are not being compared.

AQ counts as a psyopt, remember...

Originally Posted by guest0815

How would I go about evaluating its use? I compared a few before/after frames and all I noticed was a slight image degradation but flat areas (provided I understand what those are correctly. I used backgrounds of a cloudy sky, a wooden table and a marbled floor) looked pretty much the same. Pixel may have been shuffled around a little bit but it didn't seem like an improvement to me. The raws could have been too smooth or maybe x264 is just so awesome that there is no detail lost to begin with . If I can't make out a visual difference should I have it turned on or off then?

You should general leave it on unless there's good reason not to. Also, all comparisons should be done at the same bitrate.

[lubczyk]

Is there a point to leaving AQ on if you're doing CRF 18 encoding?

[Dark Shikari]

Originally Posted by lubczyk

Is there a point to leaving AQ on if you're doing CRF 18 encoding?

Sure--you often need QPs as low as 10-12 to stop banding in completely flat areas, so it certainly still helps.

[lubczyk]

Originally Posted by Dark Shikari

Sure--you often need QPs as low as 10-12 to stop banding in completely flat areas, so it certainly still helps.

Thanks for the advice.