[peter the llama]

Originally Posted by Starks

I may not be the best encoder out there, but I know when its time to embrace the future.

I want to dump Xvid (and possibly VirtualDubMod) and learn how to use x264 and MeGUI.

Good call: H.264 compresses anime MUCH better than MPEG4 ASP (xvid) can. The in-loop deblocking filter and multiple reference frames esp. are very effective on animated content. With no film grain and lots of solid colour areas, you can often get away with very low bitrates.

So, what kinds of barebone settings should I be running in MeGUI for anime?

Lots of reference frames helps more on anime than in live action. If a character walks in front of something, it will usually look _exactly_ the same half a second later when it's revealed again, and in some animated content you have repetitive mouth or other movements. 1/2 second = ~12 frames.

Also, what filesize should I aim for if I have 1024x576 raw?

Would someone please tell me why everyone always scales video before encoding, instead of keeping it anamorphic and letting the player scale it on playback? What kind of player imposes this stupid requirement, and why not just let people with some bogus hardware transcode or something.

Maybe some hardware players suck and couldn't read an aspect ratio from an AVI (maybe only mplayer does that), but Matroska supports aspect ratio info in a standard way, so any MKV player should handle non-square pixels. H.264 itself includes a PAR (pixel aspect ratio) in the bitstream, with no container support required. I _HATE_ all the low quality rips out there that lose vertical resolution by scaling to 1:1 pixels at 640x272 or something. 1024x576 at least doesn't lose resolution, but upscaling makes the codec use more bits to encode the same actual information. Always minimize the amount of transformations you put your video through.

Actually, I don't know about the MP4 container, and I've heard that e.g. apple tv doesn't support h.264 PAR != 1:1. Is that why people still aren't making anamorphic rips? A comment in http://www.linuxjournal.com/article/9005 says that MP4 can have an aspect ratio set for the whole picture, and if quicktime doesn't show it right you can manually set the display size.

As for filesize, it totally depends on the content. I usually aim for a quantizer of 18-20 with live action, because any higher and you start to notice artifacts if you watch SD content on a nice 22" widescreen monitor and sit close to the screen for full effect. I usually encode some of the source at crf=18, pass=1, to see what kind of bitrate that gives me. If I like it, I can do a pass=3 at a similar bitrate with the same settings (I use mencoder on GNU/Linux and write shell scripts to encode stuff, since none of the tools I've found can do a decent h.264 encode and get the audio in sync in a mkv... Don't know if megui is good about putting the passlog file somewhere you can use it to do just a second pass). So yeah, encode a 1 min clip at a few different quality levels (crf=18, crf=20, ...) and see what the bitrate vs. quality tradeoff looks like. It's not easy to pick a clip that's representative of what the whole movie will do, though, since dark scenes or lots of action make a huge difference to required bitrate. And stuff with more texture will only look good at lower QP.

That said, anime looks good at higher QP than live action. esp. simple shading, with blocks of solid colour. With higher QP on live action, you lose things like skin texture from a closeup of a face. If you don't have detailed textures or film grain, QP=25 still looks good. e.g. I'm currently transcoding some Family Guy episodes from 1000kbit xvid (576x432) to h.264. At QP=25, the only thing I've noticed so far are that some lines are slightly stair-stepped (probably encoded as just solid 8x8 or 4x4 blocks, without a lot of detail in the blocks). I'm hoping to fix that by giving I frames more bitrate, since the artifact is only noticeable because it stays on screen as a reference picture for so long. It's ok to have very minor artifacts in moving mouths, etc. Family Guy is a classic case of easy-to-compress animation. There are large blocks of solid colour, static backgrounds that just pan if they move at all, and long periods where the only part of the picture changing is someones mouth. At crf=25, season 4 ep. 01 compresses to 411kbit/s. That's with everything cranked, IIRC:
crf=25, subq=7, me=umh,
frameref=10, mixed_refs, direct_pred=auto, maybe even me_range=32 or something
bframes=5, b_pyramid, weight_b, brdo, bime,
8x8dct, partitions=all (i.e. include p4x4, which defaults to not included)
keyint=500 (fewer I frames: esp. good with static backgrounds)

trellis=2 is a waste of time. partitions=all, instead of the default (all but p4x4) is not always useful for live action, but I don't know about anime. It's recommended only at low rez, according the the mplayer man page.

I'd suggest as a base something like:
bframes=5, b_pyramid, weight_b,
me=umh, subq>=6, (major speed impact, but very important)
8x8dct, turbo=1 (only does anything when pass=1)
frameref >= 4, (major speed impact, but also fairly large quality gain)
everything else at defaults (dct_decimate, chroma_me, fast_pskip, direct_pred=spatial)

That's kind of a basic good quality setting. Higher quality would be
brdo, bime
me_range=32 (depending on the anime, if things move in steps > 16 pixels, then I think this will help. It can make it a lot slower, though.)
frameref=9-12
mixed_refs, maybe partitions=all
trellis=1, no_dct_decimate (only in 2pass. apparently trellis isn't a good idea with dct_decimate, or in crf mode.)
direct_pred=auto (usually goes 99% spatial on live action, maybe bigger impact on anime)
keyint=500 (trade off seek accuracy for fewer bits spent on I frames)

I sometimes use qcomp=0.7, to get more constant QP and less constant bitrate, since I don't like having any sections that look bad. This might not be sensible, since quantizer and quality aren't at all the same thing.

I also use nr=100 or nr=200 on sources with film grain, but it tends to lose fine texture detail. Probably not an issue with anime from PAL DVDs, unless it's a terrible MPEG encode like some of the copies of Castle of Cagliostro floating around...

BTW, 1000kbit/s is probably a good bitrate for DVD rez live action with settings cranked up. A lot of animated stuff will look fine at lower bitrate. Look at the SSIM and/or PSNR numbers, and see how that varies with bitrate/QP for the source you're encoding.

Hope this helps. BTW, the main reason I replied was to express my shock and horror at people upscaling before encoding. Please someone tell me why! Have people never seen e.g. http://handbrake.m0k.org/trac/wiki/AnamorphicGuide