H.264 muxed from AVI to MKV is just as bad as plain H.264 in AVI!
It's just an AVI wrapped in an MKV. Don't do it.
Try with just the x264 commandline and see what happens. Personally I'd guess on either a decoder bork or some MeGUI fishyness. Can you upload a sample?
__________________
| ffmpegsource
17:43:13 <~deculture> Also, TheFluff, you are so fucking slowpoke.jpg that people think we dropped the DVD's.
17:43:16 <~deculture> nice job, fag!
01:04:41 < Plorkyeran> it was annoying to typeset so it should be annoying to read