You could give mencoder a try. It's the encoding partner to the open-sourced mplayer media player and is usually bundled with it. Downloads for the OS X version are
here. You'll also need a codec pack from that page.
Warning: mencoder is a command line program with a LOT of different switches and settings. If you've never used anything but programs with graphical user interfaces you'll need to do some on-the-job learning.