Converting Video files on a Mac
Although Macs don't have that many encoding tools at allow you to convert MKVs or any video file to a format compatible with iPods and other devices, There are a few programs that allow you to convert these files and don't require much in way of steps to convert it to a suitable file.
Quicktime Pro and Perian
If you own a copy of Quicktime Pro, you can easily convert your video file to a MP4 which will be compatible to a device depending on what you select. Also, Quicktime Pro can hardsub your fansubs, so you don't really need to play around with other programs
1. Open the Video File (It can be any video file, including MKV if you installed Perian)
2. Click Export under the File Menu
3. Under Export, You can select to convert to any of Apple devices (iPhone/iPod/Apple TV) or MP4 format. Select MPEG-4 when converting to devices other than Apples (such as Sony's Playstation 3 or XBOX 360). When you select Export to MP4, you need to first select MP4 as the file format (Not ISMA, or it won't playback on PS3/Xbox 360) H264 as the Video Format, current for resolution (or HD resolutions if your video is in HDTV format) and also recalculate your bitrate (I suggest using a bitrate calculator to determine the best quality). You should also adjust the datarate of the audio if you wish. Once you are finish, just press Save and it should start encoding.
Handbrake
Handbrake is a good and also a open sourced video converter, but it cannot hardsub the subs. Handbrake is useful for ripping DVDs and converting into formats compatible with your devices (MP4). To convert a file, just click "Source" and then select the video file to convert. Then, depending on your device you want to convert to, just select the profile that matches your device and press Start and it should start encoding.
Handbrake have support for subtitles only on DVD movies, not MKV, although it's still helpful as a video converter. Handbrake is also available on Windows.
FFMpegX
FFMpegX is just a frontend to ffmpeg, a open source Unix encoder (costs $15 dollars, although you don't have to register it, but you should to support the developer). You have to manually download the files ffmpegx provide URLs to on the first one and locate them so that the files used for encoding can be used. Afterwards, drag the file you desire to convert to the From area. There are already presets that you can already use, but to hardsub your fansubs, you need to use the mencoder as the encoder in H264 format. This can be done by going to filters and checking the VOB Subtitles option. Note that this only adds the subtitles, but it won't have full support for ASS, so you are stuck with regular subtitles.
Also, make sure you have the best bitrate as possible. You can easily calculate the bitrate for best quality by clicking "Best" button under the bitrate calculator.
Before converting, you may also want to check the Audio bitrate if it's sufficient enough. (192 kbps or higher will give the best experience without any audio distortion with lower bitrates)
There are also profiles to convert to PSP/iPod/iPhone if you wish to use it, but you need to use menencoder mentioned earlier to burn the subtitles stored in a MKV file.
Sometimes, files encoded by ffmpegx may not recognize on the PS3, so you may need to do a reencode using Handbrake to a compatible format.