A quick search at avsforum.com for "ps3 maximum file size" suggests that files larger than 4 GB may be problematic. (This limit is not the one imposed by using FAT32 file systems on external devices.)
One solution you might try is to stream the files from the computer to the PS3 using one of the DLNA media servers.
This one has been getting very positive reviews. I tried streaming from my Linux box to the PS3 with this server, and it worked fine except for the various limitation on what codecs the PS3 chooses to play. I was just experimenting though; normally I just play the files from the PC directly to my HDTV set.
I recommend a little browsing at avsforum.com and doom9.org, two excellent resources for learning about video conversion and consumer electronic devices.