I just downloaded the PS3 media server and it seems that in only encodes to MPEG2 by default, which I have no knowledge about. (if it can be tuned to meet your demands).
x264 can most likely do what you want, but I don't know if it can be used with PS3 media server. I only found an old thread about it on their forum:
http://www.ps3mediaserver.org/forum/...php?f=6&t=3025
I don't know whether support has been added in the mean time.