ok I googled myself.. should have done that lol
here is what I found
http://arstechnica.com/guides/tweaks/ipod-video.ars
As you can see you have to follow the requirements.
Both the PSP and iPod can both only play H.264/MPEG-4 encoded videos with AAC audio tracks and not much else. Within that, there are limits to the resolution and bit rate of the video files you can use on each device. The iPod's tech specs page states:
H.264 video: up to 768 Kbps, 320x240, 30 frames per sec., Baseline Profile up to Level 1.3 with AAC-LC up to 160 Kbps, 48 Khz, stereo audio in .m4v, .mp4 and .mov file formats.
MPEG-4 video: up to 2.5 mbps, 480x480, 30 frames per sec., Simple Profile with AAC-LC up to 160 Kbps, 48 Khz, stereo audio in .m4v, .mp4 and .mov file formats
See thats what you need to do.. Now you gotta find encoders that does this kind.