Keep in mind that even though a project is GPL, that does not mean you can't contact the author's about an LGPL option on the underlying code.
A possible roll your own solution would be to use openGL as a compositing surface for the video and obtain a behind the scenes library like ffmpeg if you need to process specific video types.
NeHe has an example of rendering AVI's to openGL: http://nehe.gamedev.net/data/lessons/lesson.asp?lesson=35
FFMpeg has recently been ported to iPhone and is an LGPL based product: http://geek.thinkunique.org/2008/03/05/ffmpeg-on-the-iphone/
(Note: There is some debate over the commercial use of LGPL on iPhone because the license references the phrase "dynamic" when referring to library linkage, which iPhone doesn't allow. I have not seen any project teams balk at their code being used on the iPhone statically, but you should contact the authors directly for clarification.)
Another (though GPL) version of an OpenGL video player is: http://code.google.com/p/glover/
What your getting through a solution like this is basically a bypass on the iPhone/Mac/CALayer specific technical details and leveraging an existing knowledge base of video through OpenGL which although not extensive, is still broadly supported.
If you are dealing with a specific video style, then you may want to see if a library is avaiable for the specific video format direct from the vendor instead of using a multi-purpose tool like FFMpeg. Once you have the compositing working, the video can come from most any library.
Barney