Interesting question. I have thought of doing something similar myself sometime ago. I don't see the mobile server to be a problem, since you can always write a custom TCP application on the mobile device and connect it to a "proxy" application in the PC that will act as the VLC server.
The problem is the camera API. The blog post suggested by Matt is the best source of information you can get. Unfortunately not all driver implementations support the full DirectShow specification. This means that even if you climb the really steep DirectShow learning curve, you may find out that your device doesn't play nice with it.
I suggest that you start your research from the camera driver of the device at hand. It will be difficult or impossible to build something that it will work in all devices.