Embedded Linux Read and Write Video from a USB port

Question

I've got an embedded Linux project where I need to read video in through a USB port, make some modifications to that video (such as compression), and then send it back to the host via preferably the same USB port. It would also be a preferred feature if we could send commands to change things such as the video compression level from the host to the device.

The host can be assumed to be a desktop running some version of Linux, and the device which actually does this video processing is a Gumstix running embedded Linux.

I'm basically at a lost as to how to start this project or what to research in the division of input/output in Linux. First off, does it sound possible to both read video, write video, and send commands through the same USB port? The video is relatively small, especially after compression by the device, so bandwidth isn't a problem. The main problem is knowing where to start research for this project. Is there anything like TCP/IP for USB where you could open multiple connections between the host and device to transfer data?

Will we have to write our own USB device drivers? Develop our own protocol or does USB support doing multiple things on the same port relatively easily?

Should I begin researching Linux drivers, user level programming and APIs, or the USB protocol? Or something entirely different?

I'm a senior computer engineering student, so my experience level is a mix of programming in C, C++, Java, Verilog, TCL, etc. Almost all socket related programming has been in Java (OS independent) so I really am not sure where to start on writing a Linux application to perform such a task. I've had a decent amount of experience with microprocessor programming (AVR, coldfire), but again despite being close to what I need, it isn't really helping me figure out where to start this.

Answer 1

What you want is called a "Gadget" driver on Linux - a driver that makes Linux act as a USB device. This LWN article on the USB composite framework is a good starting place.

Answer 2

USB ports come in two flavors; host which is the controller of the bus and is found on PC's and device or in Linux speak a gadget (there is a version commonly found in stills cameras where the port can switch type at connection called USB on the go). A host port can connect to multiple gadget ports (via hubs). The host is aways very much in control with gadgets operating as slaves. The hardware for the two port types is very different and many embedded cores contain examples of each type of controller. The Linux kernel contains host drivers for OHCI EHCI and UHCI host controller hardware and various protocol drivers (printers, keyboards etc). For some device types the USB organization has implemented standard protocols so you don't need manufacturers drivers for USB disk drives as they should all comply to a free and publicly available standard.

If you want your device to talk to a PC host you will need to implement a device/gadget interface, there is far greater variety in hardware designs for the other end of the cable, but you should find the Linux kernel supports a selection of the more common ones.

The gadget directory also includes a selection of protocol drivers as well. One trick that could save a lot of work would be to configure your device to appear as a USB network interface, this avoids writing a lot of low level stuff and you can take advantage of the network diagnostic tools and the USB layer is abstracted out of your application on both ends. This even works with a Windows host without needing to write custom Windows drivers (XP only supports the Microsoft RNDIS USB protocol and the Windows driver is buggy and can hang on disconnects but the kernel includes a RNDIS wrapper and this solution can be made to work well).

I wasn't sure from your question is the video source was an independent USB device in which case it will need a separate USB host port on your device or if it was supplied by the host.

Get your self set of kernel sources and study the USB gadget directory, also download the USB specifications from USB.org You will want to understand chapters 8 9 10 11 as they explain what happens once a host detects a gadget and also the concept of endpoints. Mindshare did USB book that was a ok as a quick introduction to USB.