This is possible. There is a project called SharpUSBLib that you should check out. This a wrapper library written in C# to the USBLib for Unix & Linux which is written in C. There is also a port of USBLib to Win32.
One of the things that you may need to be aware of is that you may not need the SharpUSBLib depending on what type of USB device you are trying to talk to. For instance, most USB HIDs (Human Interface Device) like Keyboards and Mice are full Plug'n'Play capable and require no drivers or interface libs. I have a small USB Unitech barcode scanner which when plugged into the USB port works like a keyboard and the system (both Linux and Win32) see it as a HID.