I don't believe there's a public way to do what you're talking about. Your best approach is probably to reverse-engineer the iChat AV system (the receiving side) and see if you can replicate it. Some initial work has been done by the ICP project. It's very sketchy, but it's a start.
Another approach is the QuickLook API, which has the advantage of not having to run the source application. So far Apple hasn't made the reading side of that API available either. Ciarán Walsh did some handy reverse engineering on QL a couple of years ago, and I've played with that approach, but it is somewhat klunky. You can generate the panel as Ciarán explains, but put it off screen. You can then copy the contents into an NSImage using NSBitmapImageRep -initWithFocusedViewRect:
. Unfortunately you can wind up with some funky visual artifacts in this (like scroll bars in some cases), but for some applications it can be effective.