Is it possible to fork() Cocoa application?
Yes, but you pretty much have to exec immediately. Core Foundation will throw an exception if you try to use certain Cocoa methods or CF functions between fork and exec (or without execking at all). You might get away with some things (I was able to ask a window its frame, for example), but nothing is safe.
Launching an NSTask, of course, counts as fork and exec together, averting the problems of skipping or postponing the exec.
How UI, threads, GC, KVO, runloops are going to behave when forked?
UI: Windows (the actual ones on the screen) are not duplicated. Of course, you can't talk to your NSWindow and NSView objects anyway.
Threads: Not carried over to the subprocess. This is not as good as it may sound, as problem cases abound; for one, another thread might have held a lock in the parent, which remains locked in the child even though the thread that held it is absent.
GC: Well, the garbage collector runs on a thread…
KVO: Should be fine, since observation is normally triggered either explicitly or by KVO-supplied wrapper accessors.
Run loops: One per thread, so the main thread's run loop should still exist, but it will die if you return to it.
Can I avoid creating standalone executable launched via NSTask?
Nope.
If I launch separate process, how can I send and receive ObjC object instances?
If you don't exec, you don't.
Otherwise, you can use DO.
(I'd rather not serialize/unserialize them myself, and I need to keep them after child process ends).
Then you'll need to make a copy in the parent process. I don't know whether you can use copyWithZone: here; probably not. I suspect you will have to do some sort of plist- or archive-based serialization/unserialization.
How does OS X handle this problem for Spotlight and Quicklook plugins?
Spotlight has mdworker; Quick Look has something similar.