Very good question. I've personally spend quite a bit of time on trying to find documentation on the private api's, and even though I'm pretty sure I've come acros it some time ago, right now I can't really find it anymore. However, I do have some answers to your questions. Let me share what I have so far:
filesystem layout
- An old, but still
mostly true guide on filesystem
layout.
'replacing' apps
I think most of the 'hijacking' of original iphone apps is based on catching and re-implementing objective-c messages through posing and comparable techniques. A good guide on this is available here. I'm note entirely sure that this is the mechanism at work though.
tools
- A toolchain for iphone dev is available on google code. See this guide for info on how to install it.
- See this guide for info on how to use the default apple xcode environment for development on a jailbroken phone.
`