Framemaker is just too expensive. The book will have source code (listings) and needs an subject index, and a lot of styles formatting. (on Mac OS X)
Depends what you're planning to do with the book. If you're actually getting it published, the publisher will generally have requirements as to what software you have to use. It's often Microsoft Word.
There's a lot of good information here:
For most tech books, I'd recommend LaTeX. There is the MacTeX distribution for OS X. LaTeX allows you to use semantic markup for layouting and defining your own styles, although the defaults are of high quality and wide spread use in the academia.
LaTeX is pretty much the standard for books that require any type of Scientific notation.
The software required will pretty much depend on what your publisher/editor requires (or the publisher that you are targeting needs).
I avoided using LaTeX to write my dissertation by using MS Word 08 (Mac) + MathMagic Personal Ed to do the formulas - but I regretted it. Turns out that the drag and drop feature from MathMagic to MS Word didn't work very well - when the word document was opened in windows all of my formulas were parsed incorrectly, and extraneous characters were added (workaround is to turn your formulas into jpegs within MathMagic and then import them).
Anyhow, lesson learnt: use the standard if you can and avoid workarounds.
Mellel has very powerful styling. And it can export to MS Word. However I do not know how accurate this export feature is. I would suggest downloading a trial and take it for a spin for some time.