The best way I have found to do complex UI is to forget all the Nielsen metrics and so on and just paper prototype it with paper and colouring in pencils (Crayons if you must). It doesn't matter how rough it looks. Just getting the functionality in there is enough.
There is actually a book on the subject with a forward by Jakob Nielsen you can buy off the site.
http://www.paperprototyping.com/what.html
I have paper prototyped a few projects, iterating through many prototypes, eliminating UI issue after UI issue before I even got to the point of writing a single line of code. I've tested paper protoypes on my brother (he can usually figure out most stuff), and if it passes the brother test I go for the father test (Dad is quite good at not being able to figure out some interfaces), and once it passes the Dad test I go for the Grandmother test (where I test out my prototype on my Grandma). If software passes the Grandma test you really know you are onto a good design.
I find I develop so much faster too when I know exactly what I'm building after I've knocked out a paper prototype. I wish companies I've worked for in the past would adopt a similar aproach (I've tried to get them to) but they seem to think it is childish (it's not)
If you need ideas about how to achieve certain things then ui-patterns.com is also chocked full of ideas. ui-patterns are a little like design patterns only for user interfaces.
Hope this helps.
John
P.S.
Excel also makes quite a good prototyping tool (although I've always stuck to paper)
http://www.amit-deshpande.com/2007/09/prototyping-using-excel.html
I have a spreadsheet with a lot of controls set up already on it. Write me if you want a copy.