I need to create an architecture diagram for an existing system. I want to show the "big picture" as well as small details.
What makes more sense:
You can use a UML tool like Sparx EA, especially if your problem is written in an object-oriented or service-oriented style.
Personally, I find single, all encompassing diagrams to be too busy. I have trouble getting that gestalt if there's too much going on. I prefer a larger, simpler picture with accompanying drill-down diagrams. It could just be the way I'm wired.
In general, in most areas of development it is generally a bad idea to mix levels of abstraction. In many ways, OO is a way of organising levels of abstraction while encapsulation is a way of enforcing it.
In my opinion, this also applies to design. So I think you should create your diagrams at the level of abstraction that is appropriate. If you are not sure what hardware will be use and what 3rd party SW will be used then you may need a seperate deployment diagram. If Software layout is a no-brainer then you may want to start sketching out ideas using class diagrams or sequence diagrams.
I think it is appropriate to the apply the 7+-2 rule - in that developers can only succesfully keep 7 (+-2) concepts in their mind at one time - having an all encompasing diagram with all levels of abstraction would break that rule!!!
Having said that, only spend time on creating diagram if it is going to be beneficial to the team and drives the project forward. If the best way to proceed is understood by the team, then diagramming that understanding isn't really going to add any value. However, if you are unsure of the next step, then diagramming can be a useful way of communicating ideas between the team - only adding enough detail so that the idea is clear. But don't be afraid to throw diagrams away too!
In my opinion the following book provides the best demonstration of the level and detail of diagramming that is useful for a project:
http://www.amazon.com/Agile-Principles-Patterns-Practices-C/dp/0131857258
Your first option sounds like the right one (one diagram for the big picture, and a separate diagram for each smaller component) - and if you use a modelling tool you'll be able to link it all together.
High-level conceptual diagrams are good for establishing the context of the system and it's breadth - a white-board is good for this. Bonus point: I always take photos of white-board diagrams with the camera in my phone, sync it up with my PC, and email out copies as necessary.