I'd say a restricted feature-set is always language-dependent. For instance, many languages will be programmable using the code-compile-run-debug cycle (e.g., C, C++, Java), where others are more interactive, like Python and Common Lisp.
So, for instance, in the first group I'd want an IDE that is capable of launching a compiler (linking its output messages to my code), running the produced binaries, put break points on the code and inspect stack-traces on breaks. (Eclipse is an example of such IDE)
For the second group I'd rather use a Read-Eval-Print-Loop based IDE, that allows me to evaluate code bits on the code artifact itself, inspecting variables, and generally keeping a tight relation between the code artifacts and the run-time memory state - dynamically. (Emacs+SLIME is an example of such an IDE).
This is not, by any means, a complete coverage of Programming Language groups/families; it was just to make my point, an IDE must offer a tight connection to a specific PL development style. Some other features, like code completion and syntax highlighting are too general, and thus they made their way to those programmer-oriented-text-editors. But the 'ID' in IDE suggests Integrated Development, so it should cover all you need to do without resorting to additional tools, such as a terminal (or at least, have that external tool-set reduced to a minimum!).
About specific features, like the mentioned visual/diagram editors, they make sense if you use it to develop your program. So UI editors make sense for IDE's for a subset of all programming languages only. But things get a bit harder to define once you consider mind-map editors and "virtual" whiteboards/sketchbooks. It's obvious that you may use it to develop a product in any PL, but it's not that obvious (for me, at least) that an IDE should provide support for them... The same goes to collaboration support. So that kind of features sure do enrich an IDE - as well as any productivity tool. Maybe this is a sign that it should be provided by the operating system? :) I believe the notion of Integrated Development covers the entire cycle from input artifact (programmer-produced input) to final output binary. This means things like sratchpads won't be a crucial part of the IDE, but a UML diagram editor will probably be (if code can be inferred from it).
There are other SO threads talking about favorite features on IDE's, see e.g. this one.
(Out of curiosity, how would you go about picking the "right" answer for you? This looks more like a discussion starter, maybe a wiki, not so much like a Q&A thread)