The argument "Eclipse for Java" is a different argument than "Eclipse for [something that isn't java]". Eclipse does rock for Java.
I mean, vi is like a screwdriver, or maybe a swiss army knife, and Eclipse is like a big CNC combo mill and asphalt spreader. You don't exactly compare them, you kind of just use both.
Also, are you working inside something giant, which you know little about, but which Eclipse understands? An example would be working on Eclipse itself. Here, Eclipse has perfect visibility, total language support, and you need the toast prompts and the documentation links.
But if you are typing in a 100-line Ruby program to convert an SQL database, Eclipse doesn't add much value, especially considering its baggage.
It's also critical to set up vi right, or you won't grok the appeal. Autoindent, showmatch, tab handling, and various other options should be set. You should have a easy way to generate a tags file. Google can find tag generators, or just write one from scratch, with a few lines of shellcode and sed(1).
I don't consider refactoring to be a criteria. That's not something you do once an hour or even once a day. Sure, fire up the big IDE when you need to refactor. Oh, and don't expect automated refactoring of anything except Java.
Finally, vi can actually do a lot of things that the IDE can't begin to do. The grouped regular expression global substitution is kind of a generalized refactor-anything engine. To appreciate the vi gestalt you need to learn the line (":") mode. Briefly, it's like having sed(1) inside your editor.