The complaints about IoC are easy to understand: IoC turns something simple into something complicated. Let's say you wanted to do something to each element of a list (in pseudocode):
for each i in list:
do_something(i)
IoC, essentially, is moving the responsibility for loop iteration to someone else. Thus, we end up with something more like this:
ioc = new IocContainer()
ioc.register_callback(do_something)
ioc.go()
Notice that even in this simple form, the code is longer than the original... and that's not counting the implementation of IocContainer.
Then, in its fanciest form, the IocContainer can get initialized statically or by a static Main() function or somewhere else hidden away, and the register_callback() functions get called automatically based on some other code that iterates through an XML file that lists all your do_something() functions and (sometimes) even the contents of the lists to iterate through.
Yay, the XML file has made your software more configurable! Right? You can change which somethings are done to your list just by changing a simple text file!
Except, ironically, your source code is now longer and harder to understand, it depends on all sorts of new, possibly buggy, libraries (including the IocContainer and an XML parser), and you've actually made it harder to maintain: XML code is harder to understand and edit than the original simple source code loop (no matter what language the loop is written in). You also have less control over exactly how you want to do the iterating (sorted or unsorted? depth-first or breadth-first?) since IocContainer is taking that responsibility away from you.
For things like plugins, it can make sense to use IoC, since it's logical for the main application to be in control of the execution process, and to just ask the plugin for help here and there. This is called "providing hooks" and has been around much longer than the IoC terminology.
For things like unit tests (which is where I've usually seen it thrown around), IoC usually turns out to be unhelpful, because your tests need to be able to simulate a wide variety of odd situations, and IoC just constantly gets in the way of this.
The fundamental assumption of IoC is that loading data and looping through is somehow difficult and needs to be abtracted away; this assumption just isn't true, because it was never more than a couple of lines anyway (or you could move it to a separate function), and even if it did save code, it reduces your flexibility.