Here's my take on it...
I use a framework when I either want to build a one-off system, or rapidly prototype a system. Notice the word prototype. Be prepared to throw it away after it's built. The one-off system note is where you want to build something to do a well defined task, and the risk for both changes and copies are small (And hence maintenance is not a huge concern).
I use a set of libraries when I want to build a system that will have many copies made, or will need to be modified extensively over the course of its lifetime. Basically, this enforces a proper architectural design step from the beginning. This way you can ensure that the application will be designed to suit your requirements and anticipated changes from the beginning, rather than treating it as an afterthought.
Notice I said "set of libraries". What I mean by that, is that I take a "framework" that I like, and use it only as a set of helper functions. I don't build the app in that "mentality". I actually build the app to the architecture that was laid out (which often differs significantly from the frameworks preferred approach). I use the libraries when I can, and write custom classes where I can't.
Why do I say this? Well, my view point on it is simple. What makes a modern PHP Framework a "RAD" framework (Rapid Application Development)? Basically, it lets you skip the high architecture (which rightfully takes a fair bit of time), and get right to coding (since the framework developers did an architecture round for you). The problem with this viewpoint (from my perspective) is that there's nothing to say that the architecture takes into account the things that you need it to. It's a generic approach to a generic problem. The second you have an actual problem, a specific approach (if intelligently designed) should be far better than any generic approach...
I do use pre-made frameworks, but I use them depending on the need of the problem. If the problem warrants a full blown architecture (most that I deal with do IMHO), then I am not using the full potential of the framework. But I am trading some of that potential for making the application behave exactly as I need it to. It's a tradeoff, and you must realize that. But I feel that what I gain is far outweighed by what I loose... You must make the choice for yourself.
Just my $0.02...