I think it's important to distinguish between a framework and a library when answering.
A framework follows the Hollywood principle: "Don't call us; we'll call you." You plug your code into the framework according to its API. The framework acts as a constraint that solves the particular problem it was designed to solve (e.g., web application development).
A framework will use a combination of your code and 3rd party libraries to solve a particular problem. It will treat your code as one of those 3rd party libraries.
If you eschew a framework, the roles are reversed. Now you and your code are in charge: "I'll do the calling, thank you." A library is a self-contained piece that plugs into the software that you write.
So why prefer a framework? Use one that is written better than the scaffolding that you would be able to write yourself. A framework is likely to be tested more thoroughly and have a wider user base than code you'll write.
You'd write an application without a framework if you're working on a specialized problem, you have deep knowledge of the domain, and there are no frameworks available that demonstrate deeper insight than you have.