Using a framework or not using a framework means you're making a choice between
Default Application Performance under load
Speed/Stability of Development
If you decide not to use a framework, you still need to do the things a framework would do. You're just coding them yourself in raw PHP, or developing your own framework that can remain lightweight since it only has to do what you want it to do, and not what the world wants it to do. You will get better performance, but you'll spend more time developing and debugging that code that a framework handles for you automatically.
What a framework buys you is speed in development time. You don't have to write out long complicated SQL queries, or debug someone else's long complicated SQL queries. You just need to create a table and instantiate a model. You don't need to decide where you're going to escape your SQL paramaters, because the framework defines where that happens. You don't need to get into huge political fights over where the business logic vs. presentation logic goes, because the framework defines this. A framework remove the need from having a system developer on your team, or removes you from having to think about/waste time on system development. You can get to coding your application faster and get measurable, visible results sooner.
Here's another way to think of it. PHP Frameworks are slower than PHP, but PHP itself is slower than C. Why not write your application directly in C?
There's no right answer here, it's one of those software engineering/development questions that's a matter of what your current situation demands. The default choice of the industry these days is to use a framework, because if you don't your competitors will release an application that has slower PHP processing than yours, but hits the market three months earlier.
Finally, one last thing to consider from that talk. Rasmus said that Most of the time the perceived performance of your application is in the frontend. Both the Javascript code and how the browser is caching the requests it makes back to your server. PHP is an awful, horrible language that's rarely the bottleneck. When it is the bottle neck, you can usually make a few adjustments (opt code cache, focused refactoring) that will remove the performance bottleneck.