A Flex Application won't necessarily run slower than a Pure AS3 application once it's fully loaded - everything gets compiled down to bytecode in the end, and a Flex App is like an AS3 app that uses a LOT of other classes.
Think of the Flex Framework as a set of shortcuts that allow you to do things much more quickly, but the real cost is that your project gets filled up with a LOT more code - even if you're actually writing less code, and you never have to see the additional code.
I would disagree with the assertion that Flex should only ever be used for complex data-driven applications, though it's certainly very useful for such projects. You can use it for anything, as long as you understand that the final product (the .swf you export) is going to be a lot larger than it might otherwise be.
If having a large .swf is not that big a deal to you (and it might not be, depending on what you're doing with it) then I'd say give Flex a try because ArrayCollection, RemoteObject and Data-Binding will save you hours of frustration and hundreds of lines of code.
However, if you want to make sure your final app is as small and efficient as possible, do it in pure AS3 and simply opt-in to more advanced libraries as you need them.