As a general principle, I put Whoosh in the same category as SQLite: great for getting started, wonderful for single-user or really small-scale apps, but not suitable for large-scale deployment.
Whoosh is, in my experience, about an order of magnitude slower than Solr. A typical search against a bigish Solr index I've got in production takes about a hundredth of a second ; the same search using Whoosh and the same data takes roughly a tenth of second.
You should decide what's "fast enough" for you, but I don't think Whoosh is a good idea for anything where you expect high performance.