Lots of small classes sounds just fine :)
Particularly if you let each class implement an interface and have the different collaborators communicate through those interfaces instead of directly with each other, you should be able to achieve a so-called Supple Design (a term from Domain-Driven Design) with lots of loose coupling.
If you can boil it down so that important operations have the same type of output as input, you will achieve what Evans call Closure of Operations, which I've found to be a particularly strong design technique.
What tend to happen when you apply the SRP is that although all classes start out small, you constantly refactor, and from time to time it happens that a rush of insight clarifies that a few particular classes could be a lot richer than previously assumed.
Do it, but keep refactoring forever :)