Is there any down side to this? Is it just that compilation will take longer?
No, not really. It does mean that every time you make a change to any header, the whole project gets recompiled. But on the other hand, because it is precompiled there might be a saving in the case where you need to recompile but don't change your headers.
There's an advantage to stuffing all your headers in the pch file because it means you are less likely to omit a header by accident. Sometimes Objective-C compiles things wrongly if it doesn't have a header file e.g. if you have two classes with a methods declared like:
-(void) doSomethingWith: (float) aNumber;
and
-(void) doSomethingWith: (double) aNumber;
without the correct headers, Objective-C might guess the wrong doSomethingWith: and pass a double where a float is expected or vice versa.
Having said all that, I never bother with the pch. I tend to only import headers in .m files where possible and forward declare classes in header files.