If there's any downside to including too many libraries?
Hardly any.
If the compiler works out which ones are needed and ignores the rest?
Almost exactly right: it's the linker that examines each library and pulls only the object code that it needs.
The one downside to including a very large number of libraries is that it can be hard to get them in the right order. Recent versions of the GNU linker have some special options that can help with the libraries-out-of-order problem (and it's about time too), but such options remain nonportable. On the other hand, if you include libraries that aren't actually needed, it's not going to matter what order they appear in, because the linker will carefully examine each one and decide that none of its contents are needed.
For all those people out there who are looking for projects, here's one I'd love to have: give me a tool that takes a list of libraries and does a topological sort on the interlibrary dependencies, then tells me an order I can put them on the command line so that there are no gratuitously undefined symbols.