One consideration for this is just how frequently said libraries are reused. If they tend to be used across completely different projects, you have to be careful not to paint yourself into a corner, where you find out that changes you made three weeks ago to the library to support some new app turned out to be breaking changes for a few other applications.
Basically what I'm saying is that you don't want to have to resist the temptation to make changes to a library willy-nilly; better to not put that temptation in front of you in the first place. If a library is designed for reuse, any significant changes to it should be designed and implemented very carefully, and thoroughly tested against all dependent libraries/apps. It becomes considerably more difficult to take a disciplined approach when you literally have the source right in front of you, waiting to be modified.
My approach is to create solutions of related libraries; for example, I might have one assembly for the core interfaces and abstract classes, a few other assemblies for different concrete implementations, another for unit tests, and so on. If there are layers of dependent reusable libraries then they will often all get lumped into the same solution.
But it stops at the application level. Any project that's not always going to get deployed with the core libraries does not share a solution, it simply references the compiled DLL. It forces me to be disciplined about library changes and not start tweaking it in order to support some specific UI function.
I don't know if this is the "right" approach, but I have been bitten before by making premature changes to libraries without properly testing dependencies, and it's always been a result of being too focused on a single app and not thinking about side-effects. My take on this is that when you work on a library, you need to be focused on the library itself and not how it's being used in a particular scenario.