I would think it would depend on how involved you want dependent assemblies to be in managing the settings.
A set-once-don't-think-about-it-anymore approach could be satisfied very well by the registry. A disadvantage of using the registry is lack of built-in support for retrieving the connection string in ADO.NET classes.
A this-might-be-different-on-a-per-application-basis approach might be satisfied better by demanding settings be in the application's app.config or web.config. The config files offer the added benefit of builtin connection string support for many ADO.NET-related classes.
I'm always a little leery of modifying the machine.config. I cannot explain why, really, except that the setting is global. Every connection string name used in a machine.config is a name that propagates to other configs and must either be used, ignored, or removed. It also complicates syncing development environments with production. That said, I do not suppose these reasons are super-compelling reasons to avoid the machine.config. Just weigh the tradeoffs before making a decision.
Custom files in isolated storage is another viable option, though it is typically used in limited trust environments; since the GAC is full trust, isolated storage may not be as compelling an option.