Security best practices start with locking EVERYTHING down, then only opening up pieces as necessary to the people that actually need to get to it.
By allowing the "public" access to even connect, regardless of the fact that they don't have rights to do anything, opens the door just a crack. Sometimes, that's enough to exploit a buffer overflow or another type of attack.
UPDATE
A Login and a User are two different things. A login is server level, a User is database specific. Every User should have a corresponding login; but not every login will have a database user account.
Let's say you are using Active Directory integrated security; and you've assigned a particular AD group to have access to the server. Now, only some of those within the group may actually have access to certain databases. Those are the people you want to allow access to connect. Whereas the other members of the group don't have regular database user accounts.
By definition this is a problem as some of the group members can connect and list databases on the server even though they don't have access to any of those databases. That's the "crack" in the door I'm talking about.
If you can stop those people from even being able to connect then you've gone a long way towards securing your server.
There are other server level permissions that the Public role grants to logins. If possible, you should turn it off just like you would a guest account.