What happens when a instance is idling with no connections being made. Does it still cost us money?
Yes, it costs money when the instance is powered-on, no matter if it does productive work or not.
It would be sad to have instances idling and costing us money when we do not use them...
The advantage of EC2 is that you can shut down idle instances and restart them later.
Of course, for a public-facing web service, you need at least one web server running at all time, so this applies more for peak-time extra capacity.
In contrast, Google App Engine manages server instance lifecycle automatically, and only bills for CPU cycles (and other resources) that you actually use. But in order for them to be able to do that, you are severely limited in what you can do, and have to trust them to properly scale your application (no way to take an active part in server deployment).