Remember that the browser can download AT MOST two files in parallel from the same domain (that's the default on the modern browsers - i'm certain about IE6 and IE7, not sure about others). This means that if your page references 20 tiny javascript files, many will get downloaded sequentially.
To add to what was already said: remember that if you combine/minify your javascript files, it's better to merge them all into one - the compression will work better. Additionally, even if you don't minify your files, remember to turn on GZIP in your web server config; then, if you combine all your JS files into one and include that file as , the GZIP compression will work best (because compressing two JS files together produces better results than compressing each separately).
If you're looking for a good JS minification utility, try JS Packer (http://dean.edwards.name/packer/)