For the first time, yes. After that, as long as the file has not changed, the browser should cache it, so the load time would not be very high.
Anyway your site (with all its images and javascript files) should load, and this will also take some time the first time. The overhead (on top of this) caused by an external hosted script should not be very high.
Whether you want to use a local version depends on your requirement as well as the purpose of the js file.
jquery.js
can be downloaded and used from your server, but, for instance, it does not make much sense to have google analytics scrips locally.
Note: You might see the delay during development because all other files load from the same (local) network, and google hosted scripts load from the internet. (I mention this because someone in my previous project panicked that "google is slow" during a similar scenario. We were using google anayltics scripts)