As my comment says:
The REFERER is spoofable, so it is probably unlikely that Google will use it as a means of verification. See this wikipedia entry.
My guess is that Google probably uses the IP address of the caller along with a DNS lookup. DNS is not really spoofable, as your DNS entries have to be correct for the website to even get to you.
But, even that has its problems, because if a server uses a Round-Robin IP Address DNS setup, Google will be redirected to a different IP address when doing a DNS lookup.
From the FAQ
Note that a key for http://www.mygooglemapssite.com/ will only be accepted when the site is accessed using this address. It will not be accepted if the site is accessed by IP address (eg. http://10.1.2.3/) or by a hostname that is aliased to www.mygooglemapssite.com using a DNS CNAME record.
My guess is that it might be using the Host
header that is sent when requesting the page, which would work as normally Google asks you to include it's API script directly into the page. Then that script has access to the headers for the current page and can use that to check.
My guess is backed up with the fact that it does not work for IP addresses or Aliases, which means it isn't doing a DNS check.
THIS method cannot be spoofed, as it must be the correct header to access the page. However, this means that any aliases to the domain will not work.
However, this also means that you MUST provide a Javascript library to access the code, as you can't check this server side, I believe.