Possible to download entire whois database / list of registered domains?

Question

I wanted to do some analysis on registered domain names. Looks like I can hit whois.internic.net to get information about each domain, but it also looks like there are rate limits that prevent me from doing large numbers of queries.

Is there a way to periodically (say daily) grab the entire whois database? I really only care about whether a domain is registered or not, so I don't need the full whois information.

Answer 1

Back in 2006 Dennis Forbes did some analysis you can see more about it here. He also mentions in the sidebar where he got the data. Seems he was looking at Verisign zone files rather than the whois database. I don't the details but this might be another direction to look in.

Answer 2

For .com and .net, Verisign provides the TLD Zone Access Program where you can apply for download access to the zone files. Other registrars usually provide similar services.

Here is a list for some other TLDs (though I haven't tried to apply for any of these):

• .org: http://www.pir.org/index.php?db=content/Website&tbl=Registrars&id=7
• .biz: https://www.neulevel.biz/zonefile/
• .info: http://www.afilias.info/faqs/for_registrars/general_registrar
• .name: http://gnr.name/zone-file-access.html
• .coop: http://www.nic.coop/faqs.asp?Cat=22#170
• .pro: http://www.registrypro.pro
• .mobi: http://mtld.mobi/domain/zonefile

Answer 3

I am interested in identifying any new domains. Is there any page/company which provides this?

Answer 4

These guys will sell you monthly access to the major zone files (plus lists of changes):

http://www.premiumdrops.com/zones.html