What would be the shortest method to block * and only allow just Major Search Engines to index the index page of the site only?
User-agent: *
Disallow: /
User-agent: Googlebot
Disallow: /
Allow: index.html
User-agent: Slurp
Disallow: /
Allow: index.html
User-agent: msn
Disallow: /
Allow: index.html
Would this work?
Yes, that would be the shortest way to do it. It's not necessarily correct.
Not all bots support the Allow directive. And some bots are confused about how to interpret the robots.txt when there is both a User-agent: * section and a User-agent: Specific-bot section that apply.
To be sure it would work, you'd want to do something like this:
User-agent: Googlebot Disallow: /file1 Disallow: /file2 Disallow: /file3 # etc. until you have blocked every path except index.html User-agent: Slurp Disallow: /file1 Disallow: /file2 Disallow: /file3 # etc. until you have blocked every path except index.html User-agent: msn Disallow: /file1 Disallow: /file2 Disallow: /file3 # etc. until you have blocked every path except index.html User-agent: * Disallow: /
If you don't want to do all that work, then the best thing to do would be to test each of the engine's you're interested in and see if they'll accept the robots.txt file you proposed. If they don't, try the longer version.
Thank you very much - I will give the long shot a try, I really do not have that much to close off so the list wouldn't be long.