views:

21

answers:

1

Hi,

I have a website that has a directory that contains 100+ html files. I want crawlers to crawl all the html files that directory. I have already added following sentence to my robots.txt:

Allow /DirName/*.html$

Is there any way to include the files in the directory in sitemap.xml file so that all html files in the directory will get crawled ? Something like this :

< url > < loc >MyWebsiteName/DirName/*.html< /loc > < /url >

Thanks for the help.

A: 

Hi,

The sitemap protocol neither restricts or allows the use of wildcards; to be honest this is the first time i hear this. Also, I'm pretty much sure that search engines can't make use of the wildcards in sitemaps.

Please take a look at Google's recommendation of sitemap generators. There are tons of tools you can create a sitemap with in a blink of an eye.

methode