Hi,
I have a website that has a directory that contains 100+ html files. I want crawlers to crawl all the html files that directory. I have already added following sentence to my robots.txt:
Allow /DirName/*.html$
Is there any way to include the files in the directory in sitemap.xml file so that all html files in the directory will get crawled ? Something like this :
< url > < loc >MyWebsiteName/DirName/*.html< /loc > < /url >
Thanks for the help.