search-engine-bots

Allow search bots to crawl your sites without session IDs

Google's Webmaster guidelines state Allow search bots to crawl your sites without session IDs or arguments that track their path through the site. These techniques are useful for tracking individual user behavior, but the access pattern of bots is entirely different. Using these techniques may result in incomplete indexing of your s...

allow and disallow in robots.txt file

hi i want to disallow all files and folders on my site from SE bots,except a special folder and files in it. can i use these lines at robots.txt file? User-agent: * Disallow: / Allow: /thatfolder is it right? ...

Google crawler finds robots.txt, but can't download it

Can anyone tell me what's wrong with this robots.txt? http://bizup.cloudapp.net/robots.txt The following is the error I get in Google Webmaster Tools: Sitemap errors and warnings Line Status Details Errors - Network unreachable: robots.txt unreachable We were unable to crawl your Sitemap because we found a robots.txt file at t...

Using Javascript to get around SEO concerns

Hello! I would like to know at which stage is it okay to start manipulating HTML elements/content using Javascript so as not to impair SEO? I have read somewhere that HTML content that is hidden using the CSS property display:none is often penalized by Google crawlers, with good reason from what I'm led to believe...I ask this as I in...