Optimizing/Customizing Sharepoint Search Crawling

Question

With SharePoint Server 2007, there is also a Search Feature and a Crawler. However, the Crawler is somewhat limited in that it only supports Basic Auth when crawling external sites and that there is no way to tell it to ignore noindex,nofollow attributes.

Now, there is a site i'd like to index, unfortunately this site uses it's own Authentication System, and it uses noindex,nofollow on the pages. As I control that site, i can remove the Attributes, but it's a PITA to do so. Also, it does not solve the Authentication issue.

So I just wonder if it's possible to extend Sharepoint's Crawler somehow?

Answer 1

Hi Michael,

The limitation of MOSS crawling sites with different forms authentication should have been addressed in MOSS SP1. :

http://www.microsoft.com/downloads/details.aspx?FamilyID=ad59175c-ad6a-4027-8c2f-db25322f791b&displaylang=en

Here's a link to a post which describes how to get the hotfix for pre-SP1 MOSS to enable the crawling of sites with forms authentication:

http://blogs.microsoft.co.il/blogs/adir_ron/archive/2007/10/11/moss-search-for-sso-form-based-authentication-sites.aspx

Hope that helps!