Cannot crawl complex URL's without setting a site-wide rule to 'crawl as http content'.

Question

I have pages within a site containing a control that uses a query string to provide dynamic data to the user (http://site/pages/example.aspx?id=1).

I can get my content source to index these dynamic pages only if I create a rule which sets the root site (http://site/*) to 'include complex urls' and 'crawl sharepoint content as http content'. This is NOT acceptable as changing the crawling protocol from SharePoint's to HTTP will prevent any metadata from being collected on the indexed items. The managed metadata feature is a critical component to our SharePoint applications.

To dispel any wondering of whether or not this is simply a configuration error on my part refer to http://social.technet.microsoft.com/Forums/en-US/sharepointsearch/thread/4ff26b26-84ab-4f5f-a14a-48ab7ec121d5 . The issue mentioned is my exact problem but the solution is unusable as I mentioned before.

Keep in mind this is for an external publishing site and my search scope is being trimmed using content classes to only include documents/pages (STS_List_850 and STS_ListItem_DocumentLibrary). Creating a new web site content source and adding it to my scope presents 2 problems: duplicate content in scope and no content class defining it that I know of.

What options do I have?

Answer 1

Just a thought: maybe you should create two data sources, one - SharePoint - for metadata and items and one - HTTP - for the pages. Set rules on each one to exclude the other's content. Would that solve your problem?

Answer 2

I have decided to take a different approach to this problem as combining dynamic http content and sharepoint content into one scope is a non trivial problem and is better suited to a entirely new project and not a retrofit as I was attempting.

If you have dynamic content from a separate system which you want to crawl without sacrificing SharePoint metadata information from the rest of your site it seems the only option is to write a BCS application/search connector, crawl the two content sources separately and combine them with a scope and possibly an extended core results webpart. Good luck!