How we can optimize my web site for deep web searching? (Invisible web)

Question

From Wikipedia:

The deep Web (also called Deepnet, the invisible Web, dark Web or the hidden Web) refers to World Wide Web content that is not part of the surface Web, which is indexed by standard search engines.

Mike Bergman, credited with coining the phrase,has said that searching on the Internet today can be compared to dragging a net across the surface of the ocean; a great deal may be caught in the net, but there is a wealth of information that is deep and therefore missed. Most of the Web's information is buried far down on dynamically generated sites, and standard search engines do not find it. Traditional search engines cannot "see" or retrieve content in the deep Web – those pages do not exist until they are created dynamically as the result of a specific search. The deep Web is several orders of magnitude larger than the surface Web.

Now how can we (as a programmer) optimize my web sites for deep web searching?

Answer 1

1. Provide site map pages and site map XML. For dynamically generated pages, you might need to generate site maps.
2. Ensure that dynamic pages have not more than three query string parameters. Try to use the query string, not the form data for passing dynamic page parameters.
3. Provide content specifically for search engines. This may require simplification of the dynamic page generation logic so that a surrogate static page is seen by a bot.
4. For pages that require authentication, provide stub content specifically for search engines. If you have a serious project, it could even make sense to provide an alternative data layer available to search engines only.

In general, you should design your site according to Google's Webmaster Guidelines.