Having moved to the site, the robot starts walking down the pages and downloads the information, they have (on the site it can be presented as a text, image, video and other files). This process is called “crawling” Robot also needs to establish, when it will visit the site again, so it creates the schedule with the help of which it will define the time of the next visit in order to check the already existing information about the site and also to add new pages to the Database, if they exist.
It is important to “meet” the search robot in the effective manner, to simplify the process of the site indexation as much as possible because its time is limited and it has a great amount of sites, except yours, in its “visit schedule”. That’s why we have to create the appropriate conditions. We can do it in this way:
- At first it is necessary to be sure in the site availability, it can be checked by the typing the domain name of the site in the address line of the browser.
- Be sure that there are no problems with the site navigation, try not to use Java Script and Flash, the menu, designed with the help of these technologies is not recognized by the robots effectively.
- Test your site for the absence of the errors 404.
- Also don’t send the robot on the pages, accessible only for the registered users.
- Take care of the levels of the nestings of your pages, because there are some limitations as for the deep site penetration.
- Take into account that the maximal size of the text, represented on the page consists of 256 kb, try not to exceed this limit.
In general, it is difficult to answer your question directly. Try to read http://phpforms.net/tutorial/tutorial.html To understand Google and other SS