Web crawling and its limitations

Question

Let's say that we place a file on the web that is publicly assessable if you know the direct URL. There are no links pointing to the file and directory listings have been disabled on the server as well. So while it is publicly accessible, there is no way to reach the page except for typing in the exact URL to this file. What are the chances that a web crawler of any sort (nice or malicious) is able to locate this file by crawling and then indexing the file.

To me, even though it is publicly accessible, its going to require luck or specific knowledge of finding the file. Much like burying gold in by back yard and having someone find it without a map or knowing something is buried there.

I just can't see any other way it would be discovered, but that's why I'm asking the stackoverflow community.

Thanks.

Answer 1

Links can occur anywhere - someone could Twitter a link to it, or post it on Facebook, or in a comment on a blog. It only takes one.

If it's vital that it not show up anywhere, put it behind a password.

If it's not vital but you'd still prefer it not be easily accessible via search engine, use a robots.txt file to block well behaved crawlers.

Answer 2

In the past, such hidden locations have been allegedly "found" using the Google Toolbar (and probably other such browser plugins), used by the owner/uploader.

Answer 3

you can use google search api. for the webpage unlinked with any other webpage. we have not idea about that.

Answer 4

Assuming this:

• Directory listing: disabled. No one
• knows the existence of the page.
• Your file doesn't contain any links (your browser could then send the referer to the linked site)
• You have set up the robots.txt properly
• You trust all the people won't spread your link to anyone else.
• You are lucky

Well, your page won't probably be found or discovered.

Conclusion ?

Use an .htaccess file to protect your data.

Answer 5

Security through obscurity never works. You say, you're not going to link to it, and I believe you. But nothing stops your user from linking to it, intentionally or uninteionally. As ceejayoz indicated, there are so many different places to post links now. And there are even "bookmark synchronizers" that people may think are private but are actually open to the world.

So use real authentication. If you don't you'll regret it later.

Answer 6

You are correct. Web crawlers are, metaphorically, spiders - they need to have a way to traverse the web (hyperlinks) and arrive at your page.

To get your hypothetical page into a search engine's results, you must manually submit its URL to the search engine. There are multiple services for submitting your page to these search engines. See "submitting URLs to search engines"

Also, your page will only appear if the search engine determines that your page has enough metadata/karma within the search engine's proprietary ranking system. See "SEO" and "meta keywords".

Answer 7

yes u r right Web Crawler visits at URLs, it identifies all the hyperlinks in the page and adds them to the list of URLs to visit and called the crawl frontier but these hyperlinks and URLs have bad links. Once users click a bad link and land on the Malware site, they’re often promoted with a fake codec installation dialog. If that doesn’t get them, the site is still loaded will dozens of other tactics to infect their computer. From fake toolbars, scare ware, rogue software, and more, the sites have it all. One site that they came across even tried to install 25 different bits of Malware. Such sites are leaving people vulnerable to installations of spam bots, rootkits, password Steelers, and an assortment of Trojan horses, amongst other things.

Answer 8

Purchased/sold clickstream data may result in otherwise unlinked content discovery: http://en.wikipedia.org/wiki/Clickstream