Let's say I am testing beta pages on my server and was too lazy to password protect the pages I put up. I am putting them up in some obscure named sub-directory. Is there any way of people seeing the beta pages without guessing around at file names or looking at the directory from an FTP client?
These are all orphaned pages, there is no way of getting to them from the homepage.
As long as these pages have never been linked to at some point, and detected by a search bot, there is no way to list them.
You can test with wfuzz to try your "obscure" paths but I don't recommend this technique. At least protect them with a basic auth.
Also be aware of using Google (or other search engine) toolbar: GoogleBot visits what you visit if you have the toolbar
It's an old news but who knows... :)