+2  A: 

It is usually a good idea to set your defenses up in a way that assumes an attacker can list all the files served unless protected by HTTP AUTH (aspx auth isn't strong enough for this purpose).

EDIT: more generally, you are supposed to assume the attacker can identify all publicly accessible persistent resources. If the resource doesn't have an auth check, assume an attacker can read it.

Joshua
I agree. Don't assume that your site is secure based on the fact that the attacker shouldn't know about a specific URL.
Kibbee
I don't think that he's worried about that.
dr. evil
+1  A: 

The "robots.txt" file can give you (if it exists, of course) some information about what files\directories are there (Exmaple).

Aziz
+4  A: 
  • Brute Forcing (Use something like OWASP Dirbuster , ships with a great dictionary - also it will parse responses therefore can map the application quite quickly and then find resources even in quite deeply structured apps)
  • Yahoo, Google and other search engines as you stated
  • Robots.txt
  • sitemap.xml (quite common nowadays, and got lots of stuff in it)
  • Web Stats applications (if any installed in the server and public accessible such as /webstats/ )

Brute forcing for files and directories generally referred as "Forced Browsing", might help you google searches.

dr. evil
I like the OWASP link. I wasn't aware that the project existed.
Gavin Miller
+7  A: 

Most typical attack vector would be trying to find well known application, like for example /webstats/ or /phpMyAdmin/, look for some typical files that unexperienced user might left in production env (eg. phpinfo.php). And most dangerous: text editor backup files. Many text editors leave copy of original file with '~' appended or perpended. So imagine you have whatever.php~ or whatever.apsx~. As these are not executed, attacker might get access to source code.

vartec
+1 - Excellent attack vector!
Gavin Miller
+1  A: 

The path to resource files like CSS, JavaScript, images, video, audio, etc can also reveal directories if they are used in public pages. CSS and JavaScript could contain telling URLs in their code as well.

If you use a CMS, some CMS's put a meta tag into the head of each page that indicates the page was generated by the CMS. If your CMS is insecure, it could be an attack vector.

VirtuosiMedia
+1  A: 
  1. Can you get the whole machine? Use common / well known scanner & exploids.
  2. Try social engineering. You'll wonder about how efficient it is.
  3. Bruteforce sessions (JSessionid etc.) maybe with a fuzzer.
  4. Try common used path signatures (/admin/ /adm/ .... in the domain)
  5. Have a look for data inserts for further processing with XSS / SQL Injection / vulnerability testing
  6. Exploid weak known applications within the domain
  7. Use fishing hacks (XSS/XRF/HTML-META >> IFrame) to forward the user to your fake page (and the domain name stays).
  8. Blackbox reengineering - What programming language is used? Are there bugs in the VM/Interpreter version? Try service fingerprinting. How whould you write a page like the page you want wo attack. What are the security issues the developer of the page may have missed?

a) Try to think like a dumb developer ;)

b) Hope that the developer of the domain is dumb.

Martin K.
+14  A: 

The list on this is pretty long; there are a lot of techniques that can be used to do this; note that some of these are highly illegal:

  • See what Google, archive.org, and other web crawlers have indexed for the site.
  • Crawl through public documents on the site (including PDF, JavaScript, and Word documents) looking for private links.
  • Scan the site from different IP addresses to see if any location-based filtering is being done.
  • Compromise a computer on the site owner's network and scan from there.
  • Attack an exploit in the site's web server software and look at the data directly.
  • Go dumpster diving for auth credentials and log into the website using a password on a post-it (this happens way more often than you might think).
  • Look at common files (like robots.txt) to see if they 'protect' sensitive information.
  • Try common URLs (/secret, /corp, etc.) to see if they give a 302 (unauthorized) or 404 (page not found).
  • Get a low-level job at the company in question and attack from the inside; or, use that as an opportunity to steal credentials from legitimate users via keyboard sniffers, etc.
  • Steal a salesperson's or executive's laptop -- many don't use filesystem encryption.
  • Set up a coffee/hot dog stand offering a free WiFi hotspot near the company, proxy the traffic, and use that to get credentials.
  • Look at the company's public wiki for passwords.

And so on... you're much better off attacking the human side of the security problem than trying to come in over the network, unless you find some obvious exploits right off the bat. Office workers are much less likely to report a vulnerability, and are often incredibly sloppy in their security habits -- passwords get put into wikis and written down on post-it notes stuck to the monitor, road warriors don't encrypt their laptop hard drives, and so on.

Don Werve
I LOL'd at the dumpster diving. I thought that was just good for getting free donuts! :P
Gavin Miller
Hehe, funny ideas! I like the one with the hot-dog-stand!"Hey, we have IT security problems ... mhhh it can't be related to the new hot-dog-hootie in front of our door, they make so delicious hot dogs"
Martin K.
It's good suggestion too... my resume has hot dog stand experience.
Gavin Miller
dumpster diving for list for directories / files is a lot effort, isn't it :)
dr. evil
@LFSR Consulting: I've got the onion-mincing and mustard-sqooging skills. Let's do it! :)
Don Werve
+1  A: 

Are you talking about ethical hacking?

You can download the site with SurfOffline tools, and have a pretty idea of the folders, architecture, etc.

Best Regards!

MRFerocius
The answers will fall under the ethical hacking category, but it's mostly curiosity.
Gavin Miller
A: 

If you use mod_rewrite on your server you could something like that:

All request that does not fit the patterns can be redirected to special page. There the IP or whatever will be tracked. You you have a certain number of "attacks" you can ban this user / ip. The most efficient way you be automatically add a special rewrite condition on you mod_rewrite.

Hippo
A: 

A really good first step is to try a domain transfer against their DNS servers. Many are misconfigured, and will give you the complete list of hosts.

The fierce domain scanner does just that: http://ha.ckers.org/fierce/

It also guesses common host names from a dictionary, as well as, upon finding a live host, checking numerically close IP addresses.

rodarmor
A: 

To protect a site against attacks, call the upper management for a security meeting and tell them to never use the work password anywhere else. Most suits will carelessly use the same password everywhere: Work, home, pr0n sites, gambling, public forums, wikipedia. They are simply unaware of the fact that not all sites care not to look at the users passwords (especially when the sites offer "free" stuff).

Aaron Digulla
I think you missed the point of the question...
Gavin Miller
I don't like telling crackers how to do their job; instead I answer questions like this how to *protect*.
Aaron Digulla
+1  A: 

When attaching a new box onto "teh interwebs", I always run (ze)nmap. (I know the site looks sinister - that's a sign of quality in this context I guess...)

It's pretty much push-button and gives you a detailed explanation of how vulnerable the target (read:"your server") is.

Tom Wright
+1 Nice tool - security stuff like this can really boil down to the tools in the tool box.
Gavin Miller