We are designing a security system to prevent brute force attacks to get into an account.
One option proposed is blacklisting by IP. If an IP address attempts to login too many times, any further attempts by that IP address are blocked for a given time.
Another option is to do a more traditional account lockout, where too many attempts on a given account locks out the account until the password is reset.
The issue with the first approach is customer service - if a legitimate user calls to get back in, they just have to wait it out - their IP is blacklisted for the time period.
The issue with the second is that it opens a DoS attack, given knowledge of a legitimate user name, anyone can put in bogus passwords to lock them out.
What experiences have you had in different approaches to preventing brute force attacks against user accounts?