I am definitely not a security expert, but here are some thoughts.
This sounds somewhat similar to public key encryption. If you look into how that works, I think you will get a sense of the same sort of issues. For example, see http://en.wikipedia.org/wiki/Public-key_encryption
In addition to the challenge of users having to carry the file around with them, another issue is how to keep that file secure. What if somebody's computer or thumb drive is stolen? A common approach with public-key encryption is to encrypt the private key itself, and require a password to use it. Unless you provide the file in a form which requires this, you are counting on your users to protect the file. Even if you are willing to count on them, there is the question of how to give them the tools they need so they can protect the file.
Note that just like passwords, these files would be vulnerable if a user used one to login from a public machine (which might have all sorts of spyware on it). It's an open question whether a file-based system might slip under the spyware since they might not be looking for it. However, that is not so different from security by obscurity.
Also you would want to make sure that you hashed or encrypted the files on your system. Otherwise, you would be doing the equivalent of storing passwords in plain text which would open the possibility of someone hacking your system, and then being able to log in as any user.