In theory you could build just one library if all the external headers for the library (ie the ones pulled in by the client) don't use #ifdef _DEBUG (or any other macro that may be defined in a debug build but not a release build.
Consider a case like this:
// file: mylib.h
struct A {
int member1;
int member2;
#ifdef _DEBUG
int extraDebugOption;
#endif
};
In this case, if you were to link the library into a debug build of your own product then A will have a different size to the release build, which means you're going to have some pretty horrendous memory corruption bugs to track down (been there...).
EDIT: Forgot to mention, you should make your one configuration a Release configuration so that you don't end up with any references to the debug CRT, and also so the library is optimised. As lsalamon points out, creating a pdb file and saving this along with your .lib file will be useful in the future for debugging.