First of all, although you think you don't know good design, if you see enough portfolios, the good ones will probably still stand out to you. If you still don't trust your own judgement, get other people to look over them for you.
Get them to talk you through their work. A good designer should be a good advocate for their work. You can also use the opportunity to assess their technical knowledge. They don't need to know how to program, but they must understand how their work will interface with underlying technologies.
As others have already said here, another essential is to be able to appraise their work from a usability perspective, not merely an aesthetic one.
Also, I wouldn't take on a designer who doesn't understand at least the rudiments of xhtml/css, unless you're willing to teach them.
Finally - an alternative to setting tests is to take someone on a freelance basis to begin with, and find out for real how well they can work with you.