I think it partially depends on the kind of site(s) you're asking them to design. A brand-oriented, marketing site needs a different approach from a app, which might be different again from an portal. Are you looking for someone who's going to create something visually stunning, or is usability most important? (Not that those are necessarily incompatible, but they tend to emphasize different skills.)
What skills are you looking for? usability testing? interaction design? wireframing? content structuring? visual metaphor creation? photoshop? general visual creativity? illustration? etc.? Once you know that, I think you could probably devise a task or sequence of tasks to help probe their skills in those areas.
If you give them something specific to work on, I would make the expectations very clear about what kind and quality of deliverables you want. And I'd lean toward making them as low-fi as possible -- their thought process is probably more important than having them waste time creating a finished product. You can get their sense of polish from their portfolio (hopefully).