I'm thinking specifically about the generic class HashSet<T>. It implements several interfaces, but none exposes the correct semantics of a set. Specifically, none supports an Add method returning bool. (ICollection<T> supports void Add, which can be used in a pinch.) Also unsupported by these interfaces are common set operations like unions and intersections. (Though it must be said that some of these operations are available via extensions to IEnumerable<T>.)
This means the class can only be used like a set with its direct implementation. I.e., you can't do something like this:
ISet<int> = new HashSet<int>;
Not as far as I know, anyway. So what motivated the choice to leave this out?
Maybe the most important thing is this: even if you can cast HashSet<T> to ICollection<T> et al, you lose semantic value in the API you're exposing. That is, consumers of your API have no indication they're working with a set. So while you could call ICollection<T>.Add and get by, people will be confused if they try to add an item twice and it doesn't work. A set interface would give people the right expectations.