Try RedBlackTree.NET. It's in VB but I think it can be easily converted to C#.
And I believe some of the collection type actually uses a red-black tree internally. So you might want to decompile the framework itself and look around for some clues.
I don't think a binary tree can be replaced by a HashSet. Their performance characteristics are different, roughly:
HashSet - O(1) lookup (n) search
Binary search tree - O(log n) lookup O(log n) search
If you want to store the values and later perform a search, you will want to be using a binary tree instead of a HashSet.