Cleaning up static references

Question

Hi, I have static class with static dictionary with some informations. The dictionary has key as WeakReference (i don't want prevent garbage collection of the real key object ).

Code sample:

public static class ThreadHelper {
    private static readonly object syncRoot = new object();
    private static Dictionary<WeakReference, ThreadInfo> threadInfos = new Dictionary<WeakReference, ThreadInfo>();

    // some static thread safe methods for adding, removing and changes items
}

What is good (or best) way to clean the items in "threadInfos" dictionary which are in fact empty, and has no usage in future?

Should I check for "empty" keys in every Add/Remove/Change method or should I do it another way or should I do the concept completly differently?

Thanks

EDIT 2: It looks the answer is more complex than I thought. I provide full source of ThreadHelper class at ThreadHelper class and the simple test source code at Simple test.

EDIT: The method which finds the key looks:

private static WeakReference FindThreadReference( Thread thread ) {
    WeakReference ret = null;

    WeakReference[] keys;
    lock ( syncRoot ) {
        keys = threadInfos.Keys.ToArray();
    }

    foreach ( var key in keys ) {
        if ( key.IsAlive ) {
            var tmpThread = key.Target;
            if ( object.ReferenceEquals( tmpThread, thread ) ) {
                ret = key;
            }
        }
    }

    return ret;
}

I still don't understand why is using WeakReference as key bad idea (but it is not important for now), should I wrapp the WeakReference with some class of struct and use the wrapper as key?