The finalizer queue holds all objects that have a finalizer method defined. Recall that a finalizer is a means to collect unmanaged resources like handles. When the garbage collector collects garbage, it moves any objects with a finalizer into the finalizer queue. At some point later-- depending on memory pressure, GC heuristics, and the phase of the moon-- when the garbage collector decides to collect these objects, it walks down the queue and runs the finalizers.
Having worked with memory leaks in the past, seeing a bunch your vendor's objects in the finalizer queue could be sloppy code, but it does not indicate a memory leak. Typically, good code will expose a Dispose method that will collect both managed and unmanaged resources, and in doing so, remove themselves from the finalizer queue via GC.SuppressFinalize(). So, if the vendor's objects do implement a Dispose method, and your code doesn't call it, that could lead to a bunch of objects in the finalizer queue.
Have you tried creating a snapshot in ANTS between two points in time and comparing the objects created between them? That may help you identify any managed objects being leaked.
Also, if you want to see if the memory goes away when the finalizers are run, try this just to test with:
System.GC.Collect();
System.GC.WaitForPendingFinalizers(); // this method may block while it runs the finalizers
System.GC.Collect();
I do not recommend running this code normally. You might want to run it if you've just done a ton of work and created lots of garbage. For example, in our app, one of our functions can create about 350 MB of garbage that goes to waste after closing an MDI window. Since this is known to leave lots of garbage, we manually force garbage collection.
Also note that there is a low-level property cache in the base Windows.Forms code that will hold on to the last opened modal dialog. This could be a source of a memory leak. One sure way to get rid of this reference is to force another simple dialog to appear, then run the above GC code.