I don't believe there is any API which will give you the true size of an object, but it can be determined by measuring the ammount of memory used before and after the object is created; however, that is not really thread safe.
You could use System.Runtime.InteropServices.Marshal.SizeOf() even though this will not give you an accurate size it will give you an accurate comparison point so that you can say something like: if the size of the object increases by X% then remove Y number of elements.
Also, you could try to keep a running total but if items in the collection are editable you will need to make sure that every addition, removal, and edit is included in the calculation. You are still left with the problem of properly measuring the items being added and removed and as stated above I don't believe there is an API that will do this accurately.