The only time you should write a finalizer is when you are building a type to handle a new kind of unmanaged resource. For example, a data access layer that uses Sql Server in a business app doesn't need a finalizer anywhere, even though there are unmanaged database connections involved, because the basic SqlConnection class will already finalize those connections if needed. But if you're building a brand new database engine from scratch that has connection limits similar to sql server's and are implementing the ado.net provider for it, that connection type should implement a finalizer to be as sure as possible that your connections are released.
But you don't get any guarantees beyond what happens when a process ends.
Update:
Given this context:
I am having a discussion with a collegue over a code review I did of his code. He insists that the destructor is guranteed to be called on an object. I disagree (but am not sure) and would prefer the use of IDisposable.
You are right to criticize the use of a destructor/finalizer. As I said above, you should only use them when working with an unmanaged resource that is genuinely new. Not just that instance of the resource, but the kind of resource you are working with.
For code that wraps "normal" unmanaged resources (things like SqlConnection, TcpClient, etc), IDisposable is a better choice. Then you know the resource will be cleaned up as soon as Dispose() is called rather than needing to wait for the type to be collected. If no one calls Dispose() (which is likely your colleague's concern), by the time your new type can be collected the instance of the original type for the unmanaged resource you are wrapping should be able to be collected as well, and it's finalizer will release the resource.
The main thing you need to bring to the table is that the finalizer cannot be called until the object is collected. You have to wait on the garbage collector, meaning you may be holding the resource open even longer. IDisposable allows you to release it right away. Of course you could do both, but that doesn't sound like what's going on here, and if you do have both you have to be careful not to conflict with the original type's finalizer or you could cause unwanted and harmful exceptions. And really, your own finalizer implementation is just redundant here and adds needless complexity.
Finally, I have to take issue with this statement:
If there is still a reference to the memory it will not be garbage collected
There can be references to an object and it will still be collected. What matters is if the object is reachable: are any of the references to the object rooted. For example, you may have a list with several objects in it. The list goes out of scope. Obviously there is still a reference to all of the objects in the list, but they can still all be collected in the first pass of the GC because the reference is no longer rooted.