I don't think one actually strictly speaking ever deletes even pointers. What you delete are dynamically allocated objects (or arrays of objects) that the pointer is a handle for. If the object originates from a call to new and it is the responsibility of this class to clean up after this object, then you call delete.
It is technically possible that a reference might be referring to a dynamically allocated object:
int main()
{
//in principle a reference can also refer to a dynamically allocated object
x var(*new someRef);
}
//and if that is the intended usage:
x::~x()
{
delete &m_ref;
}
However, this would be incredibly bad style. By convention, the "owning" handle of a dynamically allocated object should not be a reference.