NHibernate one-to-many delete

Question

I am working on an ASP.NET MVC application using NHibernate as the ORM. I have two domain objects:

    public class Contact   
    {  
        public int ID { get; set }  
        public string Name { get;set; }  
        public IList<ContactNumber> ContactNumbers { get; set; }  
    }

    public class ContactNumber  
    {  
        public int ID{get;set}  
        public string Number { get; set; }  
        public int Extension { get; set; }  
    }

Contact Mapping:

HasMany(x => x.ContactNumbers)
                .Inverse()                
                .Cascade.AllDeleteOrphan();

ContactNumber Mapping:

References(x => x.Contact)

I have a ContactViewModel which similarly contains a list of ContactNumberViewModel which maps back to the domain objects using AutoMapper. I have a question about updating the domain.

Let's say contact contains two ContactNumber objects with ids 23 and 24. I assign contact.ContactNumbers to a new list containing ContactNumber 23, 24, and a new ContactNumber (id is 0 because it is new). When I save the contact object, a new ContactNumber is created in the database and all is good.

Now say I assign the contact.ContactNumbers to a new list which contains one ContactNumber, 23 (ContactNumber 24 is deleted). If I save the contact object, the ContactNumber 24 is not deleted from the database.

I know that explicitly removing the ContactNumber from contact.ContactNumbers would work, but is there a way I can achieve this through direct assignment like in my example?