Many to many as map in fluent Nhibernate

Question

Is there a way to map a many-to-many relationship using Fluent NHibernate to a dictionary?

Say I have the following (totally just made up) tables:

Person ( Id INT, Name NVARCHAR(MAX) )

Address ( Id INT, StreetAddress NVARCHAR(MAX) )

PersonAddresses ( PersonId INT,
AddressId INT, AddressType NVARCHAR(MAX) )

public class Person
{
  public virtual int Id { get; set; }
  public virtual string Name { get; set; }
  public virtual IDictionary<string, Address> Addresses { get; set; }
}

I want to map the addresses of a person to a dictionary property on the person entity that has the addresses as values and the AddressType from the PersonAddresses-table as the key. Is this at all possible using fluent NHibernate, if not, is it possible using the xml configuration? The problem I'm facing is that the key-column I want to map is in the join-table rather than in the referenced table.