Mapping a single row result through a linking table with filter in FNH

Question

Excerpts from an existing schema:

User Table:
PK int UserId, 
string UserName, etc

Vendor Table:
PK int VendorId, 
string VendorName, etc

UserVendor Table:
PK int UserId,
PK int VendorId,
PK bit IsPrimary
//note the composite key

There is no FK between User and UserVendor (don't ask, it's a legacy database). I'd like to produce a User class that looks like this:

public class User
{
  public virtual Vendor PrimaryVendor { get; set; }
  public virtual IList<Vendor> AllVendors { get; set; }
  ... // other fields
}

How would you map the User map in Fluent NHibernate? I would completely believe this is impossible to map. If it's not possible with the current schema, how would you change it to suit the desired structure?