One-to-Zero-Or-One mapping - eager load results in multiple lazy load when zero

Question

Iam trying to do some Eager loading with some entities that have a reference to one or zero elements. The problem arises when it refers to zero element.

I have the following

class Car {
 int id
 string name
 Driver driver
}

class Driver {
 int id
 string Name
}

The Driver gets loaded using References(x => x.Driver).ColumnName("driver_id").FetchType.Join().NotFound.Ignore();

This results in the correct SQL where it left joins the driver database when getting the Car.

However when this returns no object and i Afterwards try to access the Driver i get yet another Select which ofcause returns nothing.

How do make sure that it does not try to get the data on the second try ?