A: 

I'm not an expert in Fluent NH, but it seems to me that your person map should look like this:

class PersonMap : ClassMap<Person>
{
    public PersonMap()
    {  
        Id(c => c.ID).GeneratedBy.Identity();

        //this works fine
        HasMany(c => c.Pets);

        //this dosen't work, because the result contains dogs and cats
        //how can I tell NHibernate to only fetch dogs or cats?
        HasMany<Cat>(c => c.Cats);
        HasMany<Dog>(c => c.Dogs);
    }
}

Because you're Person has a Dogs property which is an IList not IList, and conversely the same for Cats.

Joseph
<code> HasMany<Pet>(c => c.Cats)</code>This look a litle bit stupid, but i think it is correct, because the Pet table has a reference to the Person table. I get this solution from an other stack overflow question.It seems to me, that HasMany<Pet>(c => c.Cat) does the same as HasMany<Cat>(c => c.Cat).KeyColumn("PetID"). In both cases i get a result of cats and dogs.
trichterwolke
@trichterwolke if you're saying that the Cat and Dog classes are actually coming from the Pet table in your DB, then I would think that you would say that in the CatMap and DogMap, not in the PersonMap. I believe that's how I did that in the past when using FNH.
Joseph