I am not aware of a way of doing what you ask through the mapping files. I do not think there is one.
How I would approach the problem is to plug in a PreInsertListener and do the select statement you provide in your question there to retrieve the value for the SortOrder answer just before the entity is saved.
Here is how it would roughly look:
public class NHibernateEventListeners : IPreInsertEventListener
{
public bool OnPreInsert(PreInsertEvent auditEvent)
{
var audit = auditEvent.Entity as YourEntityTypeHere;
if (audit == null)
return false;
int sortOrderValue =
(int)auditEvent.Session.CreateCriteria<YourEntityTypeHere>()
.SetProjection(Projections.Max("SortOrder"))
.Add(Restrictions.Eq("CategoryID", CatID)).UniqueResult();
SetValue(auditEvent.Persister, auditEvent.State, "SortOrder", sortOrderValue + 1);
audit.DateCreated = sortOrderValue + 1;
return false;
}
}