Yes it matters, changes made only to the inverse end of the association are not persisted
You may check nhibernate documentation for further details. Here you have the link:
http://www.nhforge.org/doc/nh/en/#collections-bidirectional
EDIT
My answer doesn't change with your addition, but I'll try to explain it better :-)
if you set Parent.Children with inverse = true
you need to save Child object in order to save the relation. If you ONLY save Parent then the relation won't be saved
if you set Child.Parent with inverse = true
you need to save Parent object in order to save the relation. If you ONLY save Child then the relation won't be saved
category.Items.Add(item); // The category now "knows" about the relationship
item.Categories.Add(category); // The item now "knows" about the relationship
session.Update(item); // No effect, nothing will be saved!
session.Update(category); // The relationship will be saved