One option is to take a look a Bea Stollniz's series on improving a TreeView's performance:
Part 1, Part 2, and Part 3. While what she does is more geared to TreeViews, which don't have any virtualization because they group by default, the lessons learned could definitely be applied to a custom ListView that has virtualizing groups. In fact, in part 3, she uses a ListBox as her base to create the virtualizing tree, which is a good start for virtualized grouping as well. Obviously displaying the items like in a TreeView has some diffrences, such as selection of the group nodes, from a ListView with grouping, but that could be fixed by catching the SelectionChanged.