So you want to group by node name. Not the best idea to store more than actual structure information in the XML nodes, but here you go:
<siteMenu>
<Pages>
<title>page1</title>
</Pages>
<Pages>
<title>page2</title>
</Pages>
<Links>
<title>link1</title>
</Links>
<Links>
<title>link2</title>
</Links>
<Sites>
<title>site1</title>
</Sites>
<Sites>
<title>site2</title>
</Sites>
</siteMenu>
and this XSLT 2.0 transformation (relevant fragment only):
<xsl:template match="siteMenu">
<xsl:for-each-group select="*" group-by="name()">
<xsl:sort select="name()" />
<div>
<h1><xsl:value-of-select="current-grouping-key()" /></h1>
<ul>
<xsl:for-each select="current-group()">
<li><xsl:value-of select="title" /></li>
</xsl:for-each>
</ul>
</div>
</xsl:for-each-group>
</xsl:template>
or this XSLT 1.0 transformation (relevant fragment only):
<xsl:key name="kMenu" match="siteMenu/*" use="name()" />
<xsl:template match="siteMenu">
<xsl:for-each select="*[
generate-id() = generate-id(key('kMenu', name())[1])
]">
<xsl:sort select="name()" />
<div>
<h1><xsl:value-of-select="name()" /></h1>
<ul>
<xsl:for-each select="key('kMenu', name())">
<li><xsl:value-of select="title" /></li>
</xsl:for-each>
</ul>
</div>
</xsl:for-each-group>
</xsl:template>
Hint: You might want to think about using such an XML instead, this obviously matches your site structure better and you need no grouping to output it correctly.
<siteMenu>
<section name="Pages">
<title>page1</title>
<title>page2</title>
</section>
<section name="Links">
<title>link1</title>
<title>link2</title>
</section>
<section name="Sites">
<title>site1</title>
<title>site2</title>
</section>
</siteMenu>