There are several possible solutions, depending on what you want to do. The most well-known is the Muenchen method. This method first defines a lookup table to find all elements that share a particular key by that key. Then, you iterate over all keys (in some order) and for each key, over all key-values (in some order).
<xsl:key>
can be used to make lookup tables. If you index of the first letter, you can then retrieve those elements which correspond to a particular letter.
<xsl:key name="child-by-first-letter" match="child" use="substring(.,1,1)"/>
Now, we can't iterate over all keys in a straightforward manner. However, we can iterate over all values and test whether they're the first value in the lookup table...
<xsl:for-each select="child[ generate-id()
= generate-id(key('child-by-first-letter', substring(.,1,1))[1])]"/>
<xsl:sort select="substring(.,1,1)" /> <!--sort by first letter-->
<xsl:value-of select="substring(.,1,1)">
[...]
The above code will execute something only on child tags where the generate-id function returns the same value as the generate-id function does for the first element of the list of children with the same first letter. Since generate-id generates unique id's, this will only return only those children that are "first" in the lookup table for their particular key.
Now, you're almost done: just iterate over all children with that particular first letter:
<ul>
<xsl:for-each select="key('child-by-first-letter', substring(.,1,1))">
<!--no sort, we'll do document order-->
<li><xsl:value-of select="."/></li>
</xsl:for-each>
</ul>
</xsl:for-each>
This is called the Muenchian method.