+1  A: 

You could use another test to see if the position() is within 3 of the size of the nodeset using last()-3:

This answer does not fully address the stated requirements. @Alejandro's answer should be the accepted answer.

Mads Hansen
@Mads Hansen: This produce an @class="row-last" (I think you have a type mistake) for every last three. Also, @class="col-last" would overwrite every previous @class.
Alejandro
@Alejandro - you are right. I obviously didn't test my solution (and hadn't had my coffee).
Mads Hansen
I cannot delete my answer, since it has been accepted. @Marko Ivanovski, would you de-select my answer?
Mads Hansen
Thanks to both of you - @Mads - you led me to fix my issue by slightly modifying what you provided, but Alejandro had posted the correct one in the meanwhile. Cheers!
Marko
+2  A: 

This stylesheet:

<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"&gt;
    <xsl:output method="xml" omit-xml-declaration="yes"/>
    <xsl:template match="@*|node()">
        <xsl:copy>
            <xsl:apply-templates select="@*|node()"/>
        </xsl:copy>
    </xsl:template>
    <xsl:template match="li">
        <xsl:copy>
            <xsl:apply-templates select="@*"/>
            <xsl:variable name="class">
                <xsl:if test="position() > last() - last() mod 3 - 3 * not(last() mod 3)">row-last </xsl:if>
                <xsl:if test="not(position() mod 3)">col-last </xsl:if>
            </xsl:variable>
            <xsl:if test="$class != ''">
                <xsl:attribute name="class">
                    <xsl:value-of select="normalize-space($class)"/>
                </xsl:attribute>
            </xsl:if>
            <xsl:apply-templates select="node()"/>
        </xsl:copy>
    </xsl:template>
</xsl:stylesheet>

With these inputs:

<ul>
<li>Item 1</li>
<li>Item 2</li>
<li>Item 3</li>
<li>Item 4</li>
<li>Item 5</li>
<li>Item 6</li>
</ul>

<ul>
<li>Item 1</li>
<li>Item 2</li>
<li>Item 3</li>
<li>Item 4</li>
<li>Item 5</li>
</ul>

<ul>
<li>Item 1</li>
<li>Item 2</li>
<li>Item 3</li>
<li>Item 4</li>
</ul>

Results:

<ul>
    <li>Item 1</li>
    <li>Item 2</li>
    <li class="col-last">Item 3</li>
    <li class="row-last">Item 4</li>
    <li class="row-last">Item 5</li>
    <li class="row-last col-last">Item 6</li>
</ul>

<ul>
    <li>Item 1</li>
    <li>Item 2</li>
    <li class="col-last">Item 3</li>
    <li class="row-last">Item 4</li>
    <li class="row-last">Item 5</li>
</ul>

<ul>
    <li>Item 1</li>
    <li>Item 2</li>
    <li class="col-last">Item 3</li>
    <li class="row-last">Item 4</li>
</ul>

Note: When ouput attributes take notice that last attribute with same name overwrites previous.

Alejandro
+1 for the correct answer that addresses the stated requirements.
Mads Hansen
@Mads Hansen: +1 for "civility"! I hope you'd had your coffee, ja!
Alejandro