You might try to increment your variable (I named it totalSum) only when the band (group) is equal to the one on which the subreport is. For this you would need a field in the report to give you the current band (group).
<variable name="totalSum"
class="java.lang.Integer"
resetType="Report"
incrementType="Group"
incrementGroup="ITEM_BUNDLE"
calculation="Nothing">
<variableExpression>
<![CDATA[new Boolean($F{reportPart}.equals("The_band_with_the_subreport")).booleanValue() ? $V{returnValue} : $V{totalSum}]]>
</variableExpression>
<initialValueExpression>
<![CDATA[new Integer(0)]]>
</initialValueExpression>
</variable>
I'm not sure if this works, I don't have the context to test it. But you might also try a second solution - with three variables. For example, you keep the value returned from the subreport (let's say returnValue) in a variable and you use another two variables to hold the sum - one until the subreport is called (let's say partialSum) and the second to store the sum between the returnValue and the partialSum. Let's call it totalSum. Then you would have something like this for totalSum:
<variable name="totalSum"
class="java.lang.Integer"
resetType="Report"
incrementType="Group"
incrementGroup="ITEM_BUNDLE"
calculation="Nothing">
<variableExpression>
<![CDATA[$V{returnValue} + $V{partialSum}]]>
</variableExpression>
<initialValueExpression>
<![CDATA[new Integer(0)]]>
</initialValueExpression>
</variable>
For partialSum, you'll have something like this:
<variable name="partialSum"
class="java.lang.Integer"
resetType="Report"
calculation="Sum"
incrementType="None">
<variableExpression>
<![CDATA[new Boolean($F{reportPart}.equals("The_band_with_the_subreport")).booleanValue() ? $V{returnValue} : new Integer(0)]]>
</variableExpression>
<initialValueExpression>
<![CDATA[new Integer(0)]]>
</initialValueExpression>
</variable>
I hope this helps a bit. It would be easier to make all these settings from iRport directly on the report you want to use.