Spring Batch: java.io.IOException: Stream closed exception when combining MultiResourceItemWriter and FlatFileItemWriter

Question

I have a Spring Batch process which takes a set of rows in the database and creates a number of flat files from those rows, 10 rows per file. To do this, I've created a Spring Batch process, similar to this:

<batch:job id="springTest" job-repository="jobRepository" restartable="true">
    <batch:step id="test">
        <batch:tasklet>
            <batch:chunk reader="itemReader" writer="multipleItemWriter" commit-interval="2" />
        </batch:tasklet>
    </batch:step>
</batch:job>

<bean id="itemReader" class="org.springframework.batch.item.file.FlatFileItemReader">
    <property name="resource" value="file:/temp/temp-input.txt" />
    <property name="lineMapper">
        <bean class="org.springframework.batch.item.file.mapping.PassThroughLineMapper" />
    </property>
</bean>

<bean id="multipleItemWriter" class="org.springframework.batch.item.file.MultiResourceItemWriter">
    <property name="resource" value="file:/temp/temp-out" />
    <property name="itemCountLimitPerResource" value="2" />
    <property name="delegate">
     <bean id="itemWriter" class="org.springframework.batch.item.file.FlatFileItemWriter">
         <property name="lineAggregator">
           <bean class="org.springframework.batch.item.file.transform.PassThroughLineAggregator" />
         </property>
         <property name="encoding" value="utf-8" />
         <property name="headerCallback" ref="headerFooter" />
         <property name="footerCallback" ref="headerFooter" />
     </bean>
   </property>
</bean>

<bean id="headerFooter" class="uk.co.farwell.spring.HeaderFooterCallback" />

The above example reads from a flat file and outputs to a flat file (to show the problem). Note the commit-interval=2 in the chunk, and the itemCountLimitPerResource=2 in the MultiResourceItemWriter.

The HeaderFooterCallback does the following:

public void writeHeader(Writer writer) throws IOException {
    writer.write("file header\n");
}

public void writeFooter(Writer writer) throws IOException {
    writer.write("file footer\n");
}

I need to be able to specify exactly the number of lines which appear in the file.

For the following input file:

foo1
foo2
foo3

I would expect two files on output,

---

out.1:

file header
foo1
foo2
file footer

out.2:

file header
foo3
file footer

When I run with commit-interval=2, I get an exception:

2009-11-26 15:32:46,734 ERROR .support.TransactionSynchronizationUtils - TransactionSynchronization.afterCompletion threw exception
org.springframework.batch.support.transaction.FlushFailedException: Could not write to output buffer
    at org.springframework.batch.support.transaction.TransactionAwareBufferedWriter$1.afterCompletion(TransactionAwareBufferedWriter.java:71)
    at org.springframework.transaction.support.TransactionSynchronizationUtils.invokeAfterCompletion(TransactionSynchronizationUtils.java:157)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.invokeAfterCompletion(AbstractPlatformTransactionManager.java:974)
    .
    .
    .
Caused by: java.io.IOException: Stream closed
    at sun.nio.cs.StreamEncoder.ensureOpen(Unknown Source)
    at sun.nio.cs.StreamEncoder.write(Unknown Source)
    at sun.nio.cs.StreamEncoder.write(Unknown Source)
    at java.io.Writer.write(Unknown Source)
    at org.springframework.batch.support.transaction.TransactionAwareBufferedWriter$1.afterCompletion(TransactionAwareBufferedWriter.java:67).

I think this is a bug. Wierdly, the files are as follows:

out.1:

file header
foo1
foo2

out.2:

file footer

If I have two lines in the input file, everything works correctly, but more than two does not work. If I change the commit-interval to 200, then I get three lines in one file, which is not the behaviour wanted.

If someone could tell me if I'm doing something wrong, or if not how to get around the problem, I'd be very grateful.

Answer 1

In fact, this is a bug. See http://jira.springframework.org/browse/BATCH-1452.

The workaround, according to Dave Syer, is:

The IOException is nasty. A partial workaround is to use the new transactional property in FlatFileItemWriter, setting it to false (BATCH-1449). But then you lose restartability (so if that's not an issue you are good to go). I'll try and fix it properly for 2.1.

Another workaround is to post process the files in a separate step (and not use the header/footer callbacks).

The counting issue (more than 2 items per file) is really separate - the multi-resource writer was never designed to guarantee the precise number of items per file, only to spill over if the limit is breached. You can open a JIRA for an enhancement if you want, A workaround is to use commit-interval="2" in your example (or more generally a factor of the desired file size).