I've been trying to find information on this, but due to the immaturity of the Spring Integration framework I haven't had much luck.
Here is my desired work flow:
New files are placed in an 'Incoming' directory
Files are picked up using a file:inbound-channel-adapter
The file content is streamed, N lines at a time, to a 'Stage 1' channel, which parses the line into an intermediary (shared) representation.
This parsed line is routed to multiple 'Stage 2' channels.
Each 'Stage 2' channel does its own processing on the N available lines to convert them to a final representation. This channel must have a queue which ensures no Stage 2 channel is overwhelmed in the event that one channel processes significantly slower than the others.
The final representation of the N lines is written to a file. There will be as many output files as there were routing destinations in step 4.
*'N' above stands for any reasonable number of lines to read at a time, from [1, whatever I can fit into memory reasonably], but is guaranteed to always be less than the number of lines in the full file.
How can I accomplish streaming (steps 3, 4, 5) in Spring Integration? It's fairly easy to do without streaming the files, but my files are large enough that I cannot read the entire file into memory.
As a side note, I have a working implementation of this work flow without Spring Integration, but since we're using Spring Integration in other places in our project, I'd like to try it here to see how it performs and how the resulting code compares for length and clarity.