Processing potentially large STDIN data, more than once

Question

I'd like to provide an accessor on a class that provides an NSInputStream for STDIN, which may be several hundred megabytes (or gigabytes, though unlikely, perhaps) of data.

When a caller gets this NSInputStream it should be able to read from it without worrying about exhausting the data it contains. In other words, another block of code may request the NSInputStream and will expect to be able to read from it.

Without first copying all of the data into an NSData object which (I assume) would cause memory exhaustion, what are my options for handling this? The returned NSInputStream does not have to be the same instance, it simply needs to provide the same data.

The best I can come up with right now is to copy STDIN to a temporary file and then return NSInputStream instances using that file. Is this pretty much the only way to handle it? Is there anything I should be cautious of if I go the temporary file route?

EDIT | I should mention, it's not actually STDIN, this is in a multithreaded FastCGI application and it's the FCGX_Request.in stream which did come from STDIN.

Answer 1

When reading data from a pipe or socket, you have three options:

• Process it and forget it.
• Add it to a complete record in memory and process it before or after doing so.
• Add it to a complete file and process it before or after doing so.

That's the complete list. There's nowhere else to record it but short-term or long-term storage, so the only other thing you can do with data you read is to not record it at all.

The only other way to get the data again is for whatever sent it to you to send it again.