How to change internal buffer size of DataInputStream

Question

I'm using this kind of code for my TCP/IP connection:

sock = new Socket(host, port);
sock.setKeepAlive(true);
din = new DataInputStream(sock.getInputStream());
dout = new DataOutputStream(sock.getOutputStream());

Then, in separate thread I'm checking din.available() bytes to see if there are some incoming packets to read.

The problem is, that if a packet bigger than 2048 bytes arrives, the din.available() returns 2048 anyway. Just like there was a 2048 internal buffer. I can't read those 2048 bytes when I know it's not the full packet my application is waiting for. If I don't read it however - it'll all stuck at 2048 bytes and never receive more.

Can I enlarge the buffer size of DataInputStream somehow? Socket receive buffer is 16384 as returned by sock.getReceiveBufferSize() so it's not the socket limiting me to 2048 bytes.

If there is no way to increase the DataInputStream buffer size - I guess the only way is to declare my own buffer and read everything from DataInputStream to that buffer?

Regards

Answer 1

Wrap your data input stream around larger buffered input stream:

DataInputStream din =
  new DataInputStream(
    new BufferedInputStream( sock.getInputStream( ), 4096 )
  );

But I don't think it's going to help you. You have to consume the input from the socket otherwise the sender will get stuck.

You should probably invest more time in working out a better communication protocol.

Answer 2

If you dig into the source of DataInputStream getAvailable() is actually delegated to the stream its reading from, so the 2048 is coming from the socket input stream, not the DataInputStream (which is implemented by default in native code).

Also be aware that the API states for an InputStream

Note that while some implementations of InputStream will return the total number of bytes in the stream, many will not. It is never correct to use the return value of this method to allocate a buffer intended to hold all data in this stream.

So just because you are receiving a value of 2048 does not mean there is not more data available, it is simply the amount that is guaranteed to be read without blocking. Also note that while BufferedInputStream as suggested by Alexander is an option, it makes no guarantee that the buffer will always be filled (in fact if you look at the source it only attempts to fill the buffer when one of the read calls is made).

So if you want to make sure you are always receiving "full packets" you are likely better off creating your own input stream wrapper where you can add a specialty method "byte[] readPacket()" that will block until it can fill its own buffer as data is read from the underlying socket stream.

Answer 3

This is not how you use InputStreams. You never want to use the available method, it's pretty much useless. If you need to read "packets" of data, then you need to design that into your protocol. One easy way to do that is send the length of the packet first, then send the packet. the receiver reads the length of the packet, then reads exactly that many bytes from the stream.

Answer 4

I'm going to make an assumption about what you're calling a "packet". I am going to assume that your "packet" is some unit of work being passed to your server. Ethernet TCP packets are limited to 1536 bytes. No matter what size writes the peer is performing.

So, you cannot expect to atomically read a full unit of work every time. It just won't happen. What you are writing will need to identify how large it is. This can be done by passing a value up front that tells the server how much data it should expect.

Given that, an approach would be to have a thread do a blocking read on din. Just process data as it becomes available until you have a complete packet. Then pass the packet to another thread to process the packet itself. (See ArrayBlockingQueue.)

You socket reader thread will process data at whatever rate and granularity it arrives. The packet processor thread always works in terms of complete packets.