Why does IOHandler.ReadStream blocks thread when client connects to server in Indy?

Question

Hi,

Today I faced a weird behavior using Indy 10 (shipped with Delphi 2010). Here is the problem:

Suppose we have a IdTcpClient in our client, and a IdTcpServer in our server app, and this code in OnExecute event-handler for our IdTcpServer:

procedure TForm1.IdTCPServer1Execute(AContext: TIdContext);
var
  AStream: TStringStream;
  S: string;
begin
  AStream := TStringStream.Create;
  try
    AContext.Connection.IOHandler.ReadStream(AStream);
    S := AStream.DataString;
  finally
    AStream.Free;
  end;
end;

Now, when the client tries to connect to the server, using TIdTcpClient.Connect; on the server, TIdTcpServer.OnExecute is invoked, and the thread running inside OnExecute event-handler is blocked when execution reaches AContext.Connection.IOHandler.ReadStream(AStream) line!

When I trace the code, the problem is caused when ReadLongInt is called inside ReadStream to get bytes count. ReadLongInt calls ReadBytes. Inside ReadBytes, FInputBuffer.Size is zero. There, in a loop ReadFromSource is called, and eventually execution reaches to TIdSocketListWindows.FDSelect which calls "select" function from WinSock2, and execution stops here, and nothing will be received from that client connection. I tried giving value to AByteCount and AReadUntilDisconnect parameters too, but it did not change the behavior.

If I replace ReadStream with ReadLn, then connecting to server does not block code execution, and the data sent from client is read by server.

Is there anything wrong with the code? Or is this a bug?

Regards

Answer 1

The problem is in your code, not in ReadStream(). It is acting as designed.

It accepts 3 parameters for input:

procedure ReadStream(AStream: TStream; AByteCount: TIdStreamSize = -1; AReadUntilDisconnect: Boolean = False); virtual;

You are only providing a value for the first parameter, so the other two parameters use default values.

When the AByteCount parameter is set to -1 and the AReadUntilDisconnect parameter is set to False, ReadStream() is designed to assume that the first 4 bytes received (or 8 bytes, if the IOHandler.LargeStream property is set to True) are the length of the data being sent, followed by the actual data afterwards. That is why ReadStream() is calling ReadLongInt(). Not only does this tell ReadStream() when to stop reading, but it also allows ReadStream() to pre-size the target TStream for better memory management before receiving the data.

If the client is not actually sending a 4-byte (or 8-byte) length value ahead of its data, then ReadStream() will still interpret the beginning bytes of the real data as a length. This typically (but not always, depending on the data) results in ReadLongInt() (or ReadInt64()) returning a large integer value, which would then cause ReadStream() to expect a huge amount of data that will never actually arrive, thus blocking the reading indefinately (or until a timeout occurs, if the IOHandler.ReadTimeout property is set to a non-infinite timeout).

In order to use ReadStream() effectively, it needs to know when to stop reading, either by being told how much data to expect ahead of time (ie: AByteCount >= 0), or by requiring the sender to disconnect after sending its data (ie: AReadUtilDisconnect = True). The combination of AByteCount = -1 and AReadUtilDisconnect = False is a special case, when the length is encoded directly in the streaming. This is primarily used (but not limited to) when the sender calls IOHandler.Write(TStream) with its AWriteByteCount parameter set to True (it is False by default).

When dealing with non-textual data, it is always a good idea to send the data length ahead of the actual data whenever possible. It optimizes reading operations.

The different parameter combinations of ReadStream() work out to the following logic:

1. AByteCount = -1, AReadUtilDisconnect = False: read 4/8 bytes, interpret as a length, then keep reading until that length is received.

2. AByteCount < -1, AReadUtilDisconnect = False: assume AReadUntilDisconnect is True and keep reading until disconnected.

3. AByteCount > -1, AReadUtilDisconnect = False: pre-size the target TStream and keep reading until AByteCount number of bytes are received.

4. AByteCount <= -1, AReadUtilDisconnect = True: keep reading until disconnected.

5. AByteCount > -1, AReadUtilDisconnect = True: pre-size the target TStream and keep reading until disconnected.

Depending on the kind of data the client is actually sending to the server in the first place, chances are that ReadStream() is likely not the best choice for reading that data. The IOHandler has many different kinds of reading methods available. For instance, if the client is sending delimited text (especially if it is being sent with IOHandler.WriteLn()), then ReadLn() is a better choice.