About write buffer in general network programming

Question

I'm writing server using boost.asio. I have read and write buffer for each connection and use asynchronized read/write function (async_write_some / async_read_some).

With read buffer and async_read_some, there's no problem. Just invoking async_read_some function is okay because read buffer is read only in read handler (means in same thread usually).

But, write buffer need to be accessed from several threads so it need to be locked for modifying.

FIRST QUESTION!

Are there any way to avoid LOCK for write buffer?

I write my own packet into stack buffer and copy it to the write buffer. Then, call async_write_some function to send the packet. In this way, if I send two packet in serial, is it okay invoking async_write_some function two times?

SECOND QUESTION!

What is common way for asynchronized writing in socket programming?

Thanks for reading.

Answer 1

Sorry but you have two choices:

1. Serialise the write statement, either with locks, or better start a separate writer thread which reads requests from a queue, other threads can then stack up requests on the queue without too much contention (some mutexing would be required).

2. Give each writing thread its own socket! This is actually the better solution if the program at the other end of the wire can support it.

Answer 2

You could queue your modifications and perform them on the data in the write handler.

Network would most probably be the slowest part of the pipe (assuming your modification are not computationaly expensive), so that you could perform mods while the socket layer is sending the previous data.

Incase you are handling large number of clients with frequent connect/disconnect take a look at IO completion ports or similar mechanism.

Answer 3

Answer #1:

You are correct that locking is a viable approach, but there is a much simpler way to do all of this. Boost has a nice little construct in ASIO called a strand. Any callback that has been wrapped using the strand will be serialized, guaranteed, no matter which thread executes the callback. Basically, it handles any locking for you.

This means that you can have as many writers as you want, and if they are all wrapped by the same strand (so, share your single strand among all of your writers) they will execute serially. One thing to watch out for is to make sure that you aren't trying to use the same actual buffer in memory for doing all of the writes. For example, this is what to avoid:

char buffer_to_write[256];  // shared among threads

/* ... in thread 1 ... */
memcpy(buffer_to_write, packet_1, std::min(sizeof(packet_1), sizeof(buffer_to_write)));
my_socket.async_write_some(boost::asio::buffer(buffer_to_write, sizeof(buffer_to_write)), &my_callback);

/* ... in thread 2 ... */
memcpy(buffer_to_write, packet_2, std::min(sizeof(packet_2), sizeof(buffer_to_write)));
my_socket.async_write_some(boost::asio::buffer(buffer_to_write, sizeof(buffer_to_write)), &my_callback);

There, you're sharing your actual write buffer (buffer_to_write). If you did something like this instead, you'll be okay:

/* A utility class that you can use */
class PacketWriter
{
private:
  typedef std::vector<char>  buffer_type;

  static void WriteIsComplete(boost::shared_ptr<buffer_type> op_buffer, const boost::system::error_code& error, std::size_t bytes_transferred)
  {
    // Handle your write completion here
  }

public:
  template<class IO>
  static bool WritePacket(const std::vector<char>& packet_data, IO& asio_object)
  {
    boost::shared_ptr<buffer_type> op_buffer(new buffer_type(packet_data));

    if (!op_buffer)
    {
      return (false);
    }

    asio_object.async_write_some(boost::asio::buffer(*op_buffer), boost::bind(&PacketWriter::WriteIsComplete, op_buffer, boost::asio::placeholder::error, boost::asio::placeholder::bytes_transferred));
  }
};

/* ... in thread 1 ... */
PacketWriter::WritePacket(packet_1, my_socket);

/* ... in thread 2 ... */
PacketWriter::WritePacket(packet_2, my_socket);

Here, it would help if you passed your strand into WritePacket as well. You get the idea, though.

Answer #2:

I think you are already taking a very good approach. One suggestion I would offer is to use async_write instead of async_write_some so that you are guaranteed the whole buffer is written before your callback gets called.