improve temporary data store

Question

I need your idead to improve our implementation.

we have about 10MB data need to be processed in our application for each analysis. that data is transferred from another application packet by packet. there are about 24000 pakcets for each analysis because the data is created packet by packet in that application and then sent to us immediately.

our current implementation is like the following:

communication module receives data and puts data into an object (an instance of a data class) when a packet is received. Then it will fire an event which will contain that data object.

data process module receives event and puts the received data object into an arraylist.

after all data for one analysis are received, data process module will start data process to analyze the data. At the end the data will be stored to database and those data objects will be cleanup for next analysis.

time between each analysis is more than 10 minutes.

I have removed the clone or copy operation during the internal data transfer process. So I think we are actually passing reference around. right? I have also created a pool for those events and data objects. so those events and data objects will be reused again and again.

Another suggestion is create a shared reusable memory space (internal share, between modules) for those analysis data and pass its reference around with the event. I like this idea better but more code change from me. What I did now are almost the same in terms of memory usage. am I right?

What's your opinion? any other better ideas? for the shared reusable memory space solution, what's the correct way to implement it?

thanks,

Answer 1

You can make it thread safe by doing something like this.

public sealed class SafeBytes
{
    private readonly List<byte> _data;
    private readonly object _lock;


    public SafeBytes()
    {
        _data = new List<byte>();
        _lock = new object();
    }


    public void Add(byte[] data)
    {
        lock (_lock)
        {
            _data.AddRange(data);
        }
    }

    public byte[] ToArray()
    {
        lock (_lock)
        {
            return _data.ToArray();
        }
    }
}