How to return a byte[] doing chunking using remoting over http in C#?

Question

In .Net remoting over http, we return a byte[] back to the client by doing a SerilizationInfo.AddValue(SerializationInfoValueName, ((MemoryStream)writer.BaseStream).GetBuffer(), typeof(byte[])) call.

However, sometimes the byte[] size gets too big and causes an OutOfMemory exception. The only remedy seems to be utilize some form of chunking. A move to WCF seems most logical, however that's not possible in the near future.

So any suggestions on how to implement a generic chunking solution?

Answer 1

There is no built-in chunking. You will need to implement it yourself. It should be pretty straightforward.

Some interface like the following should do the trick:

Guid GetFile(string filename);
Guid GetFile(string filename, out int chunkCount);
byte[] GetFileChunk(Guid id, int chunkIndex);

When GetFile() is called the server can 'cache' the file in a Dictionary<> against a Guid that acts as the key. Then this Guid can be returned back to the client so that it can make further requests to download the actual chunks of the file.

GetFileChunk() should return null if the chunkIndex exceeds the number of chunks for that Guid / file.

The size of the chunks if up to you. You have a choice between responsiveness and performance. The larger the chunks the better the performance will be. But if you are updating a progress bar or something on a GUI then of course the 'responsiveness' will be adversely affected. Try experimenting with what works best.

An alternative interface may be something like:

Guid GetFile(string filename, out ulong numOfBytes);
byte[] GetFileData(Guid id, ulong index, ulong count);

This would mean the client could decide the size of the chunks it wishes to download. And then you could implement some sort of scaling strategy.

Answer 2

The solution would really depend on whether or not the resource at the server is easily "seekable". For example if you were to create a simple remoting service that simply accessed a file and returned its contents, that would be a pretty simple one. You'd just have the client pass in a desired offset and length of the data to retrieve.

But if the data is more complicated to produce on the server and you need the call to complete in one logical call, you might need to delve into things like custom sinks.

It's been a very long time since I've used .NET remoting, but since Stream derives from MarshalByRefObject, you may be able to return a stream by reference to the client. However, this can get tricky as you're now using server-side objects with a lifetime longer than the logical operation which can introduce scalability issues and has no direct analogy in WCF if you decide to upgrade later.

For what it's worth, WCF does have native support for streaming data over its built in protocols.

Answer 3

A more secure way to do it is to just remote a stream on the object you're returning from the server. Stream extends MarshalByRefObject, so if you include a reference to an open Stream (eg, pointing at the file data or buffer you want to send) in an object that's sent to the client, the client will call back to the server when you call methods on the stream proxy. This way, the client can use whatever buffer sizes it wants (within reason) to move the data around.