What's the best design in .NET to use for sending data over an unreliable (3G) network connection?

Question

I'm re-designing an app I inherited that sends digital photos from a laptop to a web server. The idea is to take photos "out on the field" and have them instantly published on a web page (with some more fancy features).

Typical scenario
1. Photos are transferred from the camera to the laptop using standard USB.
2. The photos are processed in various ways. (Not important)
3. Each photo is POSTed in small pieces (~64 kb each) using a webrequest to a standard Apache web server where it's merged together again.

The problem with the current design is that it often hangs when the network connection is unreliable. As we're using a mobile network (3G) and often end up out of coverage, I need a way to handle this properly.

My question is whether there's a better solution for doing this that won't make the app hang when the connection drops every now and then.

(Bonus question is how this could be properly unit tested without having to take a hike with the laptop.)

EDIT 2008-11-24: I've now managed to set up a proper test environment for this using a combination of NetLimiter and TMnetsim (freeware). I tried setting 5 kb/sec and dropping 1% of all packets - my app still works well with the new design.

EDIT 2008-12-11: Just to update how I did this. I created one background worker (as suggested below) that is started whenever a camera is detected to copy the photos from the camera to PC. Then another background worker i started when files arrive on PC to upload using asynchronous HTTP transfer. It sure was a pain to get everything right, especially since the operation should be "cancellable" at any time... But anyhow, now it works. A big THANKS to everyone who helped me!

Answer 1

First find out why it's hanging - are the requests just sitting there? Do they time out? What happens if you lower the timeout setting?

Are you doing the POST from the UI thread? (Don't do that :)

You could potentially detect the connection dropping by making heartbeat requests with very short timeouts, too.

Answer 2

Firstly, I would put the transfer process outside of the application. Sync files with a utility that allows it to restart transfer from the middle of the last transfer.

You can simulate communication drops with some type of Faraday cage.

Answer 3

Have you looked at the Sync Framework?

Hope it helps, Bruno Figueiredo

Answer 4

You probably need to use WebRequest.BeginGetResponse instead of WebRequest.GetResponse, although there doesn't then seem to be a way to cancel the response (maybe disposing of the WebRequest will help).

Also, you could try playing with the Timeout property of WebRequest

Answer 5

A way to test this without taking out your laptop into the fields: try m0n0wall on a spare machine, and set up its firewall rules to squeeze bandwidth and drop packets.

Alternatively, install netlimiter on your server/client

Answer 6

I'd avoid using HTTP at all from any thread that has UI unless you actually want to block until the response is received. You can try using the same logic from a background thread which will run as long as it needs to. Just be sure to have logic which will detect when the connection is lost (probably from a timeout) and will retry at a regular (but not frequent) interval until connecting again.

Your best bet would be creating some sort of background worker process which will upload the photos once they are saved to a dropbox directory on the device. I will say though that creating a .NET based background process is not trivial.

Answer 7

I would try using ASP.NET Caching techniques with ADO.NET Sync services for data transfer.

try caching your images into a binary or BLOB format first and storing them in a local CE database (or Express). then Sync the data to a central server database, via WebServices.

Try creating a centralized web service to handle your cached image transfers.

The web service would receive the data via Sync services in an "occassionally connected" scenario.

for unit testing, try using a virtual machine (VMware) or a laptop on a dialup connection.