C# file-based services and concurrency issues

Question

I am designing the architecture for a set of WCF services. Due to the nature of the deployment of these services (remotely deployed onto a number of unmanageable systems on client sites, therefore we cannot afford the administrative overhead of database servers), the data store has to be file based (we are leaning quite heavily toward XML for the file format).

Once again, the nature of the services means that there is a potential for concurrency issues within individual files, and I am trying to come up with a system that will behave correctly in all instances and avoid attempting to read data when there is a write operation pending.

My current thinking is taking one of two possible routes.

1 - locking files.

This would operate in the following way. All file operations will have a locking mechanism. Reads would check to ensure the required file is not currently locked before requesting data. If the file is locked, the service should sleep for a random number of milliseconds (within an as-yet undetermined range). Write operations would set the lock, commit the data and then unlock the file.

2 - additional program in the background to provide data to the services.

This version would have a secondary application in the background, exposing various public static methods, callable by the services. The background app would be solely responsible for maintaining an in-memory representation of the data, providing the data to the services, and keeping the file copies in synch with the in memory objects. In this respect, it would behave as if it were a transactionalised database server.

Of the two (or possibly other) methods of realising the goal of creating these kinds of services, which option would provide the greatest performance with least chance of concurrency conflicts? The simplicity of the design of option 1 means that I'm more in favour of that one, but I am worried that performance may suffer as a result of the "sleep" operations.

TIA
Marc.

Answer 1

I know you say you don't want administrative overhead for database servers but why don't you just use something like SQL Express. All you need is the runtime installed. Same thing would go for say using an Access database file. just needs a runtime. Then you can get around these other issues and you can just make sure you have the runtime as part of the required components of your installer. This I think would make your life much easier and you would not have the overhead of a real db server.

Another option would be something say like SQL Lite. It only needs a couple dll's deployed with your app. No overhead at all but benefit of having a db over having to manage all the file access yourself.

SQL Lite, MySQL, and even SQL Express are small and lightweight enough that they are used as data stores on hand held devices so I don't see why something like that would not work here.

Answer 2

Check out SQL Server Compact edition. It does not require any running service and supports XCopy deployment.

Answer 3

A database is an abstraction over a set of files. They're asking you to reinvent the wheel, at a fraction of the cost. I think you really need to push back on this and explain the business impact of trying to write your own database - it will take longer, cost more, and be much less reliable.

Answer 4

If you really have to write directly to the file system, I'd recommend that you have your service objects talk to an object hosted on the service that maintains authority for the file, rather than having multiple objects directly access it. Then, you can lock that object appropriately to maintain consistency.