Memory Mapped Files .NET

Question

I have a project and it needs to access a large amount of proprietary data in ASP.NET. This was done on the Linux/PHP by loading the data in shared memory. I was wondering if trying to use Memory Mapped Files would be the way to go, or if there is a better way with better .NET support. I was thinking of using the Data Cache but not sure of all the pitfalls of size of data being saved in the Cache.

Answer 1

If you are looking for a Memory Mapped library for C#, take a peek at Tomas Restrepo's filemap wrapper. It's licensed under GPL.

Answer 2

Memory Mapped files can be used when you have a large amount of data and don't want to incur the cost of marshaling it across process boundaries. I have used it for a similar purpose. You need to be fairly comfortable with unsafe and pinned memory concepts in .NET to take advantage of MMFs. Apparently, the Enterprise Library's caching block contains code which wraps the underlying C# API. I have seen at least one other implementation elsewhere.

If you can live with the marshaling cost, it's probably easier and more elegant to use some kind of .NET remoting solution.

Answer 3

You might want to just throw it in the Cache[] object. You can set a cache expiration based on the real file. Then whenever you modify the actual file the contents will be null for the object in the cache and you can reload it. This may not be appropriate if you're dealing with a large number of bytes.

byte[] fileBytes = Cache["fileBytes"];
if (null == fileBytes) {
   // reload the file and add it to the cache.
   string fileLocation = Server.MapPath("path/to/file.txt");
   // Just a same of some bytes.
   fileBytes = new byte[10];
   Cache.Insert(fileLocation, fileBytes, new System.Web.Caching.CacheDependency(fileLocation));
}

I guess I don't have a specific answer about the performance characteristics of the cache and large amounts of data. http://www.alachisoft.com/ncache/asp-net-cache.html States that you get between 2 and 3 gigs of cache space that must be shared between your application and the cache.

Answer 4

I know this is a bit late, but the .NET 4.0 framework now supports memory-mapped files out of the box:

http://blogs.msdn.com/salvapatuel/archive/2009/06/08/working-with-memory-mapped-files-in-net-4.aspx