storing large amounts of text in memory

Question

If you need to be able to store a large amount of plain text in memory so it can be searched and edited, what kind of datatype would you use?

Let's say I load a 10000 line document into my C# application for searching with LINQ, how would you represent it in memory?

Not a string, because it's got be mutable, and strings are immutable.

Answer 1

You could always use a MemoryStream and then use a StreamReader to read the data from memory.

You might get some lift from the following link which talks about using LINQ with a StreamReader. I'm not sure if it fits exactly what you're trying to do though.

http://blogs.msdn.com/ericwhite/archive/2006/08/31/linq-to-text-files.aspx

From the blog post:

  StreamReader sr = new StreamReader("TextFile.txt");

  var t1 =
    from line in sr.Lines()
    let items = line.Split(',')
    where ! line.StartsWith("#")
    select String.Format("{0}{1}{2}",
        items[1].PadRight(16),
        items[2].PadRight(16),
        items[3].PadRight(16));

  var t2 =
    from line in t1
    select line.ToUpper();

  foreach (var t in t2)
    Console.WriteLine(t);

  sr.Close();

You'd want to change the StreamReader instantiation to something like

StreamReader sr = new StreamReader(myMemoryStreamVar)

or something similar.

Answer 2

Stringbuilder would fit. Internally it's a buffer, and is mutable.

Answer 3

Try using Memory mapped files. Its a new BCL in .net 4.0.

Here's the link. http://msdn.microsoft.com/en-us/library/system.io.memorymappedfiles.memorymappedfile%28VS.100%29.aspx

Thanks

Answer 4

Hi,

StringBuilder uses heap for memory allocation, which may result in out of memory exceptions at runtime.

Thanks