What is the most efficient way to store large amounts of data in a table in c#

Question

I would like store large amounts of data (text) in a table with six columns and up to ten thousand rows.

I will be building the table programatically. (Meaning I want to be able to add rows as I go)

Is datatable my best option? I'm planning to bind it to a Telerik Grid Control for ASP.NET

I am wondering I am better off going in with another type of class I do not know about.

I will be using the controls ability to sort and page, but I don't know if that matters too much on the underlying object.

Retrieval, then storage then programmer time should be considered for when I say efficiency.

Answer 1

If you want to be efficient in terms of programming hours or in retrieval time in a multi-user environment, you should consider using SQL Server.

Answer 2

I would create a class with properties that represent those six rows and then use it in a List. Very quickly:

public class Result
{
    public string Title { get; set; }
    public double Score { get; set; }
    public string Summary { get; set; }
    // etc.

    public Result(string title, double score, string summary)
    {
     Title = title;
     Score = score;
     Summary = summary;
    }

    public Result() { }
}

public class Test
{
    public static IList<Result> GetResults()
    {
     List<Result> results = new List<Result>();

     results.Add(new Result("First result", 0.914, "This is the summary"));
     results.Add(new Result("Second result", 0.783, "More summary...."));
     results.Add(new Result { Title = "Third", Score = 0.54, Summary = "Anonymous I am..." });
     // etc
     return results;
    }
}

You can then enumerate the List collection or bind it to a databound control or use it with LINQ etc.

Answer 3

First off, do what Dan Diplo describes. The DataTable is the underlying data structure for the DataSet class. DataSets are notorious for being inefficient with memory usage.

Second, for keeping things fast: don't return all 10,000 records onto one page. Show a subset at a time (like 20), in a paged grid. The less data you have to pass and parse the better.

Next, if possible, turn viewstate off on the grid. ViewState can balloon out your page size (2x or worse in some cases). That is an issue for both the browser (viewing the page), and the server (having to reload all of that ViewState data). Avoid it as best you can.

If you have to have ViewState, and performance is still and issue, move ViewState to the server, so it is not passed to the browser.

Answer 4

Use a Database.