Is it suggestable to use generics for large amount of data?

Question

I'm having let's say thousands of Customer records and I have to show them on a webform. Also, I have one CustomerEntity which has 10 properties. So when I fetch data in using a DataReader and convert it into List<CustomerEntity> I am required to loop through the data two times.

So is the use of generics suggestable in such a scenario? If yes then what will be my applications performance?

For E.g.

In CustomerEntity class, i'm having CustomerId & CustomerName propeties. And i'm getting 100 records from Customer Table Then for Preparing List i've wrote following code

while (dr.Read()) 
    { 
       // creation of new object of customerEntity 
       // code for getting properties of CustomerEntity 
       for (var index = 0; index <  MyProperties.Count; index++) 
       { 
         MyProperty.setValue(CustEntityObject,dr.GetValue(index)); 
        } 
      //adding CustEntity object to List<CustomerEntity> 
    }

How can i avoid these two loops. Is their any other mechanism?

Answer 1

I'm not really sure how generics ties into data-volume; they are unrelated concepts... it also isn't clear to me why this requires you to read everything twice. But yes: generics are fine when used in volume (why wouldn't they be?). But of course, the best way to find a problem is profiling (either server performance or bandwidth - perhaps more the latter in this case).

Of course the better approach is: don't show thousands of records on a web form; what is the user going to do with that? Use paging, searching, filtering, ajax, etc - every trick imaginable - but don't send thousands of records to the client.

---

Re the updated question; the loop for setting properties isn't necessarily bad. This is an entirely appropriate inner loop. Before doing anything, profile to see if this is actually a problem. I suspect that sheer bandwidth (between server and client, or server and database) is the biggest issue. If you can prove that this loop is a problem there are things you can do do optimise:

• switch to using PropertyDescriptor (rather than PropertyInfo), and use HyperDescriptor to make it a lot faster
• write code with DynamicMethod to do the job - requires some understanding of IL, but very fast
• write a .NET 3.5 / LINQ Expression to do the same and use .Compile() - like the second point, but (IMO) a bit easier

I can add examples for the first and third bullets; I don't really want to write an example for the second, simply because I wouldn't write that code myself that way any more (I'd use the 3rd option where available, else the 1st).

Answer 2

It is very difficult what to say the performance will be, but consider these things -

1. Generics provides type saftey

2. If you're going to display 10,000 records in the page, your application will probably be unusable. If records are being paged, consider returning only those records that are actually needed for the page you are on.

3. You shouldn't need to loop through the data twice. What are you doing with the data?