multithreading in c#

Question

hi i am new to threading. my boss gave me a scenario.we have a list of object which is 0f 70gb,we want to load it from a database.it takes time i want to utilize cpu(multi threading) in this scenario .i.e partializing data ,load first part, than second part,whensecond part is loaded mean while first part is processed, what i shuld do plese guide me.

Answer 1

Take a look at these resources:

• Data paging in SQL Server (how to fetch data from the database in chunks)
• Data paging in ASP.NET
• Introduction to multithreading in C#

Answer 2

The easiest way to start with multithreading in c# is to use backgroundworkers. Take a look to it LINK

Answer 3

My experience of scenarios like this is the data retrieval always takes longer, and multi-threading only makes a difference in very CPU intensive tasks.

E.g. I retrieve 5000 accounts from the DB at a time with all of their related person information using FOR XML so that I get relational data I can deserialize. This takes approximately a minute, then I run complex segmentation which runs in less than a second. In this scenario multi-threading would not make a difference.

Another scenrio I had was where I was making a simple DB call thousands of times, and each was subject to network delays, whilst the SQL server was being stretched as the table was small and fully indexed so searching was instant. in this case it was actually quicker to spawn a few threads and cal the database mutliple times at once using these threads. Spawning 8 threads gave approximately a fourfold increase.

Look into the System.Threading namespace.

Answer 4

Do you have control over how the data is loaded into the database, and what the hardware looks like?

To load 70GB of data, you will be I/O bound at first. If the data lives in a single volume, trying to use multiple threads will just cause the disk heads to thrash as they seek back-and-forth across the drive.

That means your first step should be to maximize the performance of your disk subsystem. You can do that by doing things like:

1. Limiting your disk partition size to the first third of the drive
2. Putting as many spindles as you can into a single large RAID volume, up to the speed of your disk controller
3. Using SSDs instead of rotating magnetic drives
4. Using a high-speed disk controller
5. Using multiple disk controllers
6. Spreading your drives out among multiple controllers

Once you have that part done, the next step is to partition your data among as many disks and controllers as possible, while still allowing your log file to be on a volume by itself. If you can fill two entire controllers with fast RAID volumes, then divide your data among them. In some cases, it can help to use SQL Server's table partitioning mechanisms to help with the process and to force certain parts of the table to be on certain physical volumes.

After the partitioning is done, then you can build your app to have one thread read from each physically separate partition, to avoid disk thrashing.

Once you're past being I/O bound, then you can start thinking about ways to optimize the CPU side of things -- but it's pretty unusual to get to that point.

Depending on how much speed you need, this type of thing can get complex (and expensive) quickly....

In case it helps, I discuss a number of these infrastructure issues in detail in my book: Ultra-Fast ASP.NET.