how to manage very large amount of data in folders

Question

i have more than 25000 people data how can i manage this data so that there is no performance issue

Answer 1

More info needed.

What performance issues are you concerned about? Storage space? Lookup speed? Write speed?

How is the data arranged?

Generally such a data set would be naturally stored in a DB - is there a specific reason why you're looking at storing it in folders? Folders will egenrally require you to decide on a specific lookup system/key, wheras a DB allows you to dice the data however you wish.

Answer 2

For input/output, make sure the readers and writers are buffered.

Answer 3

If you want to keep it in files (and not in a DB), one important thing would be to spread them into subfolders. 25000+ files in a single folder will be much slower to access than 250 subfolders containing 100 files on the average.