Fast Algorithm to Generate 500,000 html file.

Question

What is the fastest way (Algorithm) to generate 500,000 static html files from DB?

And is it a good practice to put all this files in single folder? or create hierarchically for this files?

We want to handle about 6,000,000 concurrent hit, so the static files will be a good solution for that. the source DB will be simple flat table without JOINS.

We want to generate this files from single table contains 500k records. the file names will be first field from this table. the HTML file will contains to display the data about 900 byte.

Answer 1

There is a file limit (atleast in linux, around 32k items) so no I wouldn't think it's smart to do that.

NTFS has a limit of 4,294,967,295 files in a folder.

Answer 2

Even if your file system can "cope" with 500,000 files in a single directory, it's unlikely to be able to perform well. Even if it can perform well, it's likely to be hard for humans to manage those files.

I'd definitely put them in a hierarchy.

As for the fastest way to generate them - you've asked for an algorithm, but without stating what you want it to do. There are any number of technologies you might want to use - whichever you're most comfortable with is probably the best bet - and any number of ways of approaching the task, depending on what it really consists of.

Answer 3

Hierarchically would be best for performance because many applications will loop though all the files in a single directory. For instance, Windows Explorer. And that will make the application slow.

The fastest way to extract them would be to write a small C program using the database's headers and fwrite() etc...

F.Y.I.

NTFS can hold 4,294,967,295 files: http://en.wikipedia.org/wiki/NTFS EXT3 can hold VolumeByteSize/2^13 files: http://en.wikipedia.org/wiki/Ext3#cite_note-0

Answer 4

Why not just store the generated HTML in the database? It seems like you'll effectively be treating the file system as a database anyway - At least if you store the HTML in a database you can rely on the DBMS to optimise lookup performance (e.g. by caching recently queried HTML) and you can add indices and analyse query performance. Otherwise you'll just be hammering the file system instead; i.e. moving the problem elsewhere.

Also, I would suggest taking a step back and seeing where the bottleneck currently lies. Storing HTML (presentation layer data) is not an elegant solution - If the real problem is due to query performance perhaps consider introducing denormalised tables into your schema containing intermediate results, from which you can quickly generate HTML.

Answer 5

If I were to do this, I'd store the generated files in a hierarchy, based on the file name (IFF the filenames are sufficiently well distributed), so "onefile.html" gets stored in "o/n/e/onefile.html" and "anotherfile.html" as "a/n/o/anotherfile.html". Using three levels of storage isn't necessary, you may require four. Also, chunking the pathnames on a per-character basis may not be the best distribution, you may be better off using two or three characters, depending on how your distribution looks.

I've used similar storage schemes for received faxes for an electronic fax service in the past (using longer and longer prefixes of the destination fax number as pathname components).

I guess the reason you're looking at generating the flat files is to amortise the cost of generating the HTML?

Answer 6

500.000 entries each of them approximately 1k in size? So we are talking about 500 MByte of data. If possible I would simply put the whole thing on a ramdisk, if you need filesystem capabilities, keep it in memory as an ordered structure (hastable, array of some sort) if you don't. Is there a specific reason, why you don't store the results in a temporary database table? (SQLite)