How to handle huge table ?

Question

I would like to display to user a table which contains ~500,000 rows. The table should be calculated based on a file (i.e. the table is known at the beginning, and does not change). I guess that building the HTML text of this table is not a good idea in terms of memory performance. So how can I build such table ?

I would like user to be able to scroll the table using a vertical scroll bar. Is it possible to build on the fly only the visible part of the table ? I'm afraid to see delays because of this.

Is it a better idea to use server side programming rather than Javascript ?

I'm not restricted to any server side programming language, so any advise would be appreciated !

Answer 1

Classic example of where to use ajax.

Use javascript in combination with a server side script.

Answer 2

Send the first 250-ish rows to the user, as he scrolls down, perhaps past row 200, fetch the next 250 rows, append to the table and so on.

This is a (ui) design pattern known as "Infinite scroll".

Answer 3

This really is a case for server-side pagination, I would say, maybe in combination with Ajax. A HTML table with 500.000 rows is going to crash a lot of browsers.

You're not specifying which server side technology you work with. If you update your question, I'm sure people will be able to give you some pointers.

Answer 4

Displaying 500,000 rows all at once to a user is a bad idea. Consider other options:

• allow user to download file as CSV
• show paginated (still not very useful)
• provide filtering mechanisms (by date etc.) to allow the user to only see the data they need

If the users really needs to see all that data at once, then viewing it in the browser is one of the worst ways to do that - they should be using a tool made specifically for viewing data, like Tableau.

Answer 5

Is it possible to build on the fly only the visible part of the table ?

If you build a "fake" scroll (e.g. jquery slider) you can retrieve parts of the table instead of the whole.