Is Javascript ready for visualizing large datasets?

Question

We've got some data (10-50 columns, hundreds of thousands of rows) that we usually visualize in excel as a line graph or stacked bar chart. Users want to be able to zoom in and out of the graph to get down to the individual samples, but these kind of operations really bring Excel to its knees. I'm thinking about embedding the data into an HTML page, with inline Javascript to handle the visualization in the browser. Something like the flotr JS charting lib would be leveraged for the charts.

Is this a stupid idea? Is the browser ready for this kind of load? Is this a solved problem that I just should have googled more thoroughly before asking?

Answer 1

Is Javascript ready for visualizing large datasets?

Yes - the language is to a point where it effortlessly, in the right environment, handles significant recordsets and manipulates, visualizes, etc them. The language itself is fine.

Is this a stupid idea?

No, in fact you can count on nearly every computer to be able to run this capable, cross platform language.

Is the browser ready for this kind of load?

Some might be - depends on what processing and actions you are actually taking. With Chrome using a fast javascript engine, and more and more people heavily relying on it, the javascript speed war is ignited. I think this is a perfectly valid usage scenario.

You'll need to be prepared for benchmarking and optimization, which means digging into the guts of javascript. Please publicize your results so deficiencies can be mended.

Answer 2

Javascript is probably ready for it, since javascript itself has gotten to be quite fast. In my experience browsers are generally not ready to handle very large DOM structures. At the least you can expect to be spending a lot of time trying to find out why things are slow. You'll also discover that a lot of "standard" javascript libraries (prototype/jquery come to mind) are not suitable for working with "excessively" large DOM structures.

Be prepared to find out that a given operation is slow on all browsers, but in the end it turns out to be for 3-4 different reasons on the different browsers. This is based on experience from working with moderately oversized DOMs. While certainly possible, it's going to cost a fair amount of work to get a decent result.

Answer 3

I highly recommend Adam's suggestion to perform some benchmarking and optimisation. I've recently done some work on plotting large datasets with Flot and experienced less than acceptable performance with Internet Explorer (e.g. the entire browser hanging for ~20s on my developer box while plotting charts).

Flot uses the canvas element for charting which is not supported in Internet Explorer. Flot provides Internet Explorer support using the ExplorerCanvas library. This library uses VML, drawing graphics by manipulating VML elements through the DOM.

Using the Internet Explorer 8 script profiler I discovered most of the time taken rendering the plot was spent calling the native insertAdjacentHTML method to create the VML elements. Because there was nothing that can be done to improve performance of calls to native methods I instead worked on reducing the number of data points plotted (in turn reducing the VML elements created in the DOM) to get acceptable performance.

If you don't need or care about Internet Explorer support you should find Flot/Flotr is quite capable of handling large datasets. But if you do need to support Internet Explorer be prepared to run into performance problems when charting large datasets.