Looking for options for a live, large data set charting platform to deal with large quantity of constantly evolving data and display it via browser in an usable manner.
Would need to be based off of a DB backend vs. the "reads XML file" approach of some of the Flash apps.
Since the charts are going to be viewable on a browser, you could use something like gnuplot to generate image files of your data and display those. Gnuplot is really flexible, you can create pretty much anything, and it is quite fast. But you're going to have to read the data and feed it into gnuplot yourself. There are some Java interfaces to gnuplot in case you want to stick to Java. The set the refresh header to come reasonable amount and watch your data update.
See here about gnuplot: http://www.gnuplot.info
See here about the java interfaces: http://www.gnuplot.info/links.html
I once had to display a gantt diagram of 150 machines and 100.000 tasks. I wrote a custom Java applet with custom rendering and the data transfer was handled by http + java serialization api. I found that any other solution (SVG, VML, Flash, image tags) was simply too slow and hard to operate.
Update: Here is an image how it looks like. Unfortunately for the community, this was an industrial project, therefore, the source code is not public. I can only share some concepts about it.
The diagram area consists several distinct components: horizontal and vertical scrollbars, the diagram area, the time label and the two tables on each side. These components are linked together via event handlers. If one is scrolling/changing it affects the others. The gantt diagram consists of filled rectangles where the color is used to indicate status of the task. The rendering is done in the paint() method by looping through each visible line and between the displayable start-end dates. The rendering uses aggressive clipping instead of relying the Graphics2D's clipping feature. The users have the ability to pan and zoom the view.
The data is stored in a serializable data structure. The server side Java code contains a cache for the entire data structure. This structure gets refreshed on every 30 seconds but only the differences are retrieved from the backing database. The data then gets queried by the applet, composed into the gantt model, serialized and returned to the client side. The data refresh on the client/applet side is not automatic: users need to click on the refresh button - this allows them to evaluate the picture without unexpected changes.
Rendering a Gantt diagram does not need that many fancyness - fillRect, AlphaComposite, drawLine. If you need more complex images you will need to do more coding with my approach.
