I'm surprised no one has mentioned WebGL, and frameworks built on it. I would consider it high on the list for state-of-the-art for 3D GPU-accelerated graphics and complex animation on HTML canvas / javascript.
WebGL is a cross-platform,
royalty-free web standard for a
low-level 3D graphics API based on
OpenGL ES 2.0, exposed through the
HTML5 Canvas element as Document
Object Model interfaces. ...
WebGL
brings plugin-free 3D to the web,
implemented right into the browser.
Major browser vendors Apple (Safari),
Google (Chrome), Mozilla (Firefox),
and Opera (Opera) are members of the
WebGL Working Group.
WebGL is very solid in its support for GPU-accelerated graphics. Check out these GLSL shader demos. :-) And see ChemDoodle as an example of user interaction.
I've been working on an app using Google's O3D framework, which manages the scene graph, and uses WebGL for rendering (it used to use its own plug-in). O3D is a work in progress, and its documentation is not completely up to date, but it is under active development, and there are some good demos out there. 3D Pool may be most up your alley. The Google developers are very responsive to questions in the discussion group.
There are a number of other frameworks built on WebGL; see here. Ones that mention game development and scene graphs include Copperlicht, SceneJS, X3DOM.
WebGL runs in recent development builds of several browsers, but not IE. I've been using Firefox ("Minefield") and Chromium with good results. You will need one of these to run the above demos.
However if your requirements are that it must have no dependencies beyond HTML 5 canvas / js, WebGL may not be the right choice. It doesn't look like IE will support it anytime soon.