I would recommend any library that Jeff Heer has worked on:
I believe all three projects are open source. Jeff Heer is pretty good at referencing papers in his code at least in the Prefuse library. Looking at the Prefuse source:
BalloonTreeLayout: The algorithm used is that of G. Melançon and I. Herman from their research paper Circular Drawings of Rooted Trees, Reports of the Centre for Mathematics and Computer Sciences, Report Number INS–9817, 1998.
ForceDirectedLayout.java: No reference mentioned, but there are lot of comments describing how it works.
FruchtermanReingoldLayout.java: references Jung. On their website I found a paper which has references to various algorithms
NodeLinkTreeLayout.java: The algorithm used is that of Christoph Buchheim, Michael Jünger, and Sebastian Leipert from their research paper Improving Walker's Algorithm to Run in Linear Time, Graph Drawing 2002. This algorithm corrects performance issues in Walker's algorithm, which generalizes Reingold and Tilford's method for tidy drawings of trees to support trees with an arbitrary number of children at any given node.
RadialTreeLayout.java: The algorithm used is that of Ka-Ping Yee, Danyel Fisher, Rachna Dhamija, and Marti Hearst in their research paper Animated Exploration of Dynamic Graphs with Radial Layout, InfoVis 2001. This algorithm computes a radial layout which factors in possible variation in sizes, and maintains both orientation and ordering constraints to facilitate smooth and understandable transitions between layout configurations.
SquarifiedTreeMapLayout: This particular algorithm is taken from Bruls, D.M., C. Huizing, and J.J. van Wijk, "Squarified Treemaps" In Data Visualization 2000, Proceedings of the Joint Eurographics and IEEE TCVG Sumposium on Visualization, 2000, pp. 33-42. Available online at http://www.win.tue.nl/~vanwijk/stm.pdf. For more information on TreeMaps in general, see http://www.cs.umd.edu/hcil/treemap-history