The thing with ATG is that it is both a framework and a solution.
At the framework level, it is a Java based application platform for hosting web-based applications, as well as RMI accessible business components, with an ORM layer, a component container, an MVC framework, and a set of tag libraries for JSP.
However, describing it thus is selling it short - even if you ignore the fact that they had most of this stuff before there was a J2EE spec.
For starters, the ORM (Repositories) does more than just mapping to relational databases. It can handle LDAP, XML and file system data sources in a coherent and consistent manner.
The component framework (The Nucleus) does far more than, say, the Spring bean container does - though I agree that properties files are less expressive than XML files
The MVC framework (the basic Formhandler pattern) is more elegant, cleaner and easier to work with than most other web MVC frameworks I have seen, and the APIs are more consistent with the Servlet APIs
The JSP tags for binding form elements on a page to values on business objects etc. are more elegent and cleaner than the form binding tags in any other framework I have seen. The mechanism of writing your own tag library equivalents (Droplets) is much more4 consistent with the Servlet API than standard J2EE tags.
(While I am on the subject of consistency, I should mention that the APIs for writing the equivalent of filters (Pipeline Servlets) is a lot more consistent with the Servlet API than filters are).
What's more, the framework at the basic level gives you an RMI container, distributed caching, distributed locking and distributed singletons, distributed events and messaging, a task scheduler, a rules engine and a mechanism for defining business workflows with custom actions and outcomes, a graphical editor for business workflows, support for versioned data, support for roles and rights, logging and auditing - all out of the box, and all using very coherent and consistent APIs
Then at the solution level, you have the components and the APIs for dealing with user profiling, identity management and personalisation, content authoring and publishing, content search, product catalogs for tangible and intangible goods, product search and guided navigation, pricing, tax calculation, promotions, shopping carts, gift lists and wish lists, payment types, shipping methods, order tracking, customer relationship management etc.
In addition, the extension points and integration points are usually very well designed and quite well documented. They support integration with pretty much anyone who is anyone in the e-commerce and publishing space for things like authoring and content management, identity management and security, product catalogs, search and guided navigation etc. Also, almost all areas of the framework are extensible and pluggable so you can write your own components to enhance or replace the ones out of the box.
Together, this makes a very compelling platform.
However, because of the way it was developed - possibly the very thing that makes it so consistent - means it is a very strange beast to most J2EE developers. ATG usually has its own way of doing pretty much everything that is now considered standard in J2EE. It used to have its own form of JSP called JHT^ML (I ahve heard that they invented it and donated the technology to Sun to become JSP), it has its own component container to host and manage the lifecycle of POJOS using dependency injection and reflection (sound familiar? they had it about a decade before Spring), it has its own APIs for developing tag libraries called Droplets, it has its own APIs for developing filters (Pipeline Servlets) and its own filter chain (the Pipeline), they have gone through two iterations of ORM - the first (Relational Views) only did ORM and used POJOs, reflection and bean mapping files (sounds familiar? they had this about a decade before Hibernate) and the second (Repositories) uses value dictionaries and gives a very consistent API for sourcing data from relational databases, LDAP, XML files and other sources. Moreover, ATG has (or rather, had) its own application server.
Not to say that ATG is not J2EE compliant - it is. You can host WARs and EARs in an ATG server, and they just work.
Over time, they seem to be moving away from the proprietary nature of things at the framework level while adding more and more value at the business solution level. They have moved away from JHTML and now use standard JSP - they have implemented all their custom tags as JSP tag libraries. They have moved away from their proprietary application server and now support deploying applications as EARs on Weblogic, WebSphere and JBoss.
Likewise, I do not think they are averse to open source - they use quite a few open source frameworks internally - but they are certainly not open source. I doubt they will be - though I would like to see them open source some of their core framework components or migrate to more widely supported open source replacements - perhaps a Spring-based Nucleus?
So, to answer your question, I do not think that there is an open source alternative to ATG. While you will find many open source alternatives to the framework bits - such as logging, ORM, bean container, tag libs, etc. you will be hard pressed to find a solution that can do so much so consistently.
Of course, depending on what you want out of a free open source replacement for ATG, you could start with a JBoss based application that uses either EJB3 or Spring, and either EJB3 or Hibernate for your compoennt container and ORM. You can then add Drools for the rules and workflow engine, Spring MVC for the MVC and form binding tags, JSTL for other tags, Commons-DBCP for your datasources, Log4J for logging, Quartz for scheduling. You could look into integrating Apache Lucene with Hibernate for advanced searching.
Acegi can provide security in Spring - but I do not know what open source frameworks can provide the full identity management/user profiling functionality that ATG provides.
You can probably hand-roll the versioning support and a content authoring workflow on top of these technologies.
There are a few open source 'shopping cart' applications out there - but I am not too familiar with how extensive they are.
Perhaps the question is best asked "Which free and open source frameworks would you recommend for replacing which aspect of ATG". I have gone and created another question framed in this way. See http://stackoverflow.com/questions/100588