I think Turbogears suffers a bit from being a little bit complex to get working (and by that, I mean non-trivial use - the quickstart system is fine), from seeming to be in flux all the time (different versions have different ORMs, auth libraries, web back-ends, template engines, etc), from having different maintainers over its short lifespan, etc. So the people who get it working are usually, through necessity, a bit more self-sufficient than you might see in the RoR or Django world, and thus less interested in nailing down firm specs, good documentation, or 'evangelizing' the technology.
For an example of the latter, when SourceForge.net announced that they were using TurboGears with MongoDB, you'd think the community or the maintainers would jump on that fact, showing that Turbogears is a great choice for the new interest in NoSQL. But instead there seemed to be a collective shrugging of the shoulders, as if to say, "yeah, we knew all along that TG was that great. What's the big deal?" :) The maintainer of Turbogears apparently uses MongoDB, but there's scant discussion of it on the mailing list and nothing about it at all in the docs. So anybody who saw the Sourceforge coverage is going to go to the TG site, look around, and end up a bit lost, maybe even disappointed.
So, I think that unless the existing maintainers and expert community of Turbogears are able to devote more time to looking outwards and considering what will broaden the appeal of the framework, the community will continue to stay quite small and focused.