It shouldn't take that long even if you used a suboptimal configuration of using embedded mode and Apache prefork MPM. Although, you could make things worse if you had MaxRequestsPerChild set to 1 for Apache processes.
Suggestions are, make sure you are using mod_wsgi daemon mode and run with a single multithreaded process (the default settings for WSGIDaemonProcess). This will ensure at least that there is only one instance of Django and not potentially one per Apache server process.
As confirmation of what you are doing, edit your question and post snippet of your current Apache configuration showing how to set up mod_wsgi bits. Include also whether you are using Apache prefork or worker MPM, determinable by running 'httpd -V' and what platform you are using.
BTW, have you tried a simple hello world WSGI program to validate your installation? See 'http://code.google.com/p/modwsgi/wiki/QuickConfigurationGuide'. And have you tried with an empty Django site, rather than your real one to see whether it is your changes?