You only have to put the following in your configuration:
<context:annotation-config/>
If you only put the annotation on your class, the framework had to load all the classes to check if the annotation is present.
To minimize this overhead, you have to put the annotation-config tag in your configuration. That way the framework knows it has to check the classes from that configuration.
You can help the framework by specifying the package where your annotated classes are with the "base-package" attribute.
//EDIT//
This also explains the note in the documentation:
Note
<context:annotation-config/> only
looks for annotations on beans in the
same application context in which it
is defined. This means that, if you
put in a
WebApplicationContext for a
DispatcherServlet, it only checks for
@Autowired beans in your controllers,
and not your services.