This is not pretty, but it is what I do. (Sources javac ant task javac man page) Using the compilerarg attribute I can pass in the annotation processing related arguments that are not directly supported by the javac ant task.
<javac srcdir="${src}" destdir="${classes}" ... >
....
<compilerarg line="-processorpath ${processorpath}"/>
<compilerarg line="-processor ${processor}"/>
<compilerarg line="-s ${whereToPutGeneratedClassFiles}"/>
</javac>
I do not use the APT tool because the documentation states
Be advised that the Apt tool does appear to be an unstable part of the JDK framework, so may change radically in future versions. In particular it is likely to be obsolete in JDK 6, which can run annotation processors as part of javac.
If you really don't care for compiler args, you can jar your annotation processors like this
<jar destfile="${annotationprocessorjar}" ... >
...
<service type="javax.annotation.processing.Processor" provider="${your.annotation.processor.fully.qualified.name}"/>
</jar>
Then you can do
<javac ... make sure ${annotationprocessorjar} is in classpath>
</javac>