How to quickly deploy assets to Amazon S3 with an Ant target?

Question

What is the quickest way to deploy content to a CDN with an Ant target? My Ant target is running on a continuous integration server (Hudson). My current solution uses curl and is a bit slow. Should I use wput or something else and how would I do that in ant?

<target name="Deploy">
<for param="file"> 
    <path> 
        <fileset dir="${basedir}/output" includes="**/*"/>
    </path>
    <sequential> 
        <echo> Deploy @{file} </echo>
        <exec executable="curl"> 
           <arg value="-F name=value"/>  <!-- params for secure access -->
           <arg value= "-F file=@{file}"/> 
           <arg value="http://cdn.com/project"/&gt;
        </exec>
    </sequential> 
</for>
</target>

---

Several ideas have come up to speed up the transfer of content to the cdn

1) max out the pipe bandwidth by using the parallel ant task to simultaneously transfer several mutually exclusive filesets. For example, if there are three sub-folders in the output folder, each can be given to a different parallel task, and each would iterate through the files, calling curl on each file to transfer it to the cdn. http://ant.apache.org/manual/Tasks/parallel.html

2) write a custom ant task (bash script?) that would have local knowledge about the build so that any files that were changed by the last build get marked and only those files would be transfered. This would prevent sending a file that is already on the cdn.

3) read the remote directory from the cdn and use timestamps to determine which files to send. This may not be possible depending on the cdn and whether it allows such queries. I was hoping wput could do this but I don't see an option for that. http://wput.sourceforge.net/wput.1.html

Answer 1

RESOLVED

I found a blog titled "Deploying assets to Amazon S3 with Ant" which was extremely helpful. It uses a python script 's3cmd sync' which only transfers files that don’t exist at the destination.

I ended up with this ant target:

<target name="s3Upload">

<property name="http.expires" value="Fri, 31 Dec 2011 12:00:00 GMT" />
<exec executable="${PYTHON_DIR}\python.exe" failonerror="true">
  <arg value="${PYTHON_DIR}\Scripts\s3cmd" />
    <arg value="--guess-mime-type" />
    <arg value="--add-header=Cache-Control:public, max-age=630657344" />
    <arg value="--add-header=Expires:${http.expires}" />
    <arg value="--encoding=UTF-8" />
    <arg value="--skip-existing" />
    <arg value="--recursive" />
    <arg value="--exclude=*.log" />
    <arg value="--acl-public" />
    <arg value="sync" />
    <arg value="${CDN_DIR}/" />
    <arg value="s3://my-project-cdn/" />
  </exec>

</target>