AWS EC2- Synching source code files with S3 - is it a proper approach?

Question

On an app server in which a few source files change frequently, Is the following approach recommended?

Use a cron job with S3tools to sync the source files with S3 private bucket (every 15 mins for example).

On server start up - Use user data script to sync with the sources bucket to retrieve the latest sources.

Advantages: 1. No need to attach EBS for app server just to save a few files 2. Similar setup to all app servers 3. Sources automatically backed up. 4. As a byproduct, distributes code to multiple app servers automatically.

Disadvantages: keeping source code on S3 other?

What do you think about this methodology? Is this the right way to use EC2 when source code change frequently (a few times a day) please recommend the best approach to run EC2 instances where sources change often.

Answer 1

If you're going to be running a large number of EC2 instances, then it will be less effort to have them sync themselves from a central location (ie, you sync to private bucket, app-servers sync from that bucket).

HOWEVER, recognize that updates to an S3 bucket are atomic only at the object level, and more importantly, are not guaranteed to be immediately consistent (although I recall seeing a recent note that the us-west endpoint does offer read-after-write consistency).

This means that your app-servers may load a set of new files that are internally inconsistent -- some will be old, some will be new. If this is a problem for you, then you should implement a scheme that uploads directly to the app-servers, and ensures changeset consistency (perhaps by uploading to a temporary directory that is then renamed).

Answer 2

I think you're better off using a proper source code repository, like Subversion or Git, rather than storing the source files on S3. That way you can have a central location for the source files while avoiding the update consistency problems that kdgregory mentioned.

You can put the source repository on one of your own servers outside of EC2, or host it on an EC2 instance (make sure the repository files are on an EBS volume in the latter case).