amazon-s3

EC2 automation tools / strategies?

What tools or strategies are you using for automation of EC2 activities? I need to be able to bring up a number of EC2 instances, provision various software to it (primarily Python packages), interact with S3 (primarily download data), and run various jobs. I'll be doing this both on-demand and on a scheduled basis. I'm trying to decid...

Python: efficiently join chunks of bytes into one big chunk?

I'm trying to jury-rig the Amazon S3 python library to allow chunked handling of large files. Right now it does a "self.body = http_response.read()", so if you have a 3G file you're going to read the entire thing into memory before getting any control over it. My current approach is to try to keep the interface for the library the same ...

Access control for static content served from S3?

I'm thinking about serving user-specific static content from S3 - the user needs to be authenticated in order to access his static content. So if user A has content c1, c2 and use B has c3, c4, only A should be able to access c1, c2 . What's a good way to accomplish this? Is there a way to perform per-user / per file authentication in S...

Rails: Image cropping with Paperclip, S3 and RMagick.

Hey all, I'm currently trying to code a custom image cropping system similar to other ones on the internet where a user can select a cropping area and then have their image cropped accordingly. The application is in Rails and we're using Paperclip with Amazon S3 to store the files. I'm having a lot of trouble though getting RMagick to ...

How do the amazon web services work?

What information have you been able to gather regarding how do the amazon web services work? What hardware do they use What web server What Operating System What storage for AWS What virtualization software for EC2/EBS What software for they distributed firewall for EC2 Physical location of their data centers. I like their services v...

Extending/replacing Html.Image for Amazon S3 (or other CDN)

Just want to confirm that there is no way to extend or replace Html.Image functionality without writing a replacement function. I want to write a function that will use Amazon's S3 service for hosting images. The best approach I've come up with is a helper method Html.SmartImage which would check a configuration property to see if i wa...

Updating permissions on Amazon S3 files that were uploaded via JungleDisk

I am starting to use Jungle Disk to upload files to an Amazon S3 bucket which corresponds to a Cloudfront distribution. i.e. I can access it via an http:// URL and I am using Amazon as a CDN. The problem I am facing is that Jungle Disk doesn't set 'read' permissions on the files so when I go to the corresponding URL in a browser I get ...

Apply a red border to all images hosted on CDN with jQuery

I am using Amazon S3 and certain images are hosted there. Its kind of a pain to check each image in turn to verify its coming from the CDN and sometimes new images are added and someone forgets to upload them, and I thought it would be nice to have a visual cue - accessible from a debug panel. I'd like to draw a red border on all images...

What's the best method for passing AWS credentials as user data to an EC2 instance?

I have a job processing architecture based on AWS that requires EC2 instances query S3 and SQS. In order for running instances to have access to the API the credentials are sent as user data (-f) in the form of a base64 encoded shell script. For example: $ cat ec2.sh ... export AWS_ACCOUNT_NUMBER='1111-1111-1111' export AWS_ACCESS_KEY_...

Affordability of Amazon Simple Storage Service (S3)

I have a website that attracts about 30,000 visitors per month. It has a lot of photos and PDF files which eat up a good deal of bandwidth. It's hosted by site5.com, which offers unlimited bandwidth & storage for ~$5 per month. According to site5's statistics, my site has about 20 GB of downloads per day, but I've seen it as high as 116...

How do I bulk upload to s3?

I recently refactored some of my code to stuff rows into a db using 'load data' and it works great -- however for each record I have I must upload 2 files to s3 -- this totally destroys the magnificent speed upgrade that I was obtaining. Whereas I was able to process 600+ of these documents/second they are now trickling in at 1/second be...

Asynchronous File Upload to Amazon S3 with Django

I am using this file storage engine to store files to Amazon S3 when they are uploaded: http://code.welldev.org/django-storages/wiki/Home It takes quite a long time to upload because the file must first be uploaded from client to web server, and then web server to Amazon S3 before a response is returned to the client. I would like to ...

404 redirect with cloud storage

I'm hoping to reach someone with some experience using a service like Amazon's S3 with this question. On my site we have a dedicated image server. And on this server, we have an automatic 404 redirect through Apapche so that, if a user tries to access an image that doesn't exist, they'll see a snazzy "Image Not Available" image. We're...

How do I secure my Amazon S3 photos but still make them available via URLs?

I'm planning on making my (family) photo collection available online. I want to use S3 and build an ASP.NET site that will display the photos. I don't want the website to pull down the S3 content and return it to the browser. I want browsers to be able to go directly to S3 without affecting my ASP.NET bandwidth. It is possible to build ...

Actionscript 2, Flash 8 and Amazon S3. Is there an AS2.0 API?

Hi, I am developing an application that loads images and video into a Flash player (currently using Flash 8 to develop so this is AS2.0). We are going to host the files on Amazon S3 servers. Can anyone point out the best way to go about loading the files into Flash Player from Amazon S3. I have been using MovieClipLoader to load images ...

Can you play S3 signed url FLV videos in Flash 10?

I am wanting to load FLV videos from S3 server into my Flash application. The original files will need to be protected (ie permissions set to read only for authenticated users) so the video files will be called with a signed url... I have managed to load and play non-signed url FLV's into the Flash app. 1) Any issues I should be aware o...

Streaming MP3s from Amazon S3

Is there a way to stream MP3s stored on Amazon S3 via a Flash widget embedded in a website, or some other method? ...

What tools can I use to deploy PHP code to an EC2 instance

We can bundle all files into an Amazon Machine Instance and upload it. But I'd like to see if there is a more efficient way to regularly upload source code on to test our app and constantly have the latest version up and running. Thanks! ...

Why does Ruby open-uri's open return a StringIO in my unit test, but a FileIO in my controller?

I inherited a Rails 2.2.2 app that stores user-uploaded images on Amazon S3. The attachment_fu-based Photo model offers a rotate method that uses open-uri to retrieve the image from S3 and MiniMagick to perform the rotation. The rotate method contains this line to retrieve the image for use with MiniMagick: temp_image = MiniMagick::Im...

Find S3 Bucket Owner

If there is a button that I have read/write access on from my AWS account, is there any way to get the CanonicalUser id of the owner? Reading the ACL Policy seems to be impossible unless you are the owner : /. ...