What tools or strategies are you using for automation of EC2 activities? I need to be able to bring up a number of EC2 instances, provision various software to it (primarily Python packages), interact with S3 (primarily download data), and run various jobs. I'll be doing this both on-demand and on a scheduled basis. I'm trying to decid...
I'm trying to jury-rig the Amazon S3 python library to allow chunked handling of large files. Right now it does a "self.body = http_response.read()", so if you have a 3G file you're going to read the entire thing into memory before getting any control over it. My current approach is to try to keep the interface for the library the same ...
I'm thinking about serving user-specific static content from S3 - the user needs to be authenticated in order to access his static content. So if user A has content c1, c2 and use B has c3, c4, only A should be able to access c1, c2 . What's a good way to accomplish this? Is there a way to perform per-user / per file authentication in S...
Hey all, I'm currently trying to code a custom image cropping system similar to other ones on the internet where a user can select a cropping area and then have their image cropped accordingly. The application is in Rails and we're using Paperclip with Amazon S3 to store the files. I'm having a lot of trouble though getting RMagick to ...
What information have you been able to gather regarding how do the amazon web services work? What hardware do they use What web server What Operating System What storage for AWS What virtualization software for EC2/EBS What software for they distributed firewall for EC2 Physical location of their data centers. I like their services v...
Just want to confirm that there is no way to extend or replace Html.Image functionality without writing a replacement function. I want to write a function that will use Amazon's S3 service for hosting images. The best approach I've come up with is a helper method Html.SmartImage which would check a configuration property to see if i wa...
I am starting to use Jungle Disk to upload files to an Amazon S3 bucket which corresponds to a Cloudfront distribution. i.e. I can access it via an http:// URL and I am using Amazon as a CDN. The problem I am facing is that Jungle Disk doesn't set 'read' permissions on the files so when I go to the corresponding URL in a browser I get ...
I am using Amazon S3 and certain images are hosted there. Its kind of a pain to check each image in turn to verify its coming from the CDN and sometimes new images are added and someone forgets to upload them, and I thought it would be nice to have a visual cue - accessible from a debug panel. I'd like to draw a red border on all images...
I have a job processing architecture based on AWS that requires EC2 instances query S3 and SQS. In order for running instances to have access to the API the credentials are sent as user data (-f) in the form of a base64 encoded shell script. For example: $ cat ec2.sh ... export AWS_ACCOUNT_NUMBER='1111-1111-1111' export AWS_ACCESS_KEY_...
I have a website that attracts about 30,000 visitors per month. It has a lot of photos and PDF files which eat up a good deal of bandwidth. It's hosted by site5.com, which offers unlimited bandwidth & storage for ~$5 per month. According to site5's statistics, my site has about 20 GB of downloads per day, but I've seen it as high as 116...
I recently refactored some of my code to stuff rows into a db using 'load data' and it works great -- however for each record I have I must upload 2 files to s3 -- this totally destroys the magnificent speed upgrade that I was obtaining. Whereas I was able to process 600+ of these documents/second they are now trickling in at 1/second be...
I am using this file storage engine to store files to Amazon S3 when they are uploaded: http://code.welldev.org/django-storages/wiki/Home It takes quite a long time to upload because the file must first be uploaded from client to web server, and then web server to Amazon S3 before a response is returned to the client. I would like to ...
I'm hoping to reach someone with some experience using a service like Amazon's S3 with this question. On my site we have a dedicated image server. And on this server, we have an automatic 404 redirect through Apapche so that, if a user tries to access an image that doesn't exist, they'll see a snazzy "Image Not Available" image. We're...
I'm planning on making my (family) photo collection available online. I want to use S3 and build an ASP.NET site that will display the photos. I don't want the website to pull down the S3 content and return it to the browser. I want browsers to be able to go directly to S3 without affecting my ASP.NET bandwidth. It is possible to build ...
Hi, I am developing an application that loads images and video into a Flash player (currently using Flash 8 to develop so this is AS2.0). We are going to host the files on Amazon S3 servers. Can anyone point out the best way to go about loading the files into Flash Player from Amazon S3. I have been using MovieClipLoader to load images ...
I am wanting to load FLV videos from S3 server into my Flash application. The original files will need to be protected (ie permissions set to read only for authenticated users) so the video files will be called with a signed url... I have managed to load and play non-signed url FLV's into the Flash app. 1) Any issues I should be aware o...
Is there a way to stream MP3s stored on Amazon S3 via a Flash widget embedded in a website, or some other method? ...
We can bundle all files into an Amazon Machine Instance and upload it. But I'd like to see if there is a more efficient way to regularly upload source code on to test our app and constantly have the latest version up and running. Thanks! ...
I inherited a Rails 2.2.2 app that stores user-uploaded images on Amazon S3. The attachment_fu-based Photo model offers a rotate method that uses open-uri to retrieve the image from S3 and MiniMagick to perform the rotation. The rotate method contains this line to retrieve the image for use with MiniMagick: temp_image = MiniMagick::Im...
If there is a button that I have read/write access on from my AWS account, is there any way to get the CanonicalUser id of the owner? Reading the ACL Policy seems to be impossible unless you are the owner : /. ...