How have you set-up one or more worker scripts for queue-oriented systems? How do you arrange to startup - and restart if necessary - worker scripts as required? (I'm thinking about such tools as init.d/, Ruby-based 'god', DJB's Daemontools, etc, etc) I'm developing an asynchronous queue/worker system, in this case using PHP & Beansta...
I'm close to releasing a rails app with the common networking features (messaging, wall, etc.). I want to use some kind of background processing (most likely Bj) for off-loading tasks from the request/response cycle. This would happen when users invite friends via email to join and for email notifications. I'm not sure if I should just...
With all things being equal whats currently the more mature PHP library for Amazon Simple Queue Service? I'm looking for good Object Oriented PHP 5 code. What were your experiences on the library you choice? Any performance problems? I'm building a system that needs to handle handing off messages to an unlimited amount of worker processe...
I'm wondering if anyone here has tried using Esper to process stuff from the Cloud? Specifically Amazon's Simple Queueing Service. I've googled about this, and we know it's feasible, but it would be nice to know if anyone has tried it. Even better has anyone tried putting Esper into the cloud as well? ...
I have a web app that sends messages to an Amazon SQS Queue. Amazon sqs lib throws a 'AmazonSQSException' since the message contained invalid binary character. The message is the referrer obtained from an incoming http request. This is what it looks like: http://ads.vrx.adbrite.com/adserver/display_iab_ads.php?sid=1220459&title_c...
Is there a good library/gem for accessing Amazon SQS from ruby1.9? The Amazon ruby example and right_aws do not work as-is with ruby1.9. I'd strongly prefer something that's known to work under reasonably heavy load (a few hundred thousand queue items or more per day). ...
What do people think are the most important issues when developing an application that is going to allow users to upload video and images to a server and have them transcoded by FFMPEG and stored in amazon S3? I have a couple of options; 1) install FFMPEG on the same server that handles file uploads, when a video is uploaded and stored ...
I am a newbie to Amazon SQS and ruby on rails. And i am working on a project that some XML messages must be send to SQS. How do i do that? Now i have this in the controller after the .save def create @thing = Thing.new(params[:thing]) respond_to do |format| if @thing.save message = @thing.to_xml and in the...
Hi, I am trying to insert characters like ä, á, ó, í, â, etc in message body of Amazon SQS. But these characters get converted to some other characters in SQS (e.g. ä). Is there a way to solve this problem. Thanks ...
I'm working on the design of a remote control application. From my iPhone or a web browser, I'll send a few commands. Soon my home computer will perform the commands and send back results. I know there are remote desktop apps, but I want something programmable, something simpler, and something that I wrote. My current direction is to...
I'm quite interested in beginning some development using Amazon SQS, perhaps SimpleDB too, my question is this, are there any open source solutions that mimic the functionality, just for the purposes of development. I've already encountered the Eucalyptus project (http://open.eucalyptus.com) for creating an EC-esque cloud. I've not had ...
I'm trying to use the right_aws (1.10.0) GEM with Rails, and I've reduced my problem to a 3-line irb session. The following works require 'rubygems' require 'right_aws' sqs = RightAws::Sqs.new("xxxxxxxxxxxxxxx", "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx") while this fails require 'rubygems' require 'right_aws' sqs = RightAws::SqsGen2.new("xxxx...
I'm designing a service for sending out emails for our eCommerce site (order confirmations, alerts etc...) The plan is to have a "SendEmail" method, that generates a chunk of XML representing the email to be sent, and sticks it on an Amazon SQS queue. My web app(s) and other applications will use this to "send" emails. I then require a...
I'm currently developing an application in C# that uses Amazon SQS The size limit for a message is 8kb. I have a method that is something like: public void QueueMessage(string message) Within this method, I'd like to first of all, compress the message (most messages are passed in as json, so are already fairly small) If the compress...
Currently I have a perl script that runs forever on my server, checking a SQS for data to compute. This script has been running for about 6 months with no problems what so ever. So, now I want to switch over to PHP'S CLI, and have the script loop there forever. Mostly because I'm more familiar with PHP. Basicly, $i="forever"; while($i...
The Amazon AWS SQS WSDL (at https://sqs.us-east-1.amazonaws.com/doc/2009-02-01/QueueService.wsdl) lists multiple "ports" with different "addresses" (near the bottom of the file), which specify the HTTP and HTTPS addresses for the service. Using the PHP SOAP object: $aws_ns = 'http://security.amazonaws.com/doc/2007-01-01/'; $sc = new So...
If I create SQS queue from an EC2 instance without specifying the region in the API call, in which region will the queue be created. When I run boto.sqs.regions() I get 4 regions from an NON ec2 machine, I get [RegionInfo:us-east-1, RegionInfo:eu-west-1, RegionInfo:us-west-1, RegionInfo:ap-southeast-1] from a EC2 machine in the As...
See also http://stackoverflow.com/questions/2977105/having-a-php-script-loop-forever-doing-computing-jobs-from-a-queue-system/, but that doesn't answer all my questions. If I want to run a PHP script forever, accessing a queue and doing jobs: What is the potential for memory problems? How to avoid them? (any flush functions or somethi...
I am trying to build a twitter-like follow mechanism. User takes an action. We make a list of all that users' followers, and then populate all of their streams with some information. Since this can take some time (if you have 10,000 followers that's 10,000 streams to insert information in, ie. 10,000 SQL calls perhaps), I want to make su...
Hello all, I am trying to use EC2 to zip up some files that are stored in an S3 bucket. I have gotten as far as successfully getting SWFUpload to work with PHP and upload the files to S3. I read that the best way to zip up S3 files without incurring huge transfer costs is to use EC2 to deal with S3. After a lot of effort I managed to ge...