Twitter Streaming API recording and Processing using Windows Azure and F#

Question

Hello,

A month ago I tried to use F# agents to process and record Twitter StreamingAPI Data here. As a little exercise I am trying to transfer the code to Windows Azure.

So far I have two roles:

• One worker role (Publisher) that puts messages (a message being the json of a tweet) to a queue.

• One worker role (Processor) that reads messages from the queue, decodes the json and dumps the data into a cloud table.

Which leads to lots of questions:

• Is it okay to think of a worker role as an agent ?
• In practice the message can be larger than 8 KB so I am going to need to use a blob storage and pass as message the reference to the blob (or is there another way?), will that impact performance ?
• Is it correct to say that if needed I can increase the number of instances of the Processor worker role, and the queue will magically be processed faster ?

Sorry for pounding all these questions, hope you don't mind,

Thanks a lot!

Answer 1

Is it okay to think of a worker role as an agent ?

This is the perfect way to think of it. Imagine the workers at McDonald's. Each worker has certain tasks and they communicate with each other via messages (spoken).

In practice the message can be larger than 8 KB so I am going to need to use a blob storage and pass as message the reference to the blob (or is there another way?), will that impact performance?

As long as the message is immutable this is the best way to do it. Strings can be very large and thus are allocated to the heap. Since they are immutable passing around references is not an issue.

Is it correct to say that if needed I can increase the number of instances of the Processor worker role, and the queue will magically be processed faster?

You need to look at what your process is doing and decide if it is IO bound or CPU bound. Typically IO bound processes will have an increase in performance by adding more agents. If you are using the ThreadPool for your agents the work will be balanced quite well even for CPU bound processes but you will hit a limit. That being said don't be afraid to mess around with your architecture and MEASURE the results of each run. This is the best way to balance the amount of agents to use.

Answer 2

Is it okay to think of a worker role as an agent?

Yes, definitely.

In practice the message can be larger than 8 KB so I am going to need to use a blob storage and pass as message the reference to the blob (or is there another way?), will that impact performance ?

Yes, using the technique you're talking about (saving the JSON to blob storage with a name of "JSONMessage-1" and then sending a message to a queue with contents of "JSONMessage-1") seems to be the standard way of passing messages in Azure that are bigger than 8KB. As you're making 4 calls to Azure storage rather than 2 (1 to get the queue message, 1 to get the blob contents, 1 to delete from the queue, 1 to delete the blob) it will be slower. Will it be noticeably slower? Probably not. If a good number of messages are going to be smaller than 8KB when Base64 encoded (this is a gotcha in the StorageClient library) then you can put in some logic to determine how to send it.

Is it correct to say that if needed I can increase the number of instances of the Processor worker role, and the queue will magically be processed faster ?

As long as you've written your worker role so that it's self contained and the instances don't get in each others way, then yes, increasing the instance count will increase the through put. If you're role is mainly just reading and writing to storage, you might benefit by multi-threading the worker role first, before increasing the instance count which will save money.