views:

481

answers:

4

Reference another SO question I had, I was given this article about Twitter moving from Rails to Scala, and in the article is this comment:

By the end of this year, Payne said, Twitter hopes to have its entire middleware infrastructure and its APIs ported to the new language. Ruby will remain, but only on the front end. "We're still happy with Rails for building user facing features... performance-wise, it's fine for people clicking around web pages. It's the heavy lifting, asynchronous processing type of stuff that we've moved away from."

What does middleware mean here? What exactly does it meant to have Ruby on the front end? Does that mean rails on the front end with very little ORM being used? How does it "meet up" with scala? What does this comment mean?

Just trying to understand the architecture. Thank you.

A: 

Messaging infrastructure in this case. They probably switched from some custom ruby based solution to the JMS one.

Dev er dev
JMS? Sorry. I'm lost in my acronyms here.
johnny
+3  A: 

It is a form of N-Tier Architecture. Initially, most sites wills start out as a 2 Tier architecture, having a webserver and a database. The webserver will serve up the pages that the user sees, while the database is accessed by those dynamic pages.

When an architecture is separated into multiple tiers, usually, you will have a Web Application at the front, with some kind of application server which the web application will call. This application server contains the business logic and makes the application functional. The "face" for the application simply makes things look pretty and formats it for display to users.

The application server, or "middleware", is simply a collection of functions that can be called by the web servers. The middleware must be powerful for a site like twitter where most of the activity is in sending messages, and there is not so much activity from people accessing the website.

The interconnect between layers is likely a standard web service technology, but it could be a custom thing, perhaps a REST or some other kind of web service where the web application(Ruby on Rails) can access application data.

In this type of architecture, Rails most certainly has relegated all ORM and data management to the middleware.

Middleware is also a vague term and can consist of many Tiers, hence why they call it N-Tier. I would bet that Twitter has a different part of their middleware devoted to messaging while another part deals with managing accounts, fetching messages, etc.

The idea is to be able to scale upwards by adding hardware, where you can have a cluster of servers serving up web pages, another cluster handling messaging, and then a cluster of database servers that support it all. It's not an exact science, each architecture is different, but generally, this is how it can be thought of.

Kekoa
+1  A: 

Try Googling for "ruby twitter scala payne" and you'll get more than you probably need on the subject.

My (probably incomplete) understanding is that much of the Twitter "back-end" not coded particularly well (see Obie Fernandez' post that picks out some Alex Payne quotes). This appears at least in part to have stemmed from the initial quick-hack nature of the app, which was overtaken by events, they were experiencing considerable pain as use of the app started to grow exponentially. There were any number of fingers pointed at Twitter (and by association Rails) for "failing to scale" at the time.

Twitter relies heavily on message-queueing and it's this part that I understand to have been written in Scala, which is probably (no personal experience) a very good choice of language for the job.

Mike Woodhouse
+1  A: 

My understanding is that what they migrated to Scala is the message "queue": from a Ruby-ORM-DB backed solution to a distributed messaging system with Scala. A service with the volume of Twitter has special requirements and they've created their own message solution. Is called "Kestrel"

GClaramunt