scalability

Rails Request Initialization

We all hear a lot about scaling issues in Rails. I was just curious what the actual costs in handling a HTTP request is in the Rails framework. Meaning, what has to happen for each and every request which comes in? Is there class parsing? Configuration? Database Connection establishment? ...

What is the best MemCache solution for ASP.NET applications?

What is the best MemCache solution for ASP.NET applications running in a windows server environment? Why? ...

Fast Text Search Over Logs

Here's the problem I'm having, I've got a set of logs that can grow fairly quickly. They're split into individual files every day, and the files can easily grow up to a gig in size. To help keep the size down, entries older than 30 days or so are cleared out. The problem is when I want to search these files for a certain string. Right n...

What is considered a good response time for a dynamic, personalized web application?

For a complex web application that includes dynamic content and personalization, what is a good response time from the server (so excluding network latency and browser rendering time)? I'm thinking about sites like Facebook, Amazon, MyYahoo, etc. A related question is what is a good response time for a backend service? ...

Experiences using software load balancing vs. a hardware load balancer?

The ASP.NET application that I am currently responsible for at my day job has hit its limit in terms of its ability to scale inside a single server. Obviously we are working toward moving session out of process and the test and hopefully deploy date draws near. I would like to draw on the experiencies of people using the built in load ba...

Pro's of databases like BigTable, SimpleDB

New school datastore paradigms like Google BigTable and Amazon SimpleDB are specifically designed for scalability, among other things. Basically, disallowing joins and denormalization are the ways this is being accomplished. In this topic, however, the consensus seems to be that joins on large tables don't necessarilly have to be too ex...

Is there some industry standard for unacceptable webapp response time?

There's a cots(commercial off-the-shelf) application that I work on customizing, where a couple of pages take an extremely long time to load for certain distributions of data. (I'm talking approximately 3 minutes for a page to load in this instance... and the time is growing exponentially). Clearly this is unacceptable but are there st...

What slows down growing database performance?

Hi all, I'm creating a database, and prototyping and benchmarking first. I am using H2, an open-source, commercially free, embeddable, relational, java database. I am not currently indexing on any column. After the database grew to about 5GB, its batch write speed doubled (the rate of writing was slowed 2x the original rate). I was ...

WCF Service & Request queueing

Is using a handrolled POCO queue class using pseudo code T Dequeue() { lock(syncRoot) { if(queue.Empty) Thread.Wait(); } } void Enqueue(T item) { queue.Enqueue(item); Thread.Notify(); } for WCF request queueing a scalable approach? ...

What puts less load on a PHP server: SimpleXML or json_decode?

I'm starting to develop a web application in PHP that I hope will become incredibly popular and make me famous and rich. :-) If that time comes, my decision whether to parse the API's data as XML with SimpleXML or to use json_decode could make a difference in the app's scalability. Does anyone know which of these approaches is more eff...

How scalable is LINQ?

Recent conversations with colleagues have produced varying points of view on this matter. What say you, SO members? I know, even the concept of scalability can be taken in so many different ways and contexts, but that was part of the discussion when this came up. Everyone seemed to have a different take on what scalability really means....

Sharding (sic!) the web tier in order to prevent a load balancer bottleneck?

How do large web sites which cannot be completely stateless achieve extreme scalability at the web tier? There are sites like ebay and Amazon, which cannot be completely stateless, as they have a shopping cart or something like that. It isn't feasible to encode every item in the shopping cart into the URL, nor is it feasible to encode e...

Database scalability - performance vs. database size

Hi all, I'm creating an app that will have to put at max 32 GB of data into my database. I am using B-tree indexing because the reads will have range queries (like from 0 < time < 1hr). At the beginning (database size = 0GB), I will get 60 and 70 writes per millisecond. After say 5GB, the three databases I've tested (H2, berkeley DB,...

Scalability on the web

I've been arguing with some friends on the university, and we can't get to a point to which is the framework with more scalability for web applications (and still very fast). One calls for jsp, the other one for ruby other for php and so on. Could i ask you to clarify us on what as more scalability potential? Tks, hope I'm not duplicat...

Is there a way to prioritize web services in ASP.NET

The goal is having a way to guarantee a certain SLA (Service Level Agreement) for some of the web services to guarantee backend database uptime. Ideally this should be achieved with no specific code in the services but by controlling the underlying infrastructure/pipeline. ...

Design pattern or code smell, denormalised data as a result of functional decomposition

I'm a big fan of http://highscalability.com/ and have been looking in my current development to decompose my application along functional boundaries as a route to being able to scale out the server side, specifically the database layer. What this involves is implementing different functional components of the application (we have several...

Searching across shards?

Short version If I split my users into shards, how do I offer a "user search"? Obviously, I don't want every search to hit every shard. Long version By shard, I mean have multiple databases where each contains a fraction of the total data. For (a naive) example, the databases UserA, UserB, etc. might contain users whose names begin ...

Best practices for web service timeouts

It there any article/book that defines upper bounded design limits for WS timeouts? Do you timeout at the server or recommend the client specific timeouts too? Is there a common best practice like "never design WS that can take longer than 60 seconds, use an asynchronous token pattern" I am interested in knowing what you do or your opi...

Resources for Database Sharding and Partitioning

I'm working with a database schema that is running into scalability issues. One of the tables in the schema has grown to around 10 million rows, and I am exploring sharding and partitioning options to allow this schema to scale to much larger datasets (say, 1 billion to 100 billion rows). Our application must also be deployable onto se...

What are the key factors that ensure successful ASP.NET application scalability?

When starting a new ASP.NET application, with the knowledge that at some point in the future it must scale, what are the most important design decisions that will allow future scalability without wholsesale refactoring? ...