What is a realistic measurement to load test a CMS web application?

Question

Our company is facing some difficulties with our CMS web application. This application was part-built by a contractor and we have been confronting some stability issues (crashing, having to put them in front of load balancers or caching mechanisms) when we think the application should be able to handle it. We put together a minimal standard measurement, but we don't know if these metrics are realistic.

We were hoping to get in this forum feedback on what is a realistic expectation of a CMS system should handle independent of the technology that was built. So if the same application was to be built in .NET instead of Java (current) you will expect to perform the same.

The metrics that we came up with are:

• Number of concurrent requests/queue length: 100 Maximum
• Time to serve a request: 2 s Minimum
• Number of requests per hour: 150,000
• Minimum number of page views per hour: 5,000

Minimum HD Requirements: - 2 GB Ram - 2 Dual-Core 2.0 Ghz

General Functionality:

• Dynamic Cross Referencing (People to News,Events to People and News, Technical Cases, Etc)
• Advanced Search Features
• Highly Configurable without programming

Answer 1

It's not reasonable to make concrete performance & scalability expectations without any information about hardware, technology, load, usage, etc. "CMS" is very broad:

• What does your server farm look like?
• What are the terms of your SLA?
• What does your typical user look like? E.g. many brief users or fewer users with long sessions and many requests?

Other important questions to answer:

• Do you want to measure "time to first byte" (I hate this but it's fairly common), or include network latency in your total "time to serve"?
• How many editors are working against the system?
• Do your editors work against the same data backing, or do they prepare content in an isolated environment and push batches of content updates?
• What kind of caching mechanism can you support? Can the content be stale for minutes/hours?

In our farm of several load-balanced 64-bit servers with ~32gb RAM (IIRC) and 4 CPUs each, we average just under 100k requests per hour with a peak load of several hundred requests/sec (uncommon). Total end-user load time (incl images and assets) must be under 5 sec. Our total CMS content database is just under 750,000 pages. We have massive amounts of cross-loaded content, querying, complex editor-configurable widgets, etc.

Answer 2

What you should start doing is tracking how people are using it. If you're keeping weblogs, you'll have requests over time and number of sessions active etc.

You should then use those metrics to base what your load/stress testing should look like. How many users at a time, load test it at peak transaction levels - zeroing out for reads vs. writes.

Answer 3

It's very hard to answer your question. You mentioned your server's hardware, but it was not clear to me if this hardware is used to the database server or to the application server (or if you are using the same hardware for both of them). And you have not supplied us with information regarding your database...

It's even harder to help without knowing your application architecture. Some architectural decisions, like database caching, advanced front-end optimization techniques, database tuning, full-text index, asynchronous processing (via Message Queues) can make a huge difference in application performance and scalability.

Anyway, I recommend you to use some load test tools like JMeter, Silk Peformer, Load Runner, etc.

Answer 4

Our company uses load runner with 3 different sets of scripts. 1 script is to mimic a basic use case for the web site, main-page, log-in, navigation to subject area etc.... the second script is to do a random site search based on 100 possible search terms. and the 3rd test is for a use case of a response to a mass e-mail to our subscribers.

each script we repeat the action 20 times.

We run the 1st script at 100 concurrent users starting with 1 user increasing by 5 every 30 seconds and we let that run and we pull the statistics.

We run the 2nd script at 20 concurrent users starting with 1 user increasing by 2 every 30 seconds

We run the 3rd script at 500 concurrent users starting with 1 and increasing by 10 every 5 seconds.

The important part is that we run the same load test and track / analyze the results with each change to the infrastructure.

Load runner tracks the response time, failure rates and a few other statistics... with remote agents it can track the database and server vitals as well.

Answer 5

Performances issues are rarely straightforward. Frankly speaking, you should really take the services of skilled integrator knowing BOTH the CMS you are using and performances optimization. It is always tricky to find out the bottlenecks between network / the CMS itself / its implementation.

Your specifications look to me quite "standard", nothing expectational here, so I suspect more the implementation or network issues than directly the CMS (I can't say it is not the CMS, I want to say, that would be surprising... and disappointing. If it is a well known close source vendor, a famous open source one, your chances that the CMS is not the cause are moving up quickly).