How do I benchmark and profile IO bound Perl web application under heavy load?

Question

How do I benchmark (compare two different implementations) and profile (find performance bottlenecks in an implementation) behavior of a web application (in Perl) under heavy load? This webapp is (supposedly) IO bound rather than CPU bound.

I'd like to compare proposed implementation (proposed improvement) with current solution, either on real load but minimizing eventual performance hit (perhaps two different implementations on different servers behind load balancer), or replay / simulate load on a new implementation, or simulate heavy IO usage.

I'd like also profile CPU and IO bottlenecks in proposed implemetation, but testing behavior under heavy load.

The OS is Linux, web server is Apache.

---

The same question is posted on ServerFault: How do I benchmark and profile IO bound Perl web application under heavy load?

I think the answer here can be about benchmarking and profiling web application written in Perl, and perhaps also about replicating realistic load conditions, or finding measure / benchmark which would predict which implementation would perform better under IO bound conditions under heavy load.

Answer 1

You could create some tests using the Benchmark module. It has functions specifically for comparing two implementations of the same functionality. While it doesn't quite handle simulating load levels for you, it would make it a lot easier to do.

Answer 2

I'd suggest you to use Munin ( http://munin-monitoring.org/ ).

It has tons of plugins to see how a system behaves: cpu, i/o, memory, network usage, mysql transactions, ...almost everything.