It's a CentOS server (I don't know the specs) and just before anybody states the obvious, keep in mind these mitigating factors:
- the server does a nightly VACUUM job
- all the tables are indexed
- it's pretty much read only (meaning the DBs are not increasing in size)
- the number of queries being ran has been the same every month
Here's a graph of the server load:
UPDATE (2010/9/15): We never found a server or config cause to this, but our application was optimized and the load is back to acceptable levels at the same usage:
- Cached some data
- Found some queries that were rewritten
- Found some redundant queries