Why is there no improvement in CPU usage with our 64-bit build of Apache?

Question

G'day,

We have built a 64bit build of Apache 2.2.14 and deployed it across various Sun servers running Sol10. Server types used for testing are Sun T2000's (32GB memory), 5120's (8GB) and 5240's (16GB).

For each of these we have noticed that there was no appreciable improvement in CPU usage and in fact the server is running slightly hotter than when running the 32bit version.

We've have confirmed this observation by rolling back to the 32bit version and then rolling forward again to the 64bit build. As this is a major web farm we have an extensive server monitoring infrastructure with RRD graphs for all sorts of measurements. Graphs are available for daily, weekly, monthly and yearly timeframes. Across these graphs the slight step change in the CPU usage is quite noticeable,

Hit rates on our servers are cyclic throughout the day but remain fairly stable with peaks in the order of 2k/second per server. Number of children is also cyclic on a daily basis and varies between 1k and 3k per server. As all servers sit behind a load balance layer, the step change cannot be attributed to variations in hit rate or children.

We used the CC compiler that ships with Sunstudio 12.1 using the following compile flags. Link flags are similar but without the -D defines.

export CFLAGS="-xopenmp=parallel -xalias_level=basic \
    -xtarget=ultraT1 -m64 -xarch=sparc -xbuiltin=%all \
    -xdepend -xmemalign=8s -xO5 -xprefetch=auto,explicit" \
    -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64

We are not doing anything unusual in the Apache config and it's pretty vanilla.

Edit: Performance was determined looking at CPU vs. Requests vs. Memory (in that order). The 64bit was doing fewer requests with same CPU and slightly hiher memory footprint. The diff was within error margins and only got noticable at close to full load

I've had a read of the question "How to develop to take advantage of 64 bit systems?" and there is some interesting info there. It seems to be mainly an issues concerning

1. addressing memory outside the 4GB limit of the 32bit address space,
2. manipulation of 64bit values.

To my mind we would also expect an improvement in SSL performance.

Any suggestions as to why our performance hasn't improved?

Or suggestions on what to look at?

Should we in fact actually see a performance improvement?

Answer 1

Do you use 64bit optimized algorithms? Is your SSL library optimized for 64bit and the ssl library compiled towards that architecture? Just compiling 32bit code in 64bit won't give much bang because most 32bit code does not even use optimizable features.

Answer 2

Apart from dealing with large amounts of memory the only other advantage of x86_64 compared to it's 32bit cousin is a richer set of registers. This should mean CPU heavy code can be more efficient as it has to maintain less state in relatively slow main memory. This however is often countered by increased memory usage as all pointers now take twice the space.

Generally given enough memory to account for this I would expect CPU bound problems to perform better as 64 bit applications. However web-serving is often an I/O bound problem and that won't be helped (and possibly hindered if the machine is under extra memory pressure).

You probably want to profile you machine (oprofile on Linux, not sure what the equivalent Solaris tool is) and see where the CPU hot spots actually are.