A) What's the memory you're allowing PHP to use? Look in php.ini and see.
B) How many modules do you have enabled, and how many of them do you really need enabled? Each module you have turned on slows things down a little.
C) Is there a block on every page that's slow? What happens if you turn that block off? Can you optimize this block or replace it with something similar but faster?
D) What are you doing on every page load? Are you using custom modules? Are they doing something in hook_init?
E) Look at the database. Is it slow because you've got too much stuff going on?
F) Have you tried enabling Memcache or something else that helps with caching speed? Can you turn on APC?
G) Do you have a node type with a rediculous number of CCK fields that gets loaded on every page load somehow?
H) Is your theme doing something it shouldn't? Is it custom?
It's hard to say what may be causing your problem, but all of these are things that I'd look into.