I would second fredley's suggestion to read up on Drupal Caching speed and performance on the link he has provided. However a quick way to optimize your site would be to use the boost module. ( http://drupal.org/project/boost ). Boost caches the HTML output of each URL (when accessed). Subsequent access to the same URL, causes the cached HTML to be sent and so PHP scripts don't need to run and consequently, neither is the database accessed. Your Apache server is blindingly fast at serving HTML pages so your server load goes down and responsiveness goes up. Boost is very intelligent e.g. if someone comments on a node it will expire that page and so on.
Please note that Boost only makes sense if your site is for anonymous users. For logged in users you will need to employ other strategies e.g. memcache etc.
Please also make sure simple things in the performance settings of your drupal site like CSS aggregation ("CSS optimization") and Javascript aggreation ("JS optimzation"), default caching options etc are enabled.
There is a lot of information available on performance optimzation so it can be overwhelming at first. If you're a book kind of person then check out https://www.packtpub.com/drupal-6-performance-tips-to-maximize-and-optimize-your-framework/book