Going to the admin/settings/performance page, turning on CSS and JS aggregation, and page caching with a minimum lifetime of 1 minute, will give you an immediate boost on a high traffic site. If you're writing your own code and doing any queries, consider writing your own discrete caching for expensive functions. The linked article covers Drupal 5, not 6, but the only change in d6 is the elimiation of the serialization requirement and the function signature for the cache_set() and cache_get() functions. (Both noted in comments on the article)
On large scale sites also consider dropping a memcached server onto the network: Using the memcached module, you can entirely bypass the drupal database for cached data. If you have huge amounts of content and search is a hot spot, you can also consider using lucene/solr as your search indexer instead of drupal's built-in search indexer. It's nice for a built-in indexer but it's not designed for heavy loads (hundreds or thousands of new pieces of content an hour, say, with heavy faceted searching). The apache solr module can tie in with that.
If you're making heavy use of Views, be sure that you've checked the queries it generates for unindexed fields; sorting and filtering by CCK fields in particular can be slow, because CCK doesn't automatically add indexes beyond the primary keys. In D6, preview the View in the admin screen, copy the text of the query, and run it through EXPLAIN in mysql or whatever query analysis tools you have.
Tools like YSlow and Firebug can also help you spot slow stuff like massive image files, JS hosted on remote servers, and so on.