The first step is to establish whether the problem is client-side or server-side.
A good YSlow grade indicates the problem probably isn't clientside. YSlow checks to see that you don't have too many objects on the page, that you have minified your javascript/CSS etc. It does not evaluate the performance of your network or server.
Using YSlow/Firebug, check to see how long it takes to load the actual HTML of your page. If that is taking a long time, then the problem is almost certainly with your server, network or server-side code.
To rule out network issues, compare accessing your site from the server itself to accessing it over the internet. If it's a lot slower over the internet the problem could be network-related.
If it's not client-side or network-related, then it's either that your server is struggling for resources or that your code is slow (perhaps because the amount of data it is mananaging has grown).In that case, check the server logs and run a profiler on your code (on a development server but with a copy of production data).