I am currently running too many sites on a server and I don't think the template cache can handle it. But, what really seems to be the biggest drag is when I load my UDF library per site. I say this because whenever I run Fusion Reactor to see where the holdup is, the stacktrace is always sitting on the template that loads the UDF's.
Is the only cure for this more RAM and a higher template cache, or is there a better way?
Maybe I am wrong as well, could there be another issue?
Before increasing the heap and template cache available, look at a few things.
First, do you actually have more templates in the system than you have template cache? If not, increasing it certainly won't help. Even if you do, if they aren't called often, it probably won't help, but that's harder to measure.
Second, examine whether the server is having difficulty actually loading the UDFs, or if the page is having a problem executing a UDF. Are the functions included on the same template that calls them?
Third, find out why it take so long to load this UDF library. Is it really that big? Can it be split into smaller libraries? Is there one (or more) particular UDF that seems to hang the compile process?
Finally, if there is a large UDF library that must be loaded for each request, I would look at using the Application scope to store it. Include the librar onApplicationStart(), then reference functions as application.myFunction(). This prevents CF from needing to load (and possibly compile) the file at each request.