ASP.NET Templating

Question

We're building a text templating engine out of a custom HttpModule that replaces tags in our HTML with whole sections of text from an XML file.

Currently the XML file is loaded into memory as a string/string Dictionary so that the lookup/replace done by the HttpModule can be performed very quickly using a regex.

We're looking to expand the use of this though to incorperate larger and larger sections of replaced text and I'm concerned over keeping more verbose text in memory at one time as part of the Dictionary, especially as we use ASP.NET caching for many uses as well.

Does anyone have a suggestion for a more efficient and scalable data structure/management strategy that we could use?

UPDATE: In response to Corbin March's great suggestion below, I don't think this is a case of us going down a 'dark road' (although I appreciate the concern). Our application is designed to be reskinned completely for different clients, right down to text anywhere on the page - including the ability to have multiple languages. The technique I've described has proven to be the most flexible way to handle this.

Answer 1

The amount of memory you are using is going to be roughly the same as the size of the file. The XML will have some overhead in the tags that the Dictionary will not, so it's a safe estimate of the memory requirements. So are you talking about 10-50 MB or 100-500 MB? I wouldn't necessarily worry about 10 to 50 MB.

If you are concerned, then you need to think about if you really need to do the replacements everytime the page is loaded. Can you take the hit of going to a database or the XML file once per page and then caching the output of the ASP.NET page and hold it for an hour? If so, consider using Page Caching.

Answer 2

A couple ideas:

Compress your dictionary values. Check out Scott Hanselman's cache compressing article to get the spirit of the exercise. If your dictionary keys are large, consider compressing those as well.

Only load items from your XML file into memory when they're requested and attach an item expiration. If the expiration occurs without another request, unload the item. The idea is that some dictionary items are used less frequently so an IO hit for the infrequent items is acceptable. Obviously, the ASP.NET Cache does this for you - I'm assuming Cache is out-of-context by the time you're crunching your output.

Just an opinion... but my spidersense warns me you may be going down a dark road. A huge part of ASP.NET is its templating features - master pages, page templates, user controls, custom controls, templated controls, resource schemes for internationalization. If at all possible, I'd try to solve your problem with these tools versus a text-crunching HttpModule.