I recommend that you not preoptimize. If you have your code working, then use it as it is. Go on to work on some code that is not finished, or which does not work.
Later, if you find you have a performance problem in that area, you can explore performance.
Brett,
Later versions of .net will build custom serializer assemblies. Click on project properties -> build and look for "Generate serialization assemblies" and change to On. The XML deserializer will use these assemblies which are customized to the classes in your project. They are much faster and less resource intensive since reflection is not involved.
I would go this route so that if you class changes you will not have to worry about serialization issues. Performance should not be an issue.