I can't really read VB.NET code, since it makes my eyes bleed, but if I understand your code correctly, you're doing the font initialization as a member initializer. This is run during construction time, but the XML deserialization happens, basically, afterwards: The object is created (that's why it needs a parameterless constructor), and then the properties that are found in the XML are being set. So, basically, you're reading the properties too early.
My solution to this was to have an actual Font property with a getter that creates the font from these properties. This way, the actual font object is created when you need it (when you get the property), and that is typically after deserialization. It's still up to you whether you just return a new font object every time in the getter, or cache it for subsequent calls. I prefer the former, and then immediately dispose of the font object after use.
Note: The syntax for code on StackOverflow is not BBCode, but just indenting everything with four spaces. Select the code, and then press the button with the zeros and ones on it. I fixed it for this post.