Yes you can create message classes in Exchange. You use the organsisational forms library.
I believe you can do this with webdav if you need to, essentially you still design the form in outlook and save it. Then use webdav to push the form template to the org library of exchange.
http://www.outlookcode.com/article.aspx?ID=35
Yes you can create a custom message class. as previous poster said, you just create a form in outlook (in design mode) and then use the Publish command to publish the form to the outlook forms library (or organisational forms library - exchange).
http://office.microsoft.com/en-au/outlook/HA012106101033.aspx
In terms of VSTO you don't use this to directly register the class, this is done for you when you use the Publish command from the forms designer. VSTO is just used to write the code that creates the Inspector window using your custom message class.
http://support.microsoft.com/kb/310244
see code example below..
Outlook.Application olApp = new Outlook.Application();
//mapifolder for earlier versions (such as ol 2003)
Outlook.Folder contacts = olApp.Session.GetDefaultFolder(Outlook.olDefaultFolders.olFolderContacts);
//must start with IPM. & must be derived from a base item type, in this case contactItem.
Outlook.ContactItem itm = (Outlook.ContactItem)contacts.Items.Add(@"IPM.Contact.CustomMessageClass");
itm.Display(false);