You could abstract the message deserialization. Have a "MessageHolder" class that just has the buffer to the object initially. It would have a method:
IMessageInterface NarrowToInterface(MessageId id);
It wasn't clear to me if your router would already know what type of message it was or not. If it does, then it would receive the messageholder instance and call the NarrowToInterface method on it.
It would pass the id of the appropriate type. If the router didn't know what type it was, then you'd also have a property on the MessageHolder object:
MessageId GetMessageType();
that the router would use to learn what message type it was to decide where to route it. More on how that is implemented later.
The IMessageInterface is an abstract class or interface that the recipient of the message would down-cast to the appropriate type, since it would know what type to expect. If all of the different messages are well-known and you have generics or templates available to you, you could have the NarrowToInterface method be a template method that took the return value as a template parameter, so that you would have better type safety. If you don't have templates, you could use the double-dispatch technique of the "Vistor" pattern. Google "double-dispatch visitor" for more info.
If the types of messages is not well-defined or could grow in the future, you'll just have to live with a (compiler-unverifiable) downcast at some point. The implementation I'm suggesting encapsulates this as much as possible and limits coupling to its absolute minimum, as far as I know.
Also, for this to work your messages have to be framed with a standard identifier in the header. i.e. there is a standard header that has the length of the entire message as well as the ID of the message type. That way the socket endpoint can parse the basics of the message and put it into the messageholder. The MessageHolder can either know about all the different messages types itself to implement the NarrowToInterface() method or there could be a global repository that would return an "IMessageDeserializer" objects to implement NarrowToInterface for each message Type. All of the loaded message clients would register all of the deserializers for all of the messages they support with the repository and also register the message type IDs that they want with the message router.