Currently, generics in C# do not allow any sane way to perform arithmetic. There are awkward workarounds available, but none of them are very neat and all of them reduce performance. According to this interview, an interface with arithmetic types is not possible to implement, and so one such workaround is suggested.
But what you could do is have your Matrix take as an argument a Calculator, and in Calculator, have a method called multiply. You go implement that and you pass it to the Matrix.
Why should I have to tell an advanced programming language how to add and multiply numbers? [Edited due to popular demand]
Why not simply allow a Generic to be restricted to a list of types?
Eg.
class Matrix<T> where T : int,long,float,double
The syntax could of course be different. But the compiler needs only to check that the type is on the list, and that the operators used work on all types, which should be much simpler than the apparently-too-difficult interface suggestion.
Are there any obvious reasons as to why this cannot be implemented?