Well, start writing! Eat a programming elephant (even the smallest ones) one byte at a time!
How do we form the difference of two images? First, convert them to doubles in case they are uint8 images, as is common. DO IT! TRY IT! Learn to write matlab code by doing so, and do it in pieces, so you can follow what you did.
First of all, you have not told us if this is to be a MSE over all three channels. Your formula says that we should end up with a different MSE for each of the red, green and blue channels.
double(M1) - double(M2)
Now, how would you form the square of each difference? Use the .^ operator.
(double(M1) - double(M2)).^2
Next, mean squared error implies that we take the mean over all rows and columns. A simple way to do this is with the mean function. This call takes the mean across the rows.
mean((double(M1) - double(M2)).^2,2)
And the next one takes the mean down the columns.
mean(mean((double(M1) - double(M2)).^2,2),1)
The result will be a 1x1x3 vector. Convert that into a 1x3 vector using the reshape function. (The squeeze function would help as well.) Packaging it all into one line, we get this...
MSE = reshape(mean(mean((double(M1) - double(M2)).^2,2),1),[1,3]);
If this seems complex to you, then you are best off splitting it into several lines, with comments that remind you what you did for later.
But the point is, you create an operation in matlab by breaking it down into manageable pieces.
EDIT:
In many cases, people want the RMSE (root-mean-squared-error) which has units the same as your original numbers. It is just the square root of the MSE.