As you may know, Fortran is a language for scientific computing. however, the kernel of the most famous high level language for scientific computing has been written in C instead of Fortran. Why?
You can find some information on the origins of Matlab by viewing a video on their web site, specifically on the page of the Chief Scientist, Cleve Moler.
I want to qualify this answer with the following statement - I know nothing about Fortran or Matlab
However i think you almost answer your own question in your question.
Why would you not use a scientific language to write a kernel. You would use the most appropriate language for the task at hand. A kernel would be best written in C as it is better suited to working with low level tasks like kernels (working with memory management etc)
A Kernel is not a scientific application therefore you wouldn't use a scientific language.
Again, i may be completely off the mark but i think that is a highly logical answer.
My understanding is that Matlab is written in a hodge-podge of languages, including but not limited to C, FORTRAN, Java, and Matlab itself.
I believe it employs best-of-breed libraries behind the scenes, including LAPACK (FORTRAN), FFTW (C) and probably many more.
I think the original version was pure FORTRAN, but as it became more polished and commercial they weren't shy about using the right language for each component. I don't blame them--I'd hate to write a parser or a GUI in FORTRAN!
Guessing on the basis of experience:
Memory management.
Fortran (well, the fortran that was around when matlab was written) has no support for dynamic memory management. Which makes it a pain for big work (see, for instance, CERNLIB).
CERNLIB solves the problem by allocating a whopping big array in a common block, and implementing a malloc like (de)allocator for the cells of the array. Clunky but working: array offsets are pointer equivalents, and away you go...
To answer your original question: Matlab was originally written in Fortran. But one of the first things that the creator of Matlab, Cleve Moler, and his partner did in 1983 was to rewrite the entire Matlab app in C:
Jack Little left his job at the consulting company and bought a new COMPAQ portable computer at Sears. The machine had only 256 KB of memory and no hard disc; Jack had to swap 5-1/4-inch floppies to compile programs. Jack and Steve took a year and a half to re- write MATLAB in C, adding new features they had envi- sioned. (The Growth of MATLAB and The MathWorks over Two Decades)
I think a more relevant question for today would be to ask: why did they switch to C back in 1983 ?
My guess is that C probably had certain features that Fortran did not have and thus the switch was more out of necessity and in the interest of code maintainability.
One killer feature missing in Fortran77 was dynamic memory allocation (Dynamic memory allocation was only added in Fortran90). Getting around that single limitation would be a horrible deal breaker in my opinion.
EDIT:
dmckee's answer has a great explanation of why using Fortran77 would be painful.