Probably SciPy, which is a set of libraries for Python that's pretty competitive with Matlab. There are various compilers out there for Python, so you can probably generate DLLs with some work.
R is a very powerful maths language, which is free.
It can be connected to a variety of languages via statconn (downloads here). I have personally used it via statconn in C# and have found it relatively easy to get going.
It depends on what you exactly going to do. For simple matrix calculations stay with C# and use a free .NET numerics library like Math.NET Numerics (google lists also some other). I have used it last year and it has the same good performance as MATLAB, it also uses LAPACK.
I have never used .NET charts for plotting myself but google shows that there are some free implementations (both winfroms and wpf).
See also http://stackoverflow.com/questions/392857/c-linear-algebra-library, http://stackoverflow.com/questions/1675735/high-performance-math-library-for-net-c-and-java and http://stackoverflow.com/questions/2815237/is-there-any-algebra-calculation-library-for-net
Octave - but I don't think you can make a dll. The alternative is to code up what you want from MATLAB in C#. Get hints from Numerical Recipies if you can read C++ or FORTRAN.