You have two options, both of which are unfortunately poorly documented.
If you are comfortable in C/C++, you can write a special DLL called an "XLL" that you can call during Excel runtime. There is some sparse documentation available. Note that this stuff isn't very fun to use.
If you prefer .NET, there is a binding for the entire Office suite outlined here that allows you to write COM-based methods that you can call from Office. It is intended for automation, but you can write any managed code you want and have Excel call into it.
There is also what Remou just suggested; I don't actually own a copy of Excel to test that out, but it may be the easiest option.