You could go for the Java approach (since Matlab runs on a JRE and can access Java objects/methods -- just be aware that the Matlab JRE is not as up-to-date as the latest JRE, the one I'm running uses Java 1.5) and use JNA to access your DLL.
Or, if you wrote the top-level DLL, you could go for the COM/ActiveX approach.
I've had good success architecting the interface to my C++ functions as COM/ActiveX libraries -- you don't have to bother with that .h stuff.
See the External Interfaces guide on COM clients, particularly the part about managing/converting data.
It would be extra work to add the COM/ActiveX layer, but would make your library more portable within the Windows world and probably more easily used in MATLAB.
If you have a lot of function calls to your DLL, the COM/ActiveX approach might be faster (not sure), but otherwise I think the JNA approach would be easier.