One common approach is to have several m-file functions that provide the public interface, e.g. sysInit.m, sysRefresh.m, etc.
Each of these m-files calls the mex function with some kind of handle, a string (or number) identifying the function to call, and any extra args. For example, sysRefresh.m might look like:
function sysRefresh(handle)
return sysMex(handle, 'refresh')
In your sysMex mex function, you can either have the handle be a raw heap pointer (easy, but not very safe), or you can maintain a mapping in C/C++ from the handle ID to the actual object pointers. This solution requires a little extra work, but it's much safer. This way someone can't accidentally pass an arbitrary number as a handle, which acts as a dangling pointer. Also, you can do fancier things like use an onCleanup function to release all memory and resources when you unload the mex function (e.g. so you don't have to restart matlab when you recompile the mex function).
You can go a bit further if you like and hide the handle behind a Matlab class. Read up on the OO features for Matlab in the docs if you're interested. If you're using a recent version, you can take advantage of their much cleaner handle objects.