The aim is this: given a C++ object created in a mex file, to pass this object or a reference to this object (ensuring the object persists in memory) back to Matlab, such that this object can be used again in future calls to a mex function (not necessarily the same one). It should also be robust to memory leaks - e.g. an error in Matlab should not cause any memory to become irretrievable.
Several solutions have been proposed (or if not, I propose them now), and I want to address the problems with each of these:
1. Make the object static: Problems - Firstly, if the mex file is cleared from memory for any reason then the object's state is lost. A call to mexLock can stop this, but then you need to create functionality to reset the object or unlock the function. Secondly, you can only use one object at a time, and cannot share it across mex files. However, there can be times when we want to instantiate two objects of a given class at the same time. A solution could be to statically allocate a pool of objects, but this wastes memory and doesn't get round the first problem. Lastly, if an error occurs in Matlab then the object needs to be reset manually by the user (e.g. by clearing the mex function).
2. Wrap your entire algorithm in a C++ function, and use mexCallMatlab to call the Matlab functions you want: My main issues with this are that it is cumbersome to set up, and it makes the solution much less portable. It may also make the algorithm difficult to interrupt (or will cause memory leaks) and to profile.
3. Allocate the object's memory (if using mxMalloc, use mexMakeMemoryPersistent to make the object persist in memory), and pass back a pointer (as a uint32) to Matlab. The code at: http://www.ee.columbia.edu/~mim/meap/paMat/mexhandle/ does this, adding in some nice type checking for robustness, but doesn't use mxMalloc so can potentially cause memory leaks. Using mxMalloc can still potentially cause memory leaks if you don't set up an exit callback function using mexAtExit, but then you have the same problem as with a static object/pointer, that the object will disappear if the mex function is cleared from memory. Even then, if you get an error you need to then clear the mex function to make sure any objects left hanging around are mopped up.
4. Allocate the object's memory using mxMalloc, and return this entire array to Matlab as a uint8 vector. This solves the problem of (some) memory leaks. You can keep and reuse the object even when the mex function is cleared, and simply delete the array when you're done with the object. If an error occurs then Matlab will delete the object for you. However, there is one further consideration. If a C++ object has allocated further memory, then this memory should be freed when the object is destroyed, and this is handled by the object's destructor function. This destructor function therefore needs to be called when the object is destroyed.
To summarise, all but solution 1 can lead to memory leaks if there is an error in Matlab, and solution 1 limits you to a specific number of objects which do not persist if the mex function is cleared, does not allow objects to be shared between mex files, and requires the mex function to be cleared from memory following an error in Matlab (to reset the object). It would be nice to have a solution that just works.
So, finally, some questions: Is is possible to create a Matlab class around a C++ object, which knows how to destroy the object properly when it is cleared in Matlab? How backwards compatible can such a solution be made?