I work with huge dataset and I have some problem with the saving of variables.
In my programme, I have a for loop which generates a moderately large matrix (around 2k by 15k) in every loop. Ultimately I need to put all the generated matrices together into a huge matrix and save it as a v7.3 mat file so I can partially read/modify the data later on. However, because the for loop executes for as many as a few hundred times, I can't hold all the generated data in RAM. Therefore, what I am trying to do is to save them in each loop and clear the variable immediately.
Following this line of thought, I have tried using save('filename.mat','variable','-append','-v7.3'); However, as my variable name is the same in each loop, I discovered that this command actually overwrites rather than appends.
Then I thought of trying to dynamically create variables with different names in each loop and append them into the mat file using evalc function. (which I know is a crime in matlab...) That worked. But, the problem now is how to tidy all of those into a big numerical array. I definitely can't open the entire mat file due to memory constraints.
Thank you so much in advance for your time and help.