Drexel dragonThe Math ForumDonate to the Math Forum



Search All of the Math Forum:

Views expressed in these public forums are not endorsed by Drexel University or The Math Forum.


Math Forum » Discussions » Software » comp.soft-sys.matlab

Topic: MAT file takes forever to save with big object array
Replies: 6   Last Post: Dec 6, 2012 2:34 PM

Advanced Search

Back to Topic List Back to Topic List Jump to Tree View Jump to Tree View   Messages: [ Previous | Next ]
Sean

Posts: 1,187
Registered: 4/16/10
Re: MAT file takes forever to save with big object array
Posted: Dec 21, 2010 12:48 PM
  Click to see the message monospaced in plain text Plain Text   Click to reply to this topic Reply

"Alexey" wrote in message <iepsb9$mrm$1@fred.mathworks.com>...
> Hi,
>
> I have a 65k-element array of objects of a particular class.
>

> >> whos results
> Name Size Bytes Class Attributes
>
> results 1x65000 3025921848 StrategyResultDef
>
> When I try to save it to a MAT file I get the following error:
>

> >save backtesting/files/technical-obj
> Warning: Variable 'results' cannot be saved to a MAT-file whose version is older
> than 7.3.
> To save this variable, use the -v7.3 switch.
> Skipping...
>
> And the variable 'results' is not saved.
>
> If I do:
>

> >save -v7.3 backtesting/files/technical-obj
>
> I do not get an error but it takes about 4 hours to save the file and its size is 1.5gb.
>
> I want to save the results for later processing but at this rate it's easier for me to re-generate the results every time.
>
> I will have a look at reducing the size of the structure but does anyone have any suggestions as to what is the best way to save the results like this?
>
> Kind Regards,
> Alexey


Hmm that does seem like a really long time. I frequently save/load 9GB files (structs with 3-3GB fields) and it usually takes a few minutes (MAC r2009b, 64bit). I have found for simple computations it's quicker to recalculate than to save. I.e: write a script that loads/generates what you need, run it, go get a coffee.

You could also try writing it to a binary file using fwrite and reading it in with fread. This is a pain though since you need to remember: dimensions, classes, bytes to skip if necessary etc.



Point your RSS reader here for a feed of the latest messages in this topic.

[Privacy Policy] [Terms of Use]

© Drexel University 1994-2014. All Rights Reserved.
The Math Forum is a research and educational enterprise of the Drexel University School of Education.