Drexel dragonThe Math ForumDonate to the Math Forum



Search All of the Math Forum:

Views expressed in these public forums are not endorsed by Drexel University or The Math Forum.


Math Forum » Discussions » Software » comp.soft-sys.matlab

Topic: MAT file takes forever to save with big object array
Replies: 6   Last Post: Dec 6, 2012 2:34 PM

Advanced Search

Back to Topic List Back to Topic List Jump to Tree View Jump to Tree View   Messages: [ Previous | Next ]
Alexey

Posts: 2
Registered: 12/21/10
Re: MAT file takes forever to save with big object array
Posted: Dec 26, 2010 6:36 PM
  Click to see the message monospaced in plain text Plain Text   Click to reply to this topic Reply

"Sean de " <sean.dewolski@nospamplease.umit.maine.edu> wrote in message <ieqp8l$sdq$1@fred.mathworks.com>...
> "Alexey" wrote in message <iepsb9$mrm$1@fred.mathworks.com>...
> > Hi,
> >
> > I have a 65k-element array of objects of a particular class.
> >

> > >> whos results
> > Name Size Bytes Class Attributes
> >
> > results 1x65000 3025921848 StrategyResultDef
> >
> > When I try to save it to a MAT file I get the following error:
> >

> > >save backtesting/files/technical-obj
> > Warning: Variable 'results' cannot be saved to a MAT-file whose version is older
> > than 7.3.
> > To save this variable, use the -v7.3 switch.
> > Skipping...
> >
> > And the variable 'results' is not saved.
> >
> > If I do:
> >

> > >save -v7.3 backtesting/files/technical-obj
> >
> > I do not get an error but it takes about 4 hours to save the file and its size is 1.5gb.
> >
> > I want to save the results for later processing but at this rate it's easier for me to re-generate the results every time.
> >
> > I will have a look at reducing the size of the structure but does anyone have any suggestions as to what is the best way to save the results like this?
> >
> > Kind Regards,
> > Alexey

>
> Hmm that does seem like a really long time. I frequently save/load 9GB files (structs with 3-3GB fields) and it usually takes a few minutes (MAC r2009b, 64bit). I have found for simple computations it's quicker to recalculate than to save. I.e: write a script that loads/generates what you need, run it, go get a coffee.
>
> You could also try writing it to a binary file using fwrite and reading it in with fread. This is a pain though since you need to remember: dimensions, classes, bytes to skip if necessary etc.



Thanks for suggestions!
I have tried to use R2010b and it has the same problem. Once the vector of objects reaches 64k elements it is impossible to save it. Just takes forever and then fails with "cannot close file" or something like this.

I'll probably consider saving to database as I wanted to do searches through the results.



Point your RSS reader here for a feed of the latest messages in this topic.

[Privacy Policy] [Terms of Use]

© Drexel University 1994-2014. All Rights Reserved.
The Math Forum is a research and educational enterprise of the Drexel University School of Education.