Drexel dragonThe Math ForumDonate to the Math Forum



Search All of the Math Forum:

Views expressed in these public forums are not endorsed by Drexel University or The Math Forum.


Math Forum » Discussions » Software » comp.soft-sys.matlab

Topic: Convert Large CSV to .mat without loading entire file into memory
Replies: 4   Last Post: Jun 21, 2014 11:03 PM

Advanced Search

Back to Topic List Back to Topic List Jump to Tree View Jump to Tree View   Messages: [ Previous | Next ]
Jonathan

Posts: 3
Registered: 6/20/14
Convert Large CSV to .mat without loading entire file into memory
Posted: Jun 20, 2014 5:32 PM
  Click to see the message monospaced in plain text Plain Text   Click to reply to this topic Reply

I have a CSV file that is >100 GB which I need to convert to a .mat. Unfortunately, no one machine has enough memory to read the entire file into disk, and then write out to a .mat.

Is there a way to more or less convert a CSV a few lines at a time into a .mat a "little bit at a time"?

On a side note, I find saving a large dataset as a .mat (e.g. a cell array of 20x1e7 strings), which is around 40 GB of data, is MUCH slower than reading the same dataset in from a CSV file. How can this be? Is there something I can do about it?

Thanks in advance,
Jon



Point your RSS reader here for a feed of the latest messages in this topic.

[Privacy Policy] [Terms of Use]

© Drexel University 1994-2014. All Rights Reserved.
The Math Forum is a research and educational enterprise of the Drexel University School of Education.