Drexel dragonThe Math ForumDonate to the Math Forum



Search All of the Math Forum:

Views expressed in these public forums are not endorsed by Drexel University or The Math Forum.


Math Forum » Discussions » Software » comp.soft-sys.matlab

Topic: saving huge char arrays
Replies: 4   Last Post: Apr 17, 2013 3:07 PM

Advanced Search

Back to Topic List Back to Topic List Jump to Tree View Jump to Tree View   Messages: [ Previous | Next ]
dpb

Posts: 8,112
Registered: 6/7/07
Re: saving huge char arrays
Posted: Apr 17, 2013 9:42 AM
  Click to see the message monospaced in plain text Plain Text   Click to reply to this topic Reply

On 4/17/2013 1:35 AM, runcyclexcski@gmail.com wrote:
....
> I have large char arrays (e.g. 3^17 by 17) which I need to save in
> plain text. Each array occupies about 2.5 Gb of disk space, and takes
> ~40 min (2646 seconds) to write line by line using fprintf:
>
> for m=1:length(array)
> fprintf(fid,'%s\r\n',array(m,:));
> end
> Elapsed time is 2646.335813 seconds.
>
> The file needs to be formatted as above to be interpreted by other
> apps.
>
> Writing 2.5 Gb of data should not take that long. Is there a way to
> do the writing faster?

...

Well, clearly stream unformatted would be the faster but if must be
foramtted w/ record delimiters for the other apps, try sotoo

fmt=[repmat('%c',1,size(array,2)) '\n'];
fid=fopen(filename,'rt');
fprintf(fid,fmt,array');

NB: the 't' on the fopen() takes care of the proper newline character(s)
for the platform.

Or, as you note, you could insert the \n character(s) in the array in
memory and then dump it w/ fwrite() if the above is still too slow.

--




Point your RSS reader here for a feed of the latest messages in this topic.

[Privacy Policy] [Terms of Use]

© Drexel University 1994-2014. All Rights Reserved.
The Math Forum is a research and educational enterprise of the Drexel University School of Education.