Drexel dragonThe Math ForumDonate to the Math Forum



Search All of the Math Forum:

Views expressed in these public forums are not endorsed by Drexel University or The Math Forum.


Math Forum » Discussions » Software » comp.soft-sys.matlab

Topic: saving huge char arrays
Replies: 4   Last Post: Apr 17, 2013 3:07 PM

Advanced Search

Back to Topic List Back to Topic List Jump to Tree View Jump to Tree View   Messages: [ Previous | Next ]
runcyclexcski

Posts: 14
Registered: 10/9/09
Re: saving huge char arrays
Posted: Apr 17, 2013 12:20 PM
  Click to see the message monospaced in plain text Plain Text   Click to reply to this topic Reply

On Wednesday, April 17, 2013 9:42:40 AM UTC-4, dpb wrote:
> On 4/17/2013 1:35 AM, ... wrote:
>
> ....
>

> > I have large char arrays (e.g. 3^17 by 17) which I need to save in
>
> > plain text. Each array occupies about 2.5 Gb of disk space, and takes
>
> > ~40 min (2646 seconds) to write line by line using fprintf:
>
> >
>
> > for m=1:length(array)
>
> > fprintf(fid,'%s\r\n',array(m,:));
>
> > end
>
> > Elapsed time is 2646.335813 seconds.
>
> >
>
> > The file needs to be formatted as above to be interpreted by other
>
> > apps.
>
> >
>
> > Writing 2.5 Gb of data should not take that long. Is there a way to
>
> > do the writing faster?
>
> ...
>
>
>
> Well, clearly stream unformatted would be the faster but if must be
>
> foramtted w/ record delimiters for the other apps, try sotoo
>
>
>
> fmt=[repmat('%c',1,size(array,2)) '\n'];
>
> fid=fopen(filename,'rt');
>
> fprintf(fid,fmt,array');
>
>
>
> NB: the 't' on the fopen() takes care of the proper newline character(s)
>
> for the platform.
>
>
>
> Or, as you note, you could insert the \n character(s) in the array in
>
> memory and then dump it w/ fwrite() if the above is still too slow.
>
>
>
> --


Thank you, sounds like the way to go. My downstream app reads the file line by line (\n by \n) and does analysis with each line on by one - hence the requirement for the carriage return.



Point your RSS reader here for a feed of the latest messages in this topic.

[Privacy Policy] [Terms of Use]

© Drexel University 1994-2014. All Rights Reserved.
The Math Forum is a research and educational enterprise of the Drexel University School of Education.