Drexel dragonThe Math ForumDonate to the Math Forum



Search All of the Math Forum:

Views expressed in these public forums are not endorsed by Drexel University or The Math Forum.


Math Forum » Discussions » Software » comp.soft-sys.matlab

Topic: saving huge char arrays
Replies: 4   Last Post: Apr 17, 2013 3:07 PM

Advanced Search

Back to Topic List Back to Topic List Jump to Tree View Jump to Tree View   Messages: [ Previous | Next ]
dpb

Posts: 8,100
Registered: 6/7/07
Re: saving huge char arrays
Posted: Apr 17, 2013 2:58 PM
  Click to see the message monospaced in plain text Plain Text   Click to reply to this topic Reply

On 4/17/2013 11:20 AM, runcyclexcski@gmail.com wrote:
> On Wednesday, April 17, 2013 9:42:40 AM UTC-4, dpb wrote:
>> On 4/17/2013 1:35 AM, ... wrote:
>>
>> ....
>>

>>> I have large char arrays (e.g. 3^17 by 17) which I need to save in
>>
>>> plain text. Each array occupies about 2.5 Gb of disk space, and takes
>>
>>> ~40 min (2646 seconds) to write line by line using fprintf:
>>
>>>
>>
>>> for m=1:length(array)
>>
>>> fprintf(fid,'%s\r\n',array(m,:));
>>
>>> end
>>
>>> Elapsed time is 2646.335813 seconds.
>>
>>>
>>
>>> The file needs to be formatted as above to be interpreted by other
>>
>>> apps.
>>
>>>
>>
>>> Writing 2.5 Gb of data should not take that long. Is there a way to
>>
>>> do the writing faster?
>>
>> ...
>>
>>
>>
>> Well, clearly stream unformatted would be the faster but if must be
>>
>> foramtted w/ record delimiters for the other apps, try sotoo
>>
>>
>>
>> fmt=[repmat('%c',1,size(array,2)) '\n'];
>>
>> fid=fopen(filename,'rt');
>>
>> fprintf(fid,fmt,array');
>>
>>
>>
>> NB: the 't' on the fopen() takes care of the proper newline character(s)
>>
>> for the platform.
>>
>>
>>
>> Or, as you note, you could insert the \n character(s) in the array in
>>
>> memory and then dump it w/ fwrite() if the above is still too slow.
>>
>>
>>
>> --

>
> Thank you, sounds like the way to go. My downstream app reads the
> file line by line (\n by \n) and does analysis with each line on by
> one - hence the requirement for the carriage return.


Actually, better than actually doing the insert would be to do the
fprintf() via sprintf() to memory instead then fwrite() to disk.

Possibly may want to segment into a few chunks to save some memory w/
the copy but you can test out various scenarios for relative performance.

--



Point your RSS reader here for a feed of the latest messages in this topic.

[Privacy Policy] [Terms of Use]

© Drexel University 1994-2014. All Rights Reserved.
The Math Forum is a research and educational enterprise of the Drexel University School of Education.