dpb
Posts:
9,824
Registered:
6/7/07
|
|
Re: saving huge char arrays
Posted:
Apr 17, 2013 9:42 AM
|
|
On 4/17/2013 1:35 AM, runcyclexcski@gmail.com wrote:
....
> I have large char arrays (e.g. 3^17 by 17) which I need to save in
> plain text. Each array occupies about 2.5 Gb of disk space, and takes
> ~40 min (2646 seconds) to write line by line using fprintf:
>
> for m=1:length(array)
> fprintf(fid,'%s\r\n',array(m,:));
> end
> Elapsed time is 2646.335813 seconds.
>
> The file needs to be formatted as above to be interpreted by other
> apps.
>
> Writing 2.5 Gb of data should not take that long. Is there a way to
> do the writing faster?
...
Well, clearly stream unformatted would be the faster but if must be
foramtted w/ record delimiters for the other apps, try sotoo
fmt=[repmat('%c',1,size(array,2)) '\n'];
fid=fopen(filename,'rt');
fprintf(fid,fmt,array');
NB: the 't' on the fopen() takes care of the proper newline character(s)
for the platform.
Or, as you note, you could insert the \n character(s) in the array in
memory and then dump it w/ fwrite() if the above is still too slow.
--
|
|
