Date: Oct 20, 2009 7:42 AM
Author: Outspan
Subject: Problem with memory usage (R2009a)

Hi all,

I have a big CSV file (50K lines) that I want to load into memory and
then save to a .mat file so that I can access the data randomly. The
CSV file is about 25MB and the resulting .mat file is about 10MB.
However, during execution memory usage is much higher (about 300MB)
and when I load the .mat file into memory, I also use up 300MB of
memory, and I don't understand why. Can you help me with this? Is
there a way to reduce memory usage?

Here's the script:

***

function bars = loadHistory(symbol)

filepath = strcat('C:\forexdata\', upper(symbol), '.csv');
fid = fopen(filepath);

if fid == -1
fprintf('Error: could not open file %s\n', filepath);
return
end

bars = struct('date', [], 'time', [], 'open', [], 'high', [], 'low',
[], 'close', [], 'volume', []);
bars = repmat(bars, [1 500000]);

i = 1;
commas = ones(1, 6);
while(1)
line = fgets(fid);
if line == -1
break;
else
j = 1;
for cnt = 1:length(line)
if line(cnt) == ','
commas(j) = cnt;
j = j+1;
end
end

bars(i).date = datenum(line(1:commas(1)-1), 'yyyy.mm.dd');
hr = str2double(line(commas(1)+1:commas(1)+2));
mn = str2double(line(commas(1)+4:commas(1)+5));
bars(i).time = hr*60 + mn;
bars(i).open = str2double(line(commas(2)+1:commas(3)-1));
bars(i).high = str2double(line(commas(3)+1:commas(4)-1));
bars(i).low = str2double(line(commas(4)+1:commas(5)-1));
bars(i).close = str2double(line(commas(5)+1:commas(6)-1));
bars(i).volume = str2double(line(commas(6)+1:max(size
(line))));

i = i+1;
end
end

fclose(fid);