Outspan
Posts:
17
Registered:
7/14/09
|
|
Problem with memory usage (R2009a)
Posted:
Oct 20, 2009 7:42 AM
|
|
Hi all,
I have a big CSV file (50K lines) that I want to load into memory and
then save to a .mat file so that I can access the data randomly. The
CSV file is about 25MB and the resulting .mat file is about 10MB.
However, during execution memory usage is much higher (about 300MB)
and when I load the .mat file into memory, I also use up 300MB of
memory, and I don't understand why. Can you help me with this? Is
there a way to reduce memory usage?
Here's the script:
***
function bars = loadHistory(symbol)
filepath = strcat('C:\forexdata\', upper(symbol), '.csv');
fid = fopen(filepath);
if fid == -1
fprintf('Error: could not open file %s\n', filepath);
return
end
bars = struct('date', [], 'time', [], 'open', [], 'high', [], 'low',
[], 'close', [], 'volume', []);
bars = repmat(bars, [1 500000]);
i = 1;
commas = ones(1, 6);
while(1)
line = fgets(fid);
if line == -1
break;
else
j = 1;
for cnt = 1:length(line)
if line(cnt) == ','
commas(j) = cnt;
j = j+1;
end
end
bars(i).date = datenum(line(1:commas(1)-1), 'yyyy.mm.dd');
hr = str2double(line(commas(1)+1:commas(1)+2));
mn = str2double(line(commas(1)+4:commas(1)+5));
bars(i).time = hr*60 + mn;
bars(i).open = str2double(line(commas(2)+1:commas(3)-1));
bars(i).high = str2double(line(commas(3)+1:commas(4)-1));
bars(i).low = str2double(line(commas(4)+1:commas(5)-1));
bars(i).close = str2double(line(commas(5)+1:commas(6)-1));
bars(i).volume = str2double(line(commas(6)+1:max(size
(line))));
i = i+1;
end
end
fclose(fid);
|
|
