Drexel dragonThe Math ForumDonate to the Math Forum



Search All of the Math Forum:

Views expressed in these public forums are not endorsed by Drexel University or The Math Forum.


Math Forum » Discussions » Software » comp.soft-sys.matlab

Topic: Problem with memory usage (R2009a)
Replies: 11   Last Post: Oct 21, 2009 10:20 AM

Advanced Search

Back to Topic List Back to Topic List Jump to Tree View Jump to Tree View   Messages: [ Previous | Next ]
Outspan

Posts: 17
Registered: 7/14/09
Problem with memory usage (R2009a)
Posted: Oct 20, 2009 7:42 AM
  Click to see the message monospaced in plain text Plain Text   Click to reply to this topic Reply

Hi all,

I have a big CSV file (50K lines) that I want to load into memory and
then save to a .mat file so that I can access the data randomly. The
CSV file is about 25MB and the resulting .mat file is about 10MB.
However, during execution memory usage is much higher (about 300MB)
and when I load the .mat file into memory, I also use up 300MB of
memory, and I don't understand why. Can you help me with this? Is
there a way to reduce memory usage?

Here's the script:

***

function bars = loadHistory(symbol)

filepath = strcat('C:\forexdata\', upper(symbol), '.csv');
fid = fopen(filepath);

if fid == -1
fprintf('Error: could not open file %s\n', filepath);
return
end

bars = struct('date', [], 'time', [], 'open', [], 'high', [], 'low',
[], 'close', [], 'volume', []);
bars = repmat(bars, [1 500000]);

i = 1;
commas = ones(1, 6);
while(1)
line = fgets(fid);
if line == -1
break;
else
j = 1;
for cnt = 1:length(line)
if line(cnt) == ','
commas(j) = cnt;
j = j+1;
end
end

bars(i).date = datenum(line(1:commas(1)-1), 'yyyy.mm.dd');
hr = str2double(line(commas(1)+1:commas(1)+2));
mn = str2double(line(commas(1)+4:commas(1)+5));
bars(i).time = hr*60 + mn;
bars(i).open = str2double(line(commas(2)+1:commas(3)-1));
bars(i).high = str2double(line(commas(3)+1:commas(4)-1));
bars(i).low = str2double(line(commas(4)+1:commas(5)-1));
bars(i).close = str2double(line(commas(5)+1:commas(6)-1));
bars(i).volume = str2double(line(commas(6)+1:max(size
(line))));

i = i+1;
end
end

fclose(fid);



Point your RSS reader here for a feed of the latest messages in this topic.

[Privacy Policy] [Terms of Use]

© Drexel University 1994-2014. All Rights Reserved.
The Math Forum is a research and educational enterprise of the Drexel University School of Education.