Drexel dragonThe Math ForumDonate to the Math Forum



Search All of the Math Forum:

Views expressed in these public forums are not endorsed by Drexel University or The Math Forum.


Math Forum » Discussions » Software » comp.soft-sys.matlab

Topic: Speedy loading of non-matlab data files
Replies: 1   Last Post: Nov 28, 1996 11:11 AM

Advanced Search

Back to Topic List Back to Topic List Jump to Tree View Jump to Tree View   Messages: [ Previous | Next ]
Myrick Hatch

Posts: 1
Registered: 12/11/04
Speedy loading of non-matlab data files
Posted: Nov 28, 1996 12:22 AM
  Click to see the message monospaced in plain text Plain Text   Click to reply to this topic Reply

I am trying to speed up reads of database files created by a non-matlab
package.

Each record in these files is formated with a date stored as yymmdd
text, followed by 10 integers apparently stored binary. Since these
files are many, big, and frequently updated by the other package's
system, storing them reformatted as matlab files is not practical.

The problem is that my current approach is painfully slow. Right now, I
have a while loop, with the core as:

mymatrix(count,1)=str2num(setstr(fread(fid, 6, 'int8')'));
mymatrix(count,2:11)=fread(fid, 10, 'int')';

Mymatrix has been pre-dimensioned before entering this loop, but this
still takes almost five seconds to read 2000 records, and I have over
400 such files. The package which originally created the file (a custom
C++ database app, source code not available) loads the same file in an
instant. I'm using matlab 4.2 on a Win95-166Pentium, with tons of RAM
and HD.

I have a hunch that I can speed this up by reading a whole record at a
time using fscanf plus a suitable format argument, ie %6d%10s or
something. Unfortunately, the matlab docs are pretty thin on these
format codes, my library of C texts are all C++, which has gone over to
fstream, and experimentation has yielded nothing but garble.

Another idea is just a byte-wise read into one long temporary vector
using some fread or fscanf command and "eof' control structure which
would be real fast, then sorting out the format into Mymatrix all in ram
using matlab matrix operations. But I haven't discovered any faster
byte-wise read loop so I haven't experimented with the tradeoffs here
either.

I'd be grateful if someone can offer either fscan guidance or some other
ideas. Or is my current approach about as fast as I can expect.

Thanks for any help!





Point your RSS reader here for a feed of the latest messages in this topic.

[Privacy Policy] [Terms of Use]

© Drexel University 1994-2014. All Rights Reserved.
The Math Forum is a research and educational enterprise of the Drexel University School of Education.