The Math Forum

Search All of the Math Forum:

Views expressed in these public forums are not endorsed by NCTM or The Math Forum.

Math Forum » Discussions » Software » comp.soft-sys.matlab

Notice: We are no longer accepting new posts, but the forums will continue to be readable.

Topic: Speedy loading of non-matlab data files
Replies: 1   Last Post: Nov 28, 1996 11:11 AM

Advanced Search

Back to Topic List Back to Topic List Jump to Tree View Jump to Tree View   Messages: [ Previous | Next ]
Myrick Hatch

Posts: 1
Registered: 12/11/04
Speedy loading of non-matlab data files
Posted: Nov 28, 1996 12:22 AM
  Click to see the message monospaced in plain text Plain Text   Click to reply to this topic Reply

I am trying to speed up reads of database files created by a non-matlab

Each record in these files is formated with a date stored as yymmdd
text, followed by 10 integers apparently stored binary. Since these
files are many, big, and frequently updated by the other package's
system, storing them reformatted as matlab files is not practical.

The problem is that my current approach is painfully slow. Right now, I
have a while loop, with the core as:

mymatrix(count,1)=str2num(setstr(fread(fid, 6, 'int8')'));
mymatrix(count,2:11)=fread(fid, 10, 'int')';

Mymatrix has been pre-dimensioned before entering this loop, but this
still takes almost five seconds to read 2000 records, and I have over
400 such files. The package which originally created the file (a custom
C++ database app, source code not available) loads the same file in an
instant. I'm using matlab 4.2 on a Win95-166Pentium, with tons of RAM
and HD.

I have a hunch that I can speed this up by reading a whole record at a
time using fscanf plus a suitable format argument, ie %6d%10s or
something. Unfortunately, the matlab docs are pretty thin on these
format codes, my library of C texts are all C++, which has gone over to
fstream, and experimentation has yielded nothing but garble.

Another idea is just a byte-wise read into one long temporary vector
using some fread or fscanf command and "eof' control structure which
would be real fast, then sorting out the format into Mymatrix all in ram
using matlab matrix operations. But I haven't discovered any faster
byte-wise read loop so I haven't experimented with the tradeoffs here

I'd be grateful if someone can offer either fscan guidance or some other
ideas. Or is my current approach about as fast as I can expect.

Thanks for any help!

Point your RSS reader here for a feed of the latest messages in this topic.

[Privacy Policy] [Terms of Use]

© The Math Forum at NCTM 1994-2018. All Rights Reserved.