Drexel dragonThe Math ForumDonate to the Math Forum



Search All of the Math Forum:

Views expressed in these public forums are not endorsed by Drexel University or The Math Forum.


Math Forum » Discussions » Software » comp.soft-sys.matlab

Topic: Typecasting After Reading in Large Binary File
Replies: 9   Last Post: Mar 1, 2013 12:22 PM

Advanced Search

Back to Topic List Back to Topic List Jump to Tree View Jump to Tree View   Messages: [ Previous | Next ]
James Tursa

Posts: 2,096
Registered: 8/5/09
Re: Typecasting After Reading in Large Binary File
Posted: Feb 28, 2013 12:56 PM
  Click to see the message monospaced in plain text Plain Text   Click to reply to this topic Reply

"Scott" wrote in message <kgnkm4$dhp$1@newscl01ah.mathworks.com>...
> "Image Analyst" wrote in message <kgmi5n$cml$1@newscl01ah.mathworks.com>...
> > "Scott" wrote in message <kgloic$s1b$1@newscl01ah.mathworks.com>...
> > > I'm reading in a large binary file ...
> > > A = 7000x2 matrix of uint8
> > > B = typecast(A,'int16')
> > > Desired Result:
> > > B = 7000x1 array of int16 where each int16 is derived from each row in A.

> > =========================================
> > Not sure what the problem was, but 7000 by 2 is actually very small. It's not like > you have tens of millions of numbers or anything. 14,000 elements is a pretty small > array by today's standards. Even a garden variety digital image is about 30 thousand > times bigger than that.

>
> Haha! I agree! That was the header. The file size can cary from 5M to 2G.
>
> It takes me about 6 minutes to read a 5M file right now - completely unacceptable. I was able to speed the header read from 30 sec to 7 ms by typecasting and byte swapping after I read all the data points in. I plan to use the same approach for the data points. The data points can be anything from a uint8 to a double and is determined by an uint16 ID that precedes the data point. From what I've been reading here, the typecasting takes the longest, so I plan to gather the data in arrays, then typecast, and byte swap if necessary. I'm hoping for a ~300M file to be processed in less than 10 seconds.


The built-in MATLAB typecast function does a deep copy of the data to get the result. If you want to avoid that deep copy you can use this typecast function from the FEX which returns a shared data copy:

http://www.mathworks.com/matlabcentral/fileexchange/17476-typecast-and-typecastx-c-mex-functions

James Tursa



Point your RSS reader here for a feed of the latest messages in this topic.

[Privacy Policy] [Terms of Use]

© Drexel University 1994-2014. All Rights Reserved.
The Math Forum is a research and educational enterprise of the Drexel University School of Education.