Search All of the Math Forum:

Views expressed in these public forums are not endorsed by NCTM or The Math Forum.

Notice: We are no longer accepting new posts, but the forums will continue to be readable.

Topic: Importing large file into table and calculating takes a long time. How to improve efficiency?
Replies: 3   Last Post: May 2, 2012 5:46 AM

 Messages: [ Previous | Next ]
 Jakub Serych Posts: 51 Registered: 1/27/08
Re: Importing large file into table and calculating takes a long time. How to improve efficiency?
Posted: May 1, 2012 2:59 PM

I haven't studied your code, but definitely the first error I can see is, that normal asignment is "=" not "==" in Mathematica.

Jakub

> -----Original Message-----
> From: Gangerolv [mailto:ipschka@yahoo.com]
> Sent: Tuesday, May 01, 2012 11:24 AM
> To: mathgroup@smc.vnet.net
> Subject: Importing large file into table and calculating takes a
> long time. How to improve efficiency?
>
> (First my disclaimer, I'm new to mathematica)
>
> I'm importing a file with three values, x,y,z in hexadecimal.
> Sample of the input data: {0., 24, "009d"}, {0., 28, 9}, {0., 28, 99}, {"00dc",
> 27, 98}, {0., 29, 95},...
>
> This set is converted to integer: {0., 36, 157}, {0., 40, 9}, {0., 40, 153}, {220,
> 39, 152}, {0., 41, 149},...
>
> A final vector is calculated using: g==sqrt(x^2 + y^2 + z^2) for each data set.
>
> The data is then plotted.
>
> All of this is good an works great for a small file (2000 data sets). But when I
> try to import and calculate a larger file (over 100k data sets), it seems to take
> forever. Either my methods are not efficient (use of Table), or I'm not using
> correct settings for importing of the data. The file is only 2Mb so I know
> mathematica should be able to handle it.
>
> Here's what I'm doing:
>
> =========================================================================
> dataHex ==
> Import["C:\\Projects\\Mathematica\\test.csv"]
> points == Length[dataHex]
> ToExpression["16^^" <> #] & /@ {
> ToString[dataHex[[i, 1]]],
> ToString[dataHex[[i, 2]]],
> ToString[dataHex[[i, 3]]]},
> {i, 1, points}]
> {i, 1, points}];
>
> dataG == MovingAverage[Table[
> Sqrt[dataX[[i]]^2 + dataY[[i]]^2 + dataZ[[i]]^2] // N,
> {i, 1, points - filter}],
> 3];
> plotG == ListLinePlot[{dataG}, PlotRange -> All];
> =========================================================================
>
> Any suggestions would be highly appreciated. Using M 8.0.4 on Win7.
>
> Cheers!
>
> jan
>

Date Subject Author
5/1/12 Gangerolv
5/1/12 Jakub Serych
5/1/12 Bob Hanlon
5/2/12 Bob Hanlon