
Re: Importing large file into table and calculating takes a long time. How to improve efficiency?
Posted:
May 1, 2012 2:59 PM


I haven't studied your code, but definitely the first error I can see is, that normal asignment is "=" not "==" in Mathematica.
Jakub
> Original Message > From: Gangerolv [mailto:ipschka@yahoo.com] > Sent: Tuesday, May 01, 2012 11:24 AM > To: mathgroup@smc.vnet.net > Subject: Importing large file into table and calculating takes a > long time. How to improve efficiency? > > (First my disclaimer, I'm new to mathematica) > > I'm importing a file with three values, x,y,z in hexadecimal. > Sample of the input data: {0., 24, "009d"}, {0., 28, 9}, {0., 28, 99}, {"00dc", > 27, 98}, {0., 29, 95},... > > This set is converted to integer: {0., 36, 157}, {0., 40, 9}, {0., 40, 153}, {220, > 39, 152}, {0., 41, 149},... > > A final vector is calculated using: g==sqrt(x^2 + y^2 + z^2) for each data set. > > The data is then plotted. > > All of this is good an works great for a small file (2000 data sets). But when I > try to import and calculate a larger file (over 100k data sets), it seems to take > forever. Either my methods are not efficient (use of Table), or I'm not using > correct settings for importing of the data. The file is only 2Mb so I know > mathematica should be able to handle it. > > Here's what I'm doing: > > ========================================================================= > dataHex == > Import["C:\\Projects\\Mathematica\\test.csv"] > points == Length[dataHex] > dataDec == Table[ > ToExpression["16^^" <> #] & /@ { > ToString[dataHex[[i, 1]]], > ToString[dataHex[[i, 2]]], > ToString[dataHex[[i, 3]]]}, > {i, 1, points}] > dataDecCompl == Table[{ > If[dataDec[[i, 1]] > 32768, dataDec[[i, 1]]  65536, dataDec[[i, 1]]], > If[dataDec[[i, 2]] > 32768, dataDec[[i, 2]]  65536, dataDec[[i, 2]]], > If[dataDec[[i, 3]] > 32768, dataDec[[i, 2]]  65536, dataDec[[i, 3]]]}, > {i, 1, points}]; > > dataG == MovingAverage[Table[ > Sqrt[dataX[[i]]^2 + dataY[[i]]^2 + dataZ[[i]]^2] // N, > {i, 1, points  filter}], > 3]; > plotG == ListLinePlot[{dataG}, PlotRange > All]; > ========================================================================= > > Any suggestions would be highly appreciated. Using M 8.0.4 on Win7. > > Cheers! > > jan >

