Drexel dragonThe Math ForumDonate to the Math Forum



Search All of the Math Forum:

Views expressed in these public forums are not endorsed by Drexel University or The Math Forum.


Math Forum » Discussions » Software » comp.soft-sys.math.mathematica

Topic: Importing large file into table and calculating takes a long time. How to improve efficiency?
Replies: 3   Last Post: May 2, 2012 5:46 AM

Advanced Search

Back to Topic List Back to Topic List Jump to Tree View Jump to Tree View   Messages: [ Previous | Next ]
Jakub Serych

Posts: 51
Registered: 1/27/08
Re: Importing large file into table and calculating takes a long time. How to improve efficiency?
Posted: May 1, 2012 2:59 PM
  Click to see the message monospaced in plain text Plain Text   Click to reply to this topic Reply

I haven't studied your code, but definitely the first error I can see is, that normal asignment is "=" not "==" in Mathematica.


Jakub

> -----Original Message-----
> From: Gangerolv [mailto:ipschka@yahoo.com]
> Sent: Tuesday, May 01, 2012 11:24 AM
> To: mathgroup@smc.vnet.net
> Subject: Importing large file into table and calculating takes a
> long time. How to improve efficiency?
>
> (First my disclaimer, I'm new to mathematica)
>
> I'm importing a file with three values, x,y,z in hexadecimal.
> Sample of the input data: {0., 24, "009d"}, {0., 28, 9}, {0., 28, 99}, {"00dc",
> 27, 98}, {0., 29, 95},...
>
> This set is converted to integer: {0., 36, 157}, {0., 40, 9}, {0., 40, 153}, {220,
> 39, 152}, {0., 41, 149},...
>
> A final vector is calculated using: g==sqrt(x^2 + y^2 + z^2) for each data set.
>
> The data is then plotted.
>
> All of this is good an works great for a small file (2000 data sets). But when I
> try to import and calculate a larger file (over 100k data sets), it seems to take
> forever. Either my methods are not efficient (use of Table), or I'm not using
> correct settings for importing of the data. The file is only 2Mb so I know
> mathematica should be able to handle it.
>
> Here's what I'm doing:
>
> =========================================================================
> dataHex ==
> Import["C:\\Projects\\Mathematica\\test.csv"]
> points == Length[dataHex]
> dataDec == Table[
> ToExpression["16^^" <> #] & /@ {
> ToString[dataHex[[i, 1]]],
> ToString[dataHex[[i, 2]]],
> ToString[dataHex[[i, 3]]]},
> {i, 1, points}]
> dataDecCompl == Table[{
> If[dataDec[[i, 1]] > 32768, dataDec[[i, 1]] - 65536, dataDec[[i, 1]]],
> If[dataDec[[i, 2]] > 32768, dataDec[[i, 2]] - 65536, dataDec[[i, 2]]],
> If[dataDec[[i, 3]] > 32768, dataDec[[i, 2]] - 65536, dataDec[[i, 3]]]},
> {i, 1, points}];
>
> dataG == MovingAverage[Table[
> Sqrt[dataX[[i]]^2 + dataY[[i]]^2 + dataZ[[i]]^2] // N,
> {i, 1, points - filter}],
> 3];
> plotG == ListLinePlot[{dataG}, PlotRange -> All];
> =========================================================================
>
> Any suggestions would be highly appreciated. Using M 8.0.4 on Win7.
>
> Cheers!
>
> jan
>





Point your RSS reader here for a feed of the latest messages in this topic.

[Privacy Policy] [Terms of Use]

© Drexel University 1994-2014. All Rights Reserved.
The Math Forum is a research and educational enterprise of the Drexel University School of Education.