Drexel dragonThe Math ForumDonate to the Math Forum



Search All of the Math Forum:

Views expressed in these public forums are not endorsed by Drexel University or The Math Forum.


Math Forum » Discussions » Software » comp.soft-sys.matlab

Topic: efficient large data handling/output
Replies: 1   Last Post: Oct 17, 2013 4:31 PM

Advanced Search

Back to Topic List Back to Topic List Jump to Tree View Jump to Tree View   Messages: [ Previous | Next ]
horst

Posts: 3
Registered: 10/17/13
efficient large data handling/output
Posted: Oct 17, 2013 8:49 AM
  Click to see the message monospaced in plain text Plain Text   Click to reply to this topic Reply

Dear Matlab Professionals,

I am writing a code that detects and tracks ocean eddies on a global daily grid on timescales of 10a. In ten years I expect a number of eddies on the order of 10^5. For each eddy I want to save a number of parameters for each time-step of its life. Each eddy on average has a lifetime of a couple months.
To make long story short:

What do you reckon would be the most efficient way to save the eddy-data? Keep in mind, that in order to track them i will have to find, for each eddy at timestep t+1, its respective past image at t. I might even have to access time-steps t-1 and t-2.

The simplest would be to create one file for each eddy but then i will end up with 10^5 files.. If I jam all eddies into one file, I will end up with a file of several 10GB which I would have to scan for each eddy at every time-step to allocate the new data to the appropriate eddy...

Splitting the grid wouldn't make sense as eddies travel long distances..

Maybe somebody has a tip?



Point your RSS reader here for a feed of the latest messages in this topic.

[Privacy Policy] [Terms of Use]

© Drexel University 1994-2014. All Rights Reserved.
The Math Forum is a research and educational enterprise of the Drexel University School of Education.