Search All of the Math Forum:
Views expressed in these public forums are not endorsed by
NCTM or The Math Forum.


horst
Posts:
3
Registered:
10/17/13


efficient large data handling/output
Posted:
Oct 17, 2013 8:49 AM


Dear Matlab Professionals,
I am writing a code that detects and tracks ocean eddies on a global daily grid on timescales of 10a. In ten years I expect a number of eddies on the order of 10^5. For each eddy I want to save a number of parameters for each timestep of its life. Each eddy on average has a lifetime of a couple months. To make long story short:
What do you reckon would be the most efficient way to save the eddydata? Keep in mind, that in order to track them i will have to find, for each eddy at timestep t+1, its respective past image at t. I might even have to access timesteps t1 and t2.
The simplest would be to create one file for each eddy but then i will end up with 10^5 files.. If I jam all eddies into one file, I will end up with a file of several 10GB which I would have to scan for each eddy at every timestep to allocate the new data to the appropriate eddy...
Splitting the grid wouldn't make sense as eddies travel long distances..
Maybe somebody has a tip?



