The Math Forum

Search All of the Math Forum:

Views expressed in these public forums are not endorsed by NCTM or The Math Forum.

Math Forum » Discussions » Software » comp.soft-sys.matlab

Notice: We are no longer accepting new posts, but the forums will continue to be readable.

Topic: Sharing big data across parfor or spmd
Replies: 8   Last Post: Aug 6, 2013 4:48 AM

Advanced Search

Back to Topic List Back to Topic List Jump to Tree View Jump to Tree View   Messages: [ Previous | Next ]
Edric Ellis

Posts: 721
Registered: 12/7/04
Re: Sharing big data across parfor or spmd
Posted: Dec 12, 2012 3:39 AM
  Click to see the message monospaced in plain text Plain Text   Click to reply to this topic Reply

"Chuck37 " <> writes:

> I have a big piece of never changing data that I'd like to be used by
> all the workers in a parallel setting. Since I'm only working with
> local workers, I don't understand why I have to eat the huge latency
> associated with transferring the data to workers each time the parfor
> loop is called. Can someone explain?
> I tried to use spmd to send data once at the beginning and have it
> stay there, but the data is kind of big (~2 GB going to 10 workers),
> and I got "error during serialization" anyway. Is there a solution to
> this problem with local workers where they can all access the data
> from the same memory? Accesses are infrequent, so simultaneous access
> shouldn't cause a big slowdown.
> Any ideas would be great.
> My setup is something like this by the way:
> M = big data;
> for x = 1:m
> stuff
> parfor y = 1:n
> a(y) = function(M,a(y));
> end
> stuff
> end
> Parfor is presently worse than 'for' because of the overhead from sending M every time.

You could use my worker object wrapper which is designed for exactly
this sort of situation. See


In your case, you could use it like this:

M = <big data>;
M = WorkerObjWrapper(M);

for ...
parfor ...
a(y) = someFunction(M.Value, ...);

By building M on the workers directly and building the WorkerObjWrapper
from the resulting Composite, the data is never actually transmitted
over the wire at any stage, so you should experience no problems with
the current 2GB transfer limit.



Point your RSS reader here for a feed of the latest messages in this topic.

[Privacy Policy] [Terms of Use]

© The Math Forum at NCTM 1994-2018. All Rights Reserved.