The Math Forum

Search All of the Math Forum:

Views expressed in these public forums are not endorsed by NCTM or The Math Forum.

Math Forum » Discussions » Software » comp.soft-sys.matlab

Notice: We are no longer accepting new posts, but the forums will continue to be readable.

Topic: Sharing big data across parfor or spmd
Replies: 8   Last Post: Aug 6, 2013 4:48 AM

Advanced Search

Back to Topic List Back to Topic List Jump to Tree View Jump to Tree View   Messages: [ Previous | Next ]

Posts: 95
Registered: 2/23/10
Sharing big data across parfor or spmd
Posted: Dec 11, 2012 1:09 PM
  Click to see the message monospaced in plain text Plain Text   Click to reply to this topic Reply

I have a big piece of never changing data that I'd like to be used by all the workers in a parallel setting. Since I'm only working with local workers, I don't understand why I have to eat the huge latency associated with transferring the data to workers each time the parfor loop is called. Can someone explain?

I tried to use spmd to send data once at the beginning and have it stay there, but the data is kind of big (~2 GB going to 10 workers), and I got "error during serialization" anyway. Is there a solution to this problem with local workers where they can all access the data from the same memory? Accesses are infrequent, so simultaneous access shouldn't cause a big slowdown.

Any ideas would be great.

My setup is something like this by the way:

M = big data;
for x = 1:m
parfor y = 1:n
a(y) = function(M,a(y));

Parfor is presently worse than 'for' because of the overhead from sending M every time.

Point your RSS reader here for a feed of the latest messages in this topic.

[Privacy Policy] [Terms of Use]

© The Math Forum at NCTM 1994-2018. All Rights Reserved.