I have a big piece of never changing data that I'd like to be used by all the workers in a parallel setting. Since I'm only working with local workers, I don't understand why I have to eat the huge latency associated with transferring the data to workers each time the parfor loop is called. Can someone explain?
I tried to use spmd to send data once at the beginning and have it stay there, but the data is kind of big (~2 GB going to 10 workers), and I got "error during serialization" anyway. Is there a solution to this problem with local workers where they can all access the data from the same memory? Accesses are infrequent, so simultaneous access shouldn't cause a big slowdown.
Any ideas would be great.
My setup is something like this by the way:
M = big data; for x = 1:m stuff parfor y = 1:n a(y) = function(M,a(y)); end stuff end
Parfor is presently worse than 'for' because of the overhead from sending M every time.