If you can stand storing it N times, then my persistent variable trick seems to work. See my previous post in this thread.
"Haoran Xu" <email@example.com> wrote in message <firstname.lastname@example.org>... > The spmd error is: > Error using distcompserialize > Error during serialization > So is this because the data is too large? Then how to solve this? > Edric M Ellis <email@example.com> wrote in message <firstname.lastname@example.org>... > > "Chuck37 " <email@example.com> writes: > > > > > I have a big piece of never changing data that I'd like to be used by > > > all the workers in a parallel setting. Since I'm only working with > > > local workers, I don't understand why I have to eat the huge latency > > > associated with transferring the data to workers each time the parfor > > > loop is called. Can someone explain? > > > > > > I tried to use spmd to send data once at the beginning and have it > > > stay there, but the data is kind of big (~2 GB going to 10 workers), > > > and I got "error during serialization" anyway. Is there a solution to > > > this problem with local workers where they can all access the data > > > from the same memory? Accesses are infrequent, so simultaneous access > > > shouldn't cause a big slowdown. > > > > > > Any ideas would be great. > > > > > > My setup is something like this by the way: > > > > > > M = big data; > > > for x = 1:m > > > stuff > > > parfor y = 1:n > > > a(y) = function(M,a(y)); > > > end > > > stuff > > > end > > > > > > Parfor is presently worse than 'for' because of the overhead from sending M every time. > > > > You could use my worker object wrapper which is designed for exactly > > this sort of situation. See > > > > <http://www.mathworks.com/matlabcentral/fileexchange/31972-worker-object-wrapper> > > > > In your case, you could use it like this: > > > > spmd > > M = <big data>; > > end > > M = WorkerObjWrapper(M); > > > > for ... > > parfor ... > > a(y) = someFunction(M.Value, ...); > > end > > end > > > > By building M on the workers directly and building the WorkerObjWrapper > > from the resulting Composite, the data is never actually transmitted > > over the wire at any stage, so you should experience no problems with > > the current 2GB transfer limit. > > > > Cheers, > > > > Edric.