Search All of the Math Forum:
Views expressed in these public forums are not endorsed by
Drexel University or The Math Forum.



Re: spmd and parfor with large broadcast variables
Posted:
Jun 15, 2012 2:55 AM


"Benjamin " <bfiles@usc.edu> writes:
> I'm new to the idea of parallel computing with matlab on a cluster. I > understand that using broadcast variables, like > > matlabpool 10 > bigd = loadBigData(); % loads a large matrix of class double > r = zeros(100,1); > parfor ii = 1:100, > r(ii) = doSomething(ii,bigd); > end > > causes a lot of extra network traffic. My first question is whether > the variable bigd is sent over the network 10 times (i.e. once per > lab) or 100 times (once per loop iteration).
Data broadcast into a PARFOR loop is sent over once per lab  however, even that is possibly more than ideal. You could for example use my Worker Object Wrapper
<http://www.mathworks.com/matlabcentral/fileexchange/31972>
to minimise the data transfer. If 'loadBigData' can run on the labs (i.e. has access to the data files), you could do:
bigdWrapper = WorkerObjWrapper(@loadBigData, {}); r = zeros(100,1); parfor ii = 1:100, r(ii) = doSomething(ii,bigdWrapper.Value); end
otherwise create bigdWrapper like so:
bigdWrapper = WorkerObjWrapper(loadBigData());
Cheers,
Edric.



