Search All of the Math Forum:
Views expressed in these public forums are not endorsed by
Drexel University or The Math Forum.
|
|
|
|
Re: give each lab one copy of my structure array ?
Posted:
Jul 16, 2012 4:09 AM
|
|
"Juliette Salexa" <juliette.physicist@gmail.com> writes:
> I'm back to this question now unfortunately ='(
>
> I'm using the suggestion of Edric M Ellis from 3 Jan 2012 (a couple posts above this):
>
> parfor idx = 1:(12*12)
> [i, j] = ind2sub([12, 12], idx);
> A(idx)=someFunction(B(i).someField,B(j).someField);
> end
>
> When the total data in the structure array B was about 1.5GB, the 12
> workers grabbed a copy of this 1.5GB structure array and applied
> calculations to it in parallel.
>
> Now I have 3.7GB of data and this is a disaster...
Unfortunately, it's not currently supported to transfer that much data
into a PARFOR loop.
> 12 workers opened, however, my task manager looks like this:
>
> 1. Main matlab client is using 11% CPU, and about 3.7GB of memory
> (which is fluctuating, since calculations are happening)
> 2. 12 matlab
> workers are open but using 0% CPU, and 142MB of memory (and the amount
> of memory being used is NOT changing AT ALL )
>
> Basically the whole calculation seems to be happening only on the main
> client and the workers don't seem to be doing anything ='(
That can happen if the data to be sent to the PARFOR loop cannot be
transferred because it is too large. You should have seen a warning
about this.
If your 'B' array is constant and the same value is needed on each
worker, what I'd suggest is using my Worker Object Wrapper
<http://www.mathworks.com/matlabcentral/fileexchange/31972-worker-object-wrapper>
like so:
% Build B on the workers so we never need to transfer it
spmd
if labindex == 1
B = <generate B>;
labBroadcast(1, B);
else
B = labBroadcast(1);
end
end
% Build a WorkerObjWrapper to wrap the Composite 'B'
Bw = WorkerObjWrapper(B);
% Use 'Bw.Value' inside PARFOR
parfor ...
B = Bw.Value;
<use B>;
end
While it's not ideal that you need to use this workaround for now, it
has the advantage that you never need to transfer 'B' from client to
workers. In particular, if you run multiple PARFOR loops, 'B' will be
there already.
If you have an encapsulated function to generate 'B', you can actually
build the WorkerObjWrapper directly like this:
Bw = WorkerObjWrapper( @buildBFcn, {} );
Hope this helps,
Edric.
|
|
|
|
