Drexel dragonThe Math ForumDonate to the Math Forum



Search All of the Math Forum:

Views expressed in these public forums are not endorsed by Drexel University or The Math Forum.


Math Forum » Discussions » Software » comp.soft-sys.matlab

Topic: PCT output data transfer
Replies: 1   Last Post: Jul 15, 2014 2:41 AM

Advanced Search

Back to Topic List Back to Topic List Jump to Tree View Jump to Tree View  
Edric Ellis

Posts: 697
Registered: 12/7/04
Re: PCT output data transfer
Posted: Jul 15, 2014 2:41 AM
  Click to see the message monospaced in plain text Plain Text   Click to reply to this topic Reply

"Sujit Das" <SujitRDas@gmail.com> writes:
> Edric M Ellis <eellis@mathworks.com> wrote in message <ytw4myrarce.fsf@uk-eellis-deb7-64.dhcp.mathworks.com>...
> Thanks for sharing the experiment. The application basically sends a
> list of loan attributes for N number of loans (each worker gets N/W
> number of loans, W being the # of workers) to each worker along with
> some static economic environment data. The worker processes each loan
> given and produces some analytics that are sent back to the
> client. For the latest round of experiment, when N=100,000, W=400,
> pool closing time is 9 minutes. For N=850,000, P=500, pool closing
> time increased to 28 minutes!


Do you have any simple code that you can post here that reproduces the
problem? I still don't really have any good ideas as to why simply
closing the pool should take so long (other than the machine running
drastically low on RAM and having to swap processes out).

>> > Earlier in the thread we discussed how each worker transfers results
>> > back to client. Understanding is that each worker finishes processing
>> > all the slices before sending the result set to client via in-memory
>> > transfer.

>>
>> Workers in PARFOR loops get sent a chunk of slices - they work on that
>> chunk and then send the results back immediately using a socket
>> connection to the client. The PARFOR machinery attempts to split the
>> loop up into roughly 3 * NumWorkers chunks of slices.
>>

> I am little confused now. Earlier in the thread you had said:
> "In the PARFOR case, workers are given a group of slices to work on
> together, and return the results of the entire group rather than one
> slice at a time."
>
> So, for example, if a worker gets 3 slices to process, it finished one
> slice, produces the result and sends the result back to the client
> using a socket connection. Then it starts 2 slice and so on. Is that
> correct?


Yes, that is correct.

Cheers,

Edric.




Point your RSS reader here for a feed of the latest messages in this topic.

[Privacy Policy] [Terms of Use]

© Drexel University 1994-2014. All Rights Reserved.
The Math Forum is a research and educational enterprise of the Drexel University School of Education.