"Sujit Das" <SujitRDas@gmail.com> writes: > Edric M Ellis <firstname.lastname@example.org> wrote in message <email@example.com>... > Thanks for sharing the experiment. The application basically sends a > list of loan attributes for N number of loans (each worker gets N/W > number of loans, W being the # of workers) to each worker along with > some static economic environment data. The worker processes each loan > given and produces some analytics that are sent back to the > client. For the latest round of experiment, when N=100,000, W=400, > pool closing time is 9 minutes. For N=850,000, P=500, pool closing > time increased to 28 minutes!
Do you have any simple code that you can post here that reproduces the problem? I still don't really have any good ideas as to why simply closing the pool should take so long (other than the machine running drastically low on RAM and having to swap processes out).
>> > Earlier in the thread we discussed how each worker transfers results >> > back to client. Understanding is that each worker finishes processing >> > all the slices before sending the result set to client via in-memory >> > transfer. >> >> Workers in PARFOR loops get sent a chunk of slices - they work on that >> chunk and then send the results back immediately using a socket >> connection to the client. The PARFOR machinery attempts to split the >> loop up into roughly 3 * NumWorkers chunks of slices. >> > I am little confused now. Earlier in the thread you had said: > "In the PARFOR case, workers are given a group of slices to work on > together, and return the results of the entire group rather than one > slice at a time." > > So, for example, if a worker gets 3 slices to process, it finished one > slice, produces the result and sends the result back to the client > using a socket connection. Then it starts 2 slice and so on. Is that > correct?