"Sujit Das" <SujitRDas@gmail.com> writes: > Edric M Ellis <email@example.com> wrote in message <firstname.lastname@example.org>... >> "Sujit Das" <SujitRDas@gmail.com> writes: >> >> > Interesting. Yes, we are using PARFOR in our application. The size of >> > result set that each worker (Linux) needs to return to client (Wintel >> > or Linux) could be as large as 1 to 2 GB. How does the worker return >> > this result set (which are in-memory Matlab variables) to client? Is >> > the transfer taking place directly from worker memory to client memory >> > (RMI??) without ever hitting the file or disk I/O unlike what happens >> > in the batch jobs? >> >> Yes, when results are returned from a PARFOR loop, a type of RMI is used >> to transfer the data from the workers back to the client, no on-disk >> storage is used. However, note that there is some overhead to doing this >> - the MATLAB data needs to be turned into a serialized form for >> transmission - this is part of the same process that happens when >> variables are saved to disk, so if for example you were transmitting an >> object with a very slow SAVEOBJ, then you'd see that slowness. >> > Thanks Edric. That is very insightful. Yes, we realize that data > serialization does happen during transfer. We have also encountered > the limit of 2 GB ( I believe) in 64-bit workers during serialization.
If you have 64-bit workers and a 64-bit client, the limit no longer applies (it was removed in R2013a).