Drexel dragonThe Math ForumDonate to the Math Forum



Search All of the Math Forum:

Views expressed in these public forums are not endorsed by Drexel University or The Math Forum.


Math Forum » Discussions » Software » comp.soft-sys.matlab

Topic: PCT output data transfer
Replies: 2   Last Post: Jun 24, 2014 12:07 PM

Advanced Search

Back to Topic List Back to Topic List Jump to Tree View Jump to Tree View   Messages: [ Previous | Next ]
Edric Ellis

Posts: 692
Registered: 12/7/04
Re: PCT output data transfer
Posted: Jun 24, 2014 2:43 AM
  Click to see the message monospaced in plain text Plain Text   Click to reply to this topic Reply

"Sujit Das" <SujitRDas@gmail.com> writes:

> Edric M Ellis <eellis@mathworks.com> wrote in message <ytwionsf5cv.fsf@uk-eellis-deb7-64.dhcp.mathworks.com>...
>> "Sujit Das" <SujitRDas@gmail.com> writes:
>>

>> > We are using PCT and experience slowness in transfer of result data
>> > from workers to clients. Does anyone know how Matlab transfers output
>> > data from workers to clients in PCT? Does it save the output data to
>> > (temp) files and then use FTP or some other file transfer mechanism to
>> > transfer to client?

>>
>> Which cluster type are you using?
>>
>> MJS (jobmanager) stores result data in a database on the host where the
>> jobmanager itself is running, and the data is sent over the network. All
>> other schedulers use the file-system in one way or another. (Most save
>> results directly to disk, but some of the third-party scheduler
>> integrations using the "generic" cluster can use SCP to transfer data).
>>

> Thanks Edric. We use both MJS (for Wintel) and LFS for linux grid. Is this
> mechanism of transfer documented somewhere by Mathworks? For large results the
> mechanism of writing to disk (files or DB) and then transferring over the
> networks could be very inefficient. Another question: Does a worker send back
> the results back to the client as soon as a slice is done? I assume that is the
> case. If not, then all the workers will have to be done producing the results
> before the results are sent back to the client.


The format is not explicitly documented - although for everything other
than MJS, standard .MAT files are used.

Also, note that this storage is used only for results of running a BATCH
job (or other job created using one of the create*Job methods). If
you're talking about parallel pools and PARFOR etc., then there is no
on-disk / database storage of results at all (you mentioned 'slices'
which made me think that's what you're referring to).

In the PARFOR case, workers are given a group of slices to work on
together, and return the results of the entire group rather than one
slice at a time.

Cheers,

Edric.



Point your RSS reader here for a feed of the latest messages in this topic.

[Privacy Policy] [Terms of Use]

© Drexel University 1994-2014. All Rights Reserved.
The Math Forum is a research and educational enterprise of the Drexel University School of Education.