Why is the performance of my distributed computing cluster decreasing over time in MATLAB Distributed Computing Server?

I am working with Parallel Computing Toolbox and MATLAB Distributed Computing Server. I am running a large number of jobs and tasks in order to perform genetic algorithms and direct search optimization. The code runs without errors, however I noticed after time the CPU usage on the workers is decreasing and the overall performance is getting lower and lower. I have no explanation to this behavior, as all the jobs and tasks are correctly finished. I would like to know what is causing this behavior.

Answer 1

[MathWorks Support Team]

When creating many jobs and tasks you should verify that they are finally destroyed after they are finished, otherwise their remains (e.g. FileDependencies that are copied to the workers) may swamp the worker over time.

Note that when a job is finished, it remains in the job manager or DataLocation directory, even if you clear all the objects from the client session. The job manager or scheduler keeps all the jobs it has executed, until you restart the job manager in a clean state. Therefore, you can retrieve information from a job at a later time or in another client session, so long as the job manager has not been restarted with the -clean option.

To permanently remove completed jobs from the job manager or scheduler's data location, use the destroy function.