Parallel Simulink simulations: "failed to load library" only after starting a matlabpool (local)

Hi,

I am trying to run simulations of Simulink models in parallel. The function I use to simulate the model is below (part of a lager program). Now the problem is that I cannot run these simulations in parallel. If I run them without doing a

matlabpool open

it will work fine, but if I open a pool I will get such an error:

??? Error using ==> parallel_function at 598
Error in ==> Simulation>(parfor body) at 183
Failed to load library 'wteLib_waterSteam' referenced by 'KVA_physical_extraction/Water Steam Progress/Counterflow H20-H20 Boiler1'

Error in ==> Simulation>Simulation.parallel_simulations at 174
                parfor n=1:num_runs

Error in ==> MDPExtractor>MDPExtractor.initial_discovery at 252
            par_out = Simulation.parallel_simulations( ...

I guess that it is a scope/model-flattening error, but I am not sure how to fix it! Unfortunately I don't think I can share the actual Simulink model because it was partly produced by an industrial partner of my research project. In any case I don't think the problem is the model as such, but rather something to do with loading libraries in parfor loops... Any ideas?

Function that calls the parallel_simulations function:

          % To discover an initial state space, simulations
          % for every action are run from the default
          % simulink source state.
          function s = initial_discovery(s)
              us = {[0]};
              ts = {[0]};

              for  n=1:length(s.actions)
                  us{n} = s.actions(n,:);
                  tss = [0:1:s.epoch_time_steps-1];
                  ts{n} = tss;
              end

              par_out = Simulation.parallel_simulations( ...
                          s.model_name, ...
                          ts, ...
                          us, ...
                          s.epoch_time_steps, ...
                          length(s.actions));

              s.par_out = par_out;

Simulation function:

function [par_sim_out] = ...
            parallel_simulations( ...
                model_name, ...
                a_t, ...
                a_u, ...
                num_steps, ...
                num_runs, ...
                a_init_state)

              par_sim_out = cell([0]);

              load_system(model_name);
              % TODO: doesnt work yet...
              %rapid_acc_target = ...
              %Simulink.BlockDiagram.buildRapidAcceleratorTarget(...
              %    model_name);

              % config parameters
              conf = struct();
              conf.SaveOutput = 'on';
              conf.OutputSavename = 'save_out';

              conf.SaveFinalState = 'on';
              conf.SaveCompleteFinalSimState = 'on';
              conf.FinalStateName = 'save_final';

              conf.LoadExternalInput = 'on';
              conf.ExternalInput = '[gbl_t, gbl_u]';

              using_init_state = (nargin == 6);

              % Set initial state if it is passed to
              % the function. The stop-time also depends
              % on whether or not an initial state is used.
              % If an initial state is used the stop-time
              % must be adjusted because we no longer
              % start simulations at t=0, but rather at the
              % time the initial state save saved at.
              if using_init_state 
                  conf.LoadInitialState = 'on';
                  conf.InitialState = 'gbl_init_state';
              else
                  % Must create dummy var here, because
                  % the loop is created and vars assigned
                  % before it knows that it won't ever need
                  % this variable.
                  conf.StopTime = mat2str(num_steps);
              end

              % TODO: fix these double loops (scope error
              % with a_init_state) even when not used, because
              % of parfor loop prep.

              if using_init_state
                  parfor n=1:num_runs
                      assignin('base', 'gbl_t', a_t{n});
                      assignin('base', 'gbl_u', a_u{n});
                      lconf = conf;

                      assignin('base', 'gbl_init_state', ...
                              a_init_state);   
                      lconf.StopTime = mat2str( ...
                          a_init_state.snapshotTime + num_steps);

                      % simulate model
                      sim_out = sim(model_name, lconf);
                      outputs = [sim_out.get('save_out').signals.values]';
                      result_state = sim_out.get('save_final');
                  end
              else
                  parfor n=1:num_runs
                      assignin('base', 'gbl_t', a_t{n});
                      assignin('base', 'gbl_u', a_u{n});
                      fprintf(2,'a_t(%u): %s\n', n, mat2str(a_t{n}));
                      fprintf(2,'a_u(%u): %s\n', n, mat2str(a_u{n}));
                      lconf = conf;

                      % simulate model
                      sim_out = sim(model_name, lconf);
                      outputs = [sim_out.get('save_out').signals.values]';
                      fprintf(2,'outputs(%u): %s\n', n, mat2str(outputs));
                      result_state = sim_out.get('save_final');

                      par_sim_out{n} = {outputs, result_state};

                  end
              end
              % Run parallel simulations.
          end % accelerated_parallel_simulations

Thank You!

Oliver

Answer 1

[Edric Ellis]

You need to give the workers access to the libraries referenced by the Simulink model. If you do not have a shared filesystem between your MATLAB client and the cluster workers, you can use the 'addfiledependencies' option to MATLABPOOL to get the files transferred. See the MATLABPOOL reference page for more details.

[Oliver]

Hey Edric,

thanks for you answer. I guess it is something to do with this. But should this be a problem with a local cluster (ie. multiple cores on one machine)?

Aside: I am using a pretty complex Simulink model (unfortunately), so do you know an automatic way of doing this dependency stuff (at least getting a list of required files)?

Thanks!

[Edric Ellis]

If you're using the localscheduler, then matlabpool attempts to synchronize your MATLAB path between the client and the workers. You could try running

which wteLib_waterSteam

On the client, and also inside an SPMD block to investigate what's happening. (i.e. spmd, which( 'wteLib_waterSteam' ), end ).

[Oliver]

Ok thanks. I will try it tomorrow!