MATLAB and Condor
This describes how to create a MATLAB stand-alone executable and submit it to a Condor pool. MATLAB stand-alone executables are very useful because they run without using a licence.
- Either MATLAB or the MCR (MATLAB Compiler Runtime) must be must be installed on the target machine.
Some MATLAB and toolbox functionality is not licensed for compilation, follow this link for details.
- If your MATLAB executable runs on a Condor node which expects execution on a single CPU core, then it should be compiled with multi-threading switched off.
EPS Condor pool MATLAB on Condor tutorials.
Experiences with Running MATLAB jobs on a power-saving Condor Pool, Ian C. Smith, University of Liverpool.
Summary (for Linux)
Write a version of your MATLAB .m file that treats all input arguments as strings, e.g. using the MATLAB function str2num.
- Compile a standalone MATLAB executable by:
- Starting MATLAB (note issues with multi-threading described below).
Enter mcc -m -R -nojvm -R -singleCompThread <filename>.m to create files <filename> and run_<filename>.sh which are required on the Condor compute node to run the executable.
<filename> is the compiled executable
run_<filename>.sh is a script which when executed sets up the necessary environment variables before launching <filename>. The first argument to run_<filename>.sh should be the MATLAB path, additional arguments are passed to <filename>.
Omit the -R -singleCompThread for a multi-threaded version.
- Various other files are also created which are not required.
Create a new directory and copy <filename> and run_<filename>.sh to it.
- In the new directory write your Condor submission script which contains:
executable = run_<filename>.sh (since run_<filename>.sh must be executed first)
arguments = <MATLAB_path> <MATLAB_executable_arguments> (i.e. first argument is MATLAB path)
universe = vanilla
transfer_input_files = <filename> (to tell Condor to transfer <filename> to the remote node).
- Submit to Condor
Switching off multi-threading
- Multi-threading was introduced in MATLAB R2007a and is turned on by default as of MATLAB R2008a.
- If multi-threading is enabled when creating a standalone executable then multi-threading will be enabled for the executable.
- Multi-threading can be switched off for running on a single core by:
- Using the maxNumCompThreads function (available in R2007b through R2010a) to set the number of threads to 1.
By compiling with the -R -singleCompThread flag, e.g. mcc -R -singleCompThread -m my_matlab_function.m
- Setting the number of threads at the operating system level.
- As of June 2010 the University of Manchester has only 5 compiler licenses.
Using mcc from the command line
- mcc can be called from the command line. However this approach holds onto a compiler licence for 1/2 hour. THEREFORE DO NOT CALL mcc DIRECTLY FROM THE COMMAND LINE.
Use of mcc from within MATLAB releases the mcc licence immediately. Use mcc within MATLAB from the command line as follows matlab -nojvm -r "mcc -R -singleCompThread -m my_matlab_function.m; exit"
Problems you might encounter
Condor scripts containing getenv = true sometimes hit problems because MATLAB expects to find certain directories in your home folder that are not available on the remote Condor node. It may be possible to avoid this problem by setting the following environment variables:
- Set MATLAB_PREFDIR to a non-existent directory (see below).
- Set MCR_CACHE_ROOT to a non-existent directory (see below).
Setting environment for Condor
See links under the High Throughput Computing section here for more information.
MATLAB stand alone executables have been known to fail periodically, with messages like
Failed copying '/var/condor-6.8.8/execute/dir_15370/generate_seeds_mcr/generate_seeds_219EABB7AC26BF192B2A6A9EDE741EE2/matlab.prf' to '/mnt/nfs_R1L0/mcbicjb2/.matlab/mcr_v77/generate_seeds_219EABB7AC26BF192B2A6A9EDE741EE2/matlab.prf'. Should have written 671 bytes to destination file, but actually wrote 0 bytes Error initializing preferences required to run the application
This can be avoided by pointing the MATLAB_PREFDIR variable to a non-existent directory, e.g.
This solution is described here.
MATLAB stand alone executables may fail with Could not access the MCR component cache. error message because MATLAB is unable to access the MCE cache directory, which it expects to be in your home directory.
This can be avoided by pointing the MCR_CACHE_ROOT variable to a non-existent directory, e.g.
Note: these environment variables can be set from the Condor submit script, i.e. environment = MCR_CACHE_ROOT=/tmp/emptydir;MATLAB_PREFDIR=/tmp/emptydir