Migrating Old Job Scripts

If you were a user of BlueBEAR prior to June 2017, then you are probably familiar with writing your job scripts for the old MOAB scheduler. This page details how to migrate your job scripts from MOAB to Slurm.

You can still use msub to submit your old jobs using:

$ msub job.sh

This is a wrapper script that will convert the job script to work with Slurm under-the-hood. This feature will become deprecated in the coming months.

Below, is a script moabtoslurm which you can run yourself to convert old job scripts to Slurm.

Script: moabtoslurm

To convert a job script with #MOAB headers to a job script with #SBATCH headers, run the command:

$ moabtoslurm <moab_job_script> > <slurm_job_script>

where <moab_job_script> is the name of the script you wish to convert and <slurm_job_script> is the name of what you would like your Slurm script to be.

For example, if you have a script matlab.moab:

#!/bin/bash
#MOAB -l 'nodes=1:ppn=1,walltime=5:0'
#MOAB -j oe
#MOAB -N cvx-test
#MOAB -q bbtest

cd "$PBS_O_WORKDIR"
module load apps/matlab/r2016a

matlab -nodisplay -r cvxtest

Then issuing the command:

$ moabtoslurm matlab.moab > matlab.slurm

Produces a file matlab.slurm which contains:

#!/bin/bash
#SBATCH --cpus-per-task 1 --ntasks 1 -t 5:0
#SBATCH --job-name cvx-test
#SBATCH --qos bbdefault
cd "$SLURM_SUBMIT_DIR"
module load apps/matlab/r2016a

matlab -nodisplay -r cvxtest

This can then be submitted to the batch system using:

$ sbatch matlab.slurm

Converting MPI Jobs

Environment variables used by MOAB/Torque such as those prefixed with 'PBS_' will not be recognised by Slurm. This will cause issues if you relied on $PBS_NODEFILE for node information during jobs. The moabtoslurm script will warn you if these are present in your job script.

For example, if you have a script abaqus.moab:

#!/bin/bash
#MOAB -l walltime=30:0
#MOAB -j oe
#MOAB -q bbdefault

cd "$PBS_O_WORKDIR"

if [ $PBS_NODEFILE ]
then
   ncores=$(cat $PBS_NODEFILE|wc -l)
else
   ncores=1
fi

module purge; module load bluebear
module load apps/abaqus/r2017

abaqus job=myjob cpus=$ncores

Then issuing the command:

$ moabtoslurm abaqus.moab > abaqus.slurm

Produces an incorrect file abaqus.slurm, which contains:

#!/bin/bash
#SBATCH -t 30:0
#SBATCH --qos bbdefault
cd "$SLURM_SUBMIT_DIR"

if [ $PBS_NODEFILE ]
then
   ncores=$(cat $PBS_NODEFILE|wc -l)
else
   ncores=1
fi

module purge; module load bluebear
module load apps/abaqus/r2017


abaqus job=myjob cpus=$ncores

Note, $PBS_O_WORKDIR is converted to $SLURM_SUBMIT_DIR.

Warnings will be output to stderr for environment variables like $PBS_NODEFILE which are not available in Slurm:

[WARNING] This line uses environment variables not available in sbatch.
[WARNING] if [ $PBS_NODEFILE ]
[WARNING] Your job script may not function as you expect.
[WARNING] Using sbatch directly is recommended in this case.

[WARNING] This line uses environment variables not available in sbatch.
[WARNING] ncores=$(cat $PBS_NODEFILE|wc -l)
[WARNING] Your job script may not function as you expect.
[WARNING] Using sbatch directly is recommended in this case.

The correct script would be:

#!/bin/bash
#SBATCH -t 30:0
#SBATCH --qos bbdefault
cd "$SLURM_SUBMIT_DIR"

module load apps/abaqus/v6.14-1
abaqus job=myjob cpus=${SLURM_NTASKS:=1}

The statement ${SLURM_NTASKS:=1} will set cpus to be SLURM_NTASKS if running in batch (as the environment variable will exist), otherwise it will be set to 1 (such as if you are running the script locally).

Some software requires a nodefile. This can be generated by including the following line in your job script:

NODEFILE=$(make_nodefile)

You can also get the number of cores and nodes allocated by including the following in your job script:

nnodes=${SLURM_JOB_NUM_NODES}
ncores=${SLURM_NTASKS}

Job Script Headers

Table showing the SBATCH equivalent for common MOAB headers.
 MOAB SBATCH
Queue: -q

QOS (Queue): --qos

e.g.

  • #SBATCH --qos bbshort
Walltime: -l walltime=[hh:mm:ss]

Walltime: --time [min] OR --time [days-hh:mm:ss]

 

e.g.

  • #SBATCH --time 10 # = 10 minutes
  • #SBATCH --time 30:0 # = 30 minutes
  • #SBATCH --time 1:0:0 # = 1 hour
  • #SBATCH --time 1-0:0:0 # = 1 day
 Nodes: -l nodes=[count]

 Nodes: --nodes [min[-max]]

e.g.

  • #SBATCH --nodes 1 # = 1 node
  • #SBATCH --nodes 1-2 # = min 1 node, max 2 nodes
Cores: -l ppn=[count]

Cores: --ntasks [count]

e.g.

  • #SBATCH --ntasks 16 # = 16 cores

Standard Output file: -o [file-name]

Standard Error file: -e [file-name]

Combine stdout/stderr: -j oe

Standard Output file: --output [file-name]

Standard Error file: --error [file-name]

Combine stdout/stderr: Use --output without --error

e.g.

  • #SBATCH --output output-%j.out # combined output/error file called output-12345.out (if the job id is 12345).

Memory Size: -l mem=[MB]

Memory Size: --mem=[mem][M|G|T]

e.g.

  • #SBATCH --mem=10G

Memory Size per CPU: --mem-per-cpu=[mem][M|G|T]

e.g.

  • #SBATCH --mem-per-cpu=300M

Note: If you are running a job on Sandybridge then you should specify:

#SBATCH --mem-per-cpu=1850M

Job Name: -N [name]

Job Name: --job-name=[name]

e.g.

  • #SBATCH --job-name=testJob
The Slurm documentation for sbatch will describe all options that are recognised and their short forms. If you are looking for something particular then do a page search (Ctrl+F) for keywords on that documentation page. Raise a ticket with the IT Service Desk if you would like assistance with choosing the right options for your job needs.

Last modified: 12 October 2017