PBS Job Script

Before we jump to a job script, here are some basic stuff.

Node: a (blade) server that has multiple cores (It's actually a network terminology, but that's fine. Tardis has 48 cores/node.)

PBS: Portable batch system that arranges jobs between users

Scratch disk: temporary disk space used to save raw outputs from MPI jobs

PBS commands

$> qsub job.sh # submit a job $> qstat -nas # check a job status $> qdel JOB_ID # cancel a job $> qstat -u USER_ID # check job status of USER $> pbsnodes -aSjUSER_ID # check job status of USER

Example job script (MPI-only jobs)

#PBS -q workq #PBS -N MyJobName #PBS -S /bin/bash #PBS -l nodes=3:ppn=48 #PBS -l walltime=6:00:00 #PBS -V #NCORE=`cat $PBS_NODEFILE | wc -l` cd $PBS_O_WORKDIR # module setting module purge module load intel20/compiler-20 module load intel20/mvapich2-2.3.4 logfile=log$(date +%Y%m%d%H%M).out NCPU=144 mpirun -machinefile $PBS_NODEFILE -np ${NCPU} ./ramses3d params.nml > ${logfile} # More example scripts can be found in /storage7/pbs_examples/*.sh

Using a specific number of cores in MPI-only jobs

#PBS -l select=6:ncpus=32:mpiprocs=16

-> Occupy 6 nodes with 32 cores, but run only 16 processes/node. This will be useful if you want to have a large mem per core. Altogether, this will use 32x6=192 cores.

Suppose that 32 cores are being used in 6 nodes. In this case, 16 cores are available in each node, which means 16x6=96 cores are free. If you want to use all of these 96 cores, you should submit with #PBS -l select=6:ncpus=16:mpiprocs=16 not #PBS -l select=2:ncpus=48:mpiprocs=48

Using a specific number of cores in MPI+OpenMP jobs

#PBS -l select=4:ncpus=16:mpiprocs=4:ompthreads=4

-> Occupy 4 nodes with 16 cores and run 4 processes/node. But each process will use 4 OMP threads, making the total number of threads 64.