Difference between revisions of "HOWTO submit jobs"

From HPC
m (→‎pisoFOAM)
m (→‎Setup)
 
(124 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 +
== Jobs ==
 +
 +
The HPC implements a batch queueing system called PBS. Any work you want to do needs to be packaged as a job that the system can do without any interaction from you. Because it's a queue, and the queue might be long, your job might only start a couple of hours from now. It's important to have your script completely self-contained so that it can run without supervision, otherwise, it might just error and all that time waiting in the queue will be wasted.
 +
 
== Submitting jobs ==
 
== Submitting jobs ==
  
Line 5: Line 9:
 
Let's say I have an executable called "myprog". Let me try and submit it to PBS:
 
Let's say I have an executable called "myprog". Let me try and submit it to PBS:
 
<pre>
 
<pre>
−
[username@launch ~]$ qsub myprog
+
[username@hpc1 ~]$ qsub myprog
 
qsub:  file must be an ascii script
 
qsub:  file must be an ascii script
 
</pre>
 
</pre>
 
Oops... That didn't work because qsub expects a shell script. Any shell should work, so use your favorite one. So I write a simple script called "myscript.sh"
 
Oops... That didn't work because qsub expects a shell script. Any shell should work, so use your favorite one. So I write a simple script called "myscript.sh"
 +
<syntaxhighlight lang="bash">
 +
#!/bin/bash
 +
cd $PBS_O_WORKDIR
 +
./myprog argument1 argument2
 +
</syntaxhighlight>
 +
and then I submit it:
 
<pre>
 
<pre>
 +
[username@hpc1 ~]$ qsub myscript.sh
 +
qsub: Job has no walltime requested. Request walltime with '-l walltime=HH:MM:SS'.
 +
</pre>
 +
 +
PBS wants to know how long every job is expected to run. This is so that it can schedule jobs optimally. Updating the script to include the PBS directive '-l walltime', we get the script
 +
<syntaxhighlight lang="bash">
 
#!/bin/bash
 
#!/bin/bash
 +
#PBS -l walltime=00:05:00
 
cd $PBS_O_WORKDIR
 
cd $PBS_O_WORKDIR
 
./myprog argument1 argument2
 
./myprog argument1 argument2
−
</pre>
+
</syntaxhighlight>
−
and then I submit it:
+
which we then submit:
 
<pre>
 
<pre>
−
[username@launch ~]$ qsub myscript.sh
+
[username@hpc1 ~]$ qsub myscript.sh
−
16.pbsserver.hpc
+
16.hpc1.hpc
 
</pre>
 
</pre>
  
−
That worked! Note the use of the <code>$PBS_O_WORKDIR</code> environment variable. This is important, since by default PBS on our cluster will start executing the commands in your shell script from your home directory. To go to the directory in which you executed <code>qsub</code>, <code>cd</code> to <code>$PBS_O_WORKDIR</code>. There are several other useful PBS environment variables that we will encounter later.
+
That worked! Note the use of the <code>$PBS_O_WORKDIR</code> environment variable. This is important since by default PBS on our cluster will start executing the commands in your shell script from your home directory. To go to the directory in which you executed <code>qsub</code>, <code>cd</code> to <code>$PBS_O_WORKDIR</code>. There are several other useful PBS environment variables that we will encounter later.
 +
 
 +
In this script, we also informed PBS that the job will be running for a maximum of 5 minutes.
  
 
=== Editing files ===
 
=== Editing files ===
Line 36: Line 55:
 
  $ dos2unix script.sub
 
  $ dos2unix script.sub
 
  dos2unix: converting file script.sub to UNIX format ...
 
  dos2unix: converting file script.sub to UNIX format ...
 +
 +
If you're using MobaXterm's default text editor, make sure your file is in the correct format by selecting UNIX in the Format menu, or checking that the penguin icon in the button bar is selected.
  
 
=== Specifying job parameters ===
 
=== Specifying job parameters ===
−
By default, any script you submit will run on a single processor for a maximum of 5 minutes. The name of the job will be the name of the script, and it will not email you when it starts, finishes, or is interrupted. stdout and stderr are collected into separate files named after the job number.
+
By default, any script you submit will run on a single processor with 1GB of memory. The name of the job will be the name of the script, and it will not email you when it starts, finishes, or is interrupted. stdout and stderr are collected into separate files named after the job number.
 
You can affect the default behaviour of PBS by passing it parameters. These parameters can be specified on the command line or inside the shell script itself. For example, let's say I want to send stdout and stderr to a file that is different from the default:
 
You can affect the default behaviour of PBS by passing it parameters. These parameters can be specified on the command line or inside the shell script itself. For example, let's say I want to send stdout and stderr to a file that is different from the default:
 
<pre>
 
<pre>
−
[username@launch ~]$ qsub -e myprog.err -o myprog.out myscript.sh
+
[username@hpc1 ~]$ qsub -e myprog.err -o myprog.out myscript.sh
 
</pre>
 
</pre>
  
 
Alternatively, I can actually edit myscript.sh to include these parameters. I can specify any PBS command line parameter I want in a line that begins with "#PBS":
 
Alternatively, I can actually edit myscript.sh to include these parameters. I can specify any PBS command line parameter I want in a line that begins with "#PBS":
−
<pre>
+
<syntaxhighlight lang="bash">
 
#!/bin/bash
 
#!/bin/bash
 +
#PBS -l walltime=00:05:00
 
#PBS -e myprog.err
 
#PBS -e myprog.err
 
#PBS -o myprog.out
 
#PBS -o myprog.out
 
cd $PBS_O_WORKDIR
 
cd $PBS_O_WORKDIR
 
./myprog argument1 argument2
 
./myprog argument1 argument2
−
</pre>
+
</syntaxhighlight>
  
 
Now I just submit my modified script with no command-line arguments
 
Now I just submit my modified script with no command-line arguments
 
<pre>
 
<pre>
−
[username@launch ~]$ qsub myscript.csh
+
[username@hpc1 ~]$ qsub myscript.sh
 
</pre>
 
</pre>
  
 
=== Useful PBS parameters ===
 
=== Useful PBS parameters ===
−
Here is an example of a more involved script that requests only 1 hour of execution time, renames the job, and sends email when the job begins, ends, or aborts:
+
Here is an example of a more involved script that requests only 1 hour of execution time, renames the job to 'My-Program', and sends email when the job begins, ends, or aborts:
−
<pre>
+
<syntaxhighlight lang="bash">
 
#!/bin/bash
 
#!/bin/bash
 
   
 
   
Line 85: Line 107:
 
   
 
   
 
echo Done!
 
echo Done!
−
</pre>
+
</syntaxhighlight>
  
 
Some more useful PBS parameters:
 
Some more useful PBS parameters:
Line 92: Line 114:
 
* -V: export all your environment variables to the batch job.
 
* -V: export all your environment variables to the batch job.
 
* -I: run an interactive job (see below).
 
* -I: run an interactive job (see below).
−
Once again, you are encouraged to consult the qsub manpage for more options.
+
Once again, you are encouraged to consult the <code>qsub</code> manpage for more options.
  
−
=== Special concerns for running OpenMP programs ===
+
=== Interactive jobs ===
−
By default, PBS assigns you 1 core on 1 node. You can, however, run your job on up to 64 cores per node. Therefore, if you want to run an OpenMP program, you must specify the number of processors per node. This is done with the flag <code>-l select=1:ncpus=<cores></code> where <code><cores></code> is the number of OpenMP threads you wish to use.
+
 
−
Keep in mind that you still must set the OMP_NUM_THREADS environment variable within your script, e.g.:
+
Normally when you call <code>qsub</code> PBS grabs your script and goes away. You get the prompt returned to you almost immediately and whatever output is produced from your job goes into a designated file. But what if you want to run an interactive job, say to compile something? This is not, in general, the right way to use an HPC, because interactive jobs waste CPU time, mostly because humans are so slow.
−
<pre>
+
 
−
#!/bin/bash
+
<syntaxhighlight lang="bash">
−
#PBS -N My-OpenMP-Script
+
[username@hpc1 ~]$ qsub -I -l walltime=2:00:00
−
#PBS -l select=1:ncpus=8
+
qsub: waiting for job 1695.hpc1.hpc to start
−
#PBS -l walltime=1:00:00
+
qsub: job 1695.hpc1.hpc ready
−
+
 
−
cd $PBS_O_WORKDIR
+
[username@comp007 ~]$ logout
−
export OMP_NUM_THREADS=8
+
 
−
./MyOpenMPProgram
+
qsub: job 1695.hpc1.hpc completed
−
</pre>
+
[username@hpc1 ~]$
 +
</syntaxhighlight>
 +
 
 +
In the example above I requested an interactive job, using the defaults of 1 core and 1GB memory and requesting it for 2 hours. If the system can honour the request, I'm immediately given an SSH session to the compute node assigned to my job. Closing the SSH session ends the job, and I'm returned to the login node.
 +
 
 +
You can also use the <code>qsubi</code> script, which will request an interactive session for 24 hours on a node with 4 cores and 14GB RAM.
 +
 
 +
<syntaxhighlight lang="bash">
 +
[username@hpc1 ~]$ qsubi
 +
Requesting interactive session with 4 cores, 14GB RAM, for 24 hours
 +
qsub: waiting for job 1696.hpc1.hpc to start
 +
qsub: job 1696.hpc1.hpc ready
 +
 
 +
[username@comp008 ~]$ logout
 +
 
 +
qsub: job 1696.hpc1.hpc completed
 +
[username@hpc1 ~]$
 +
</syntaxhighlight>
  
 
=== Jobs with large output files ===
 
=== Jobs with large output files ===
 +
 +
If your job makes frequent writes to disk, it may benefit from using scratch space instead of working directly from the home directory.
  
 
Instead of a job submission like this:
 
Instead of a job submission like this:
−
<pre>
+
<syntaxhighlight lang="bash">
 
#!/bin/bash
 
#!/bin/bash
 
#PBS -N massiveJob
 
#PBS -N massiveJob
 +
#PBS -l walltime=01:00:00
  
−
cd $PBS_O_WORKDIR
+
cd ${PBS_O_WORKDIR}
−
myprogram -i /home/me/inputfile -o /home/me/outputfile
+
myprogram -i /home/username/inputfile -o /home/username/outputfile
−
</pre>
+
</syntaxhighlight>
 
change it to something like this:
 
change it to something like this:
−
<pre>
+
<syntaxhighlight lang="bash">
 
#!/bin/bash
 
#!/bin/bash
−
#PBS -l select=1:ncpus=1:scratch=true
+
#PBS -l select=1:ncpus=1
 +
#PBS -l walltime=1:00:00
 
#PBS -N massiveJob
 
#PBS -N massiveJob
  
 
# make sure I'm the only one that can read my output
 
# make sure I'm the only one that can read my output
 
umask 0077
 
umask 0077
−
# create a temporary directory with the job ID as name in /scratch
+
# create a temporary directory with the job ID as name in /scratch-small-local
−
TMP=/scratch/${PBS_JOBID}
+
TMP=/scratch-small-local/${PBS_JOBID}
 
mkdir -p ${TMP}
 
mkdir -p ${TMP}
 
echo "Temporary work dir: ${TMP}"
 
echo "Temporary work dir: ${TMP}"
Line 144: Line 187:
 
/usr/bin/rsync -vax ${TMP}/ "${PBS_O_WORKDIR}/"
 
/usr/bin/rsync -vax ${TMP}/ "${PBS_O_WORKDIR}/"
  
−
# delete my temporary files
+
# if the copy back succeeded, delete my temporary files
 
[ $? -eq 0 ] && /bin/rm -rf ${TMP}
 
[ $? -eq 0 ] && /bin/rm -rf ${TMP}
 +
</syntaxhighlight>
  
−
</pre>
+
Any job that has to write massive amounts of data will benefit from the above. If your job doesn't complete successfully or you don't clean up your temporary files, your temporary files will be moved to the <code>/orphans</code> directory. If you can't find your output, go look for it there.
  
−
Any job that has to write massive amounts of data will benefit from the above. Take note of the <code>:scratch=true</code> that was added to the node request line. If you do not add that feature request to the script, your job may be assigned to a node without scratch space.
+
There are two scratch spaces. One is the per-node <code>/scratch-small-local</code> directory, the other is the global <code>/scratch-large-network</code> directory. If you need to write more than 200GB of output, it's probably better to use the larger, network space.
 +
 
 +
=== Running OpenMP multi-threaded programs ===
 +
By default, PBS assigns you 1 core on 1 node. You can, however, run your job on up to 64 cores per node. Therefore, if you want to run an OpenMP program, you must specify the number of processors per node. This is done with the flag <code>-l select=1:ncpus=<cores></code> where <code><cores></code> is the number of OpenMP threads you wish to use.
 +
Keep in mind that you still must set the OMP_NUM_THREADS environment variable within your script, e.g.:
 +
<syntaxhighlight lang="bash">
 +
#!/bin/bash
 +
#PBS -N My-OpenMP-Script
 +
#PBS -l select=1:ncpus=8
 +
#PBS -l walltime=1:00:00
 +
 +
cd ${PBS_O_WORKDIR}
 +
export OMP_NUM_THREADS=8
 +
./MyOpenMPProgram
 +
</syntaxhighlight>
  
 
=== Using the PBS_NODEFILE for multi-threaded jobs ===
 
=== Using the PBS_NODEFILE for multi-threaded jobs ===
−
Until now, we have only dealt with serial jobs. In a serial job, your PBS script will automatically be executed on the target node assigned by the scheduler. If you asked for more than one node, however, your script will only execute on the first node of the set of nodes allocated to you. To access the remainder of the nodes, you must either use MPI or manually launch threads. But which nodes to run on? PBS gives you a list of nodes in a file at the location pointed to by the <code>PBS_NODEFILE</code> environment variable.
+
In a serial job, your PBS script will automatically be executed on the target node assigned by the scheduler. If you asked for more than one node, however, your script will only execute on the first node of the set of nodes allocated to you. To access the remainder of the nodes, you must either use MPI or manually launch threads. But which nodes to run on? PBS gives you a list of nodes in a file at the location pointed to by the <code>PBS_NODEFILE</code> environment variable.
 
In your shell script, you may thereby ascertain the nodes on which your job can run by looking at the file in the location specified by this variable:
 
In your shell script, you may thereby ascertain the nodes on which your job can run by looking at the file in the location specified by this variable:
−
<pre>
+
<syntaxhighlight lang="bash">
 
#!/bin/bash
 
#!/bin/bash
−
#PBS -l select=2:mpiprocs=8
+
#PBS -l walltime=1:00:00
 +
#PBS -l select=2:ncpus=8:mpiprocs=8
  
−
echo The nodefile for this job is stored at `echo $PBS_NODEFILE`
+
echo "The nodefile for this job is stored at ${PBS_NODEFILE}"
−
echo `cat $PBS_NODEFILE`
+
cat ${PBS_NODEFILE}
−
</pre>
+
np=$(wc -l < ${PBS_NODEFILE})
 +
echo "Cores assigned: ${np}"
 +
</syntaxhighlight>
 
When you run this job, you should then get output similar to:
 
When you run this job, you should then get output similar to:
 
<pre>
 
<pre>
−
The nodefile for this job is stored at /var/spool/PBS/aux/33.pbsserver.hpc
+
The nodefile for this job is stored at /var/spool/PBS/aux/33.hpc1.hpc
 
comp001.hpc
 
comp001.hpc
 
comp001.hpc
 
comp001.hpc
Line 180: Line 241:
 
comp002.hpc
 
comp002.hpc
 
comp002.hpc
 
comp002.hpc
 +
Cores assigned: 16
 
</pre>
 
</pre>
 
If you have an application that manually forks processes onto the nodes of your job, you are responsible for parsing the <code>PBS_NODEFILE</code> to determine which nodes those are.
 
If you have an application that manually forks processes onto the nodes of your job, you are responsible for parsing the <code>PBS_NODEFILE</code> to determine which nodes those are.
  
−
MPI jobs also require you to feed the <code>PBS_NODEFILE</code> to <code>mpirun</code>.
+
Some MPI implementations require you to feed the <code>PBS_NODEFILE</code> to <code>mpirun</code>, e.g. for OpenMPI one may pass <code>-hostfile ${PBS_NODEFILE}</code>.
 +
 
 +
Be sure to include the <code>mpiprocs</code> parameter in your resource request. The default is 1, and will cause the <code>PBS_NODEFILE</code> to contain only one line per node if not specified correctly.
 +
 
 +
=== Selecting different nodes in one job ===
 +
Using the above information, one may allocate multiple nodes of the same type, e.g. multiple 48-core nodes.
 +
In order to mix multiple different resources, one may use the PBS' "+" notation.
 +
For example in order to mix one 48-core node and two 8-core nodes in one PBS job, one may pass:
 +
<pre>
 +
[username@hpc1 ~]$ qsub -l select=1:ncpus=48:mpiprocs=48+2:ncpus=8:mpiprocs=8 -l walltime=1:00:00 myscript.sh
 +
</pre>
  
 
== Guidelines / Rules ==
 
== Guidelines / Rules ==
  
−
* Create a temporary working directory in '''/scratch''', not '''/tmp'''
+
* Create a temporary working directory in '''/scratch-small-local''' or '''/scratch-large-network''', not '''/tmp'''
 
** '''/tmp''' is reserved for use by the operating system, and is only 5GB in size.
 
** '''/tmp''' is reserved for use by the operating system, and is only 5GB in size.
−
** Preferably specify '''/scratch/$PBS_JOBID''' in your submit script so that it's easy to associate scratch directories with their jobs.
+
** Preferably specify '''/scratch-small-local/${PBS_JOBID}''' in your submit script so that it's easy to associate scratch directories with their jobs.
 
** Copy your input files to your scratch space and work on the data there. Avoid using your home directory as much as possible.
 
** Copy your input files to your scratch space and work on the data there. Avoid using your home directory as much as possible.
−
*** If you need more than about 500GB of scratch space, you can also use '''/scratch2'''. It's a lot slower than '''/scratch''', so try to avoid that too.
 
 
** Copy only your results back to your home directory. Input files that haven't changed don't need to be copied.
 
** Copy only your results back to your home directory. Input files that haven't changed don't need to be copied.
 
** Erase your temporary working directory when you're done.
 
** Erase your temporary working directory when you're done.
  
 
* Secure your work from accidental deletion or contamination by disallowing other users access to your scratch directories
 
* Secure your work from accidental deletion or contamination by disallowing other users access to your scratch directories
−
** <code>umask 0077</code> disallows access by all other users
+
** <code>umask 0077</code> disallows access by all other users on newly created files and directories
  
 
== Examples ==
 
== Examples ==
Line 205: Line 276:
  
 
ADF script requesting 4 cores, on 1 node, -m selects to mail '''b'''egin and '''e'''nd messages and -M is the email address to send to. Requests 1 week walltime.
 
ADF script requesting 4 cores, on 1 node, -m selects to mail '''b'''egin and '''e'''nd messages and -M is the email address to send to. Requests 1 week walltime.
−
<pre>
+
<syntaxhighlight lang="bash">
 
#!/bin/bash
 
#!/bin/bash
 
#PBS -N JobName
 
#PBS -N JobName
−
#PBS -l select=1:ncpus=4:scratch=true
+
#PBS -l select=1:ncpus=4
 
#PBS -l walltime=168:00:00
 
#PBS -l walltime=168:00:00
 
#PBS -m be
 
#PBS -m be
Line 217: Line 288:
 
# make sure I'm the only one that can read my output
 
# make sure I'm the only one that can read my output
 
umask 0077
 
umask 0077
−
TMP=/scratch/${PBS_JOBID}
+
TMP=/scratch-small-local/${PBS_JOBID}
 
mkdir -p ${TMP}
 
mkdir -p ${TMP}
  
Line 227: Line 298:
 
cd ${TMP}
 
cd ${TMP}
  
−
. /apps/adf/2014.04/adfrc.sh
+
module load app/adf/2019.103
  
 
# override ADF's scratch directory
 
# override ADF's scratch directory
Line 244: Line 315:
 
# delete my temporary files
 
# delete my temporary files
 
[ $? -eq 0 ] && /bin/rm -rf ${TMP}
 
[ $? -eq 0 ] && /bin/rm -rf ${TMP}
−
</pre>
+
</syntaxhighlight>
  
 
=== ANSYS ===
 
=== ANSYS ===
Line 251: Line 322:
  
 
Fluent script requesting 4 cores, on 1 node, -m selects to mail '''b'''egin and '''e'''nd messages and -M is the email address to send to. Requests 1 week walltime.
 
Fluent script requesting 4 cores, on 1 node, -m selects to mail '''b'''egin and '''e'''nd messages and -M is the email address to send to. Requests 1 week walltime.
−
<pre>
+
<syntaxhighlight lang="bash">
 
#!/bin/bash
 
#!/bin/bash
 
#PBS -N JobName
 
#PBS -N JobName
−
#PBS -l select=1:ncpus=4:scratch=true
+
#PBS -l select=1:ncpus=4:mpiprocs=4:mem=16GB
 +
#PBS -l license_fluent=4
 
#PBS -l walltime=168:00:00
 
#PBS -l walltime=168:00:00
 
#PBS -m be
 
#PBS -m be
Line 265: Line 337:
 
# make sure I'm the only one that can read my output
 
# make sure I'm the only one that can read my output
 
umask 0077
 
umask 0077
−
TMP=/scratch/${PBS_JOBID}
+
TMP=/scratch-small-local/${PBS_JOBID}
 
mkdir -p ${TMP}
 
mkdir -p ${TMP}
  
Line 280: Line 352:
  
 
# choose version of FLUENT
 
# choose version of FLUENT
−
#module load app/ansys150
+
# Use module avail to see which versions of ansys are available
−
module load app/ansys162
+
module load app/ansys/20.1
  
 
# Automatically calculate the number of processors
 
# Automatically calculate the number of processors
−
np=$(cat ${PBS_NODEFILE} | wc -l)
+
np=$(wc -l < ${PBS_NODEFILE})
  
 
fluent 3d -pdefault -cnf=${PBS_NODEFILE} -mpi=intel -g -t${np} -ssh -i ${INPUT}
 
fluent 3d -pdefault -cnf=${PBS_NODEFILE} -mpi=intel -g -t${np} -ssh -i ${INPUT}
Line 294: Line 366:
 
# delete my temporary files
 
# delete my temporary files
 
[ $? -eq 0 ] && /bin/rm -rf ${TMP}
 
[ $? -eq 0 ] && /bin/rm -rf ${TMP}
−
</pre>
+
</syntaxhighlight>
 +
 
 +
MPI example using 3 nodes and 3 x 18 cores.
 +
 
 +
<syntaxhighlight lang="bash">
 +
#!/bin/bash
 +
 
 +
#PBS -l select=3:ncpus=18:mpiprocs=18:mem=28GB
 +
##PBS -l license_fluent=54
 +
#PBS -l walltime=24:00:00
 +
#PBS -l place=scatter
 +
#PBS -N 46krpm-grad
 +
 
 +
#PBS -e fluent.err
 +
#PBS -o fluent.out
 +
s
 +
INPUT=/home/gerhardv/46krpm-grad/journal-1.jou
 +
TMP=/scratch2
 +
mkdir -p ${TMP}
 +
 
 +
if [ ! -d "${TMP}" ]; then
 +
        echo "Cannot create temporary directory. Disk probably full."
 +
        exit 1
 +
fi
 +
 
 +
# copy the input files to ${TMP}
 +
echo "Copying from ${PBS_O_WORKDIR}/ to ${TMP}/"
 +
/usr/bin/rsync -vax "${PBS_O_WORKDIR}"/ ${TMP}/
 +
 
 +
cd ${TMP}
 +
 
 +
module load app/ansys/20.1
 +
module load intel/mpi
 +
 
 +
np=$(wc -l < ${PBS_NODEFILE})
 +
fluent 3ddp -pdefault -cnf=${PBS_NODEFILE} -mpi=intel -g -t${np} -ssh -i ${INPUT}
 +
 
 +
# job done, copy everything back
 +
echo "Copying from ${TMP}/ to ${PBS_O_WORKDIR}/"
 +
/usr/bin/rsync -vax ${TMP}/ "${PBS_O_WORKDIR}/"
 +
 
 +
# delete my temporary files
 +
[ $? -eq 0 ] && /bin/rm -rf ${TMP}
 +
</syntaxhighlight>
  
 
==== Fluid-Structure Interaction ====
 
==== Fluid-Structure Interaction ====
Line 304: Line 419:
 
* journal (.jnl) file
 
* journal (.jnl) file
 
* submit script (.sh)
 
* submit script (.sh)
−
+
 
 +
The coupling file should contain two participants. The names of these participants should not have spaces in them. In the example below, <code>Solution 4</code> should be renamed to something like <code>Solution4</code>. Make sure to replace all instances of the name in the file.
 +
 
 +
<pre>
 +
<SystemCoupling Ver="1">
 +
  <Participants Count="2">
 +
    <Participant Ver="1" Type="0">
 +
      <Name PropType="string">Solution 4</Name>
 +
      <DisplayName PropType="string">0012 V2</DisplayName>
 +
      <SupportsCouplingIterations PropType="bool">True</SupportsCouplingIterations>
 +
      <UnitSystem PropType="string">MKS_STANDARD</UnitSystem>
 +
      <Regions Count="1">
 +
<--- snip --->
 +
</pre>
 +
 
 
The journal file should contain (replace the filename on the ‘rc’ line with your case file):
 
The journal file should contain (replace the filename on the ‘rc’ line with your case file):
  
Line 320: Line 449:
 
The job script is given below. Update the <code>COUPLING</code>, <code>STRUCTURALDATA</code>, <code>JOURNAL</code> and <code>NPA</code> variables to reflect your case.
 
The job script is given below. Update the <code>COUPLING</code>, <code>STRUCTURALDATA</code>, <code>JOURNAL</code> and <code>NPA</code> variables to reflect your case.
  
−
<pre>
+
<syntaxhighlight lang="bash">
 
#!/bin/bash
 
#!/bin/bash
  
 
#PBS -N fsi
 
#PBS -N fsi
−
#PBS -l select=1:ncpus=48:mem=90GB:scratch=true
+
#PBS -l select=1:ncpus=48:mpiprocs=48:mem=90GB
 +
#PBS -l license_fluent=48
 
#PBS -l walltime=24:00:00
 
#PBS -l walltime=24:00:00
  
Line 334: Line 464:
 
NPA=8
 
NPA=8
 
# Automatically calculate the number of processors left over for Fluent
 
# Automatically calculate the number of processors left over for Fluent
−
NP=$(cat ${PBS_NODEFILE} | wc -l)
+
NP=$(wc -l < ${PBS_NODEFILE})
 
NPF=$((NP-NPA))
 
NPF=$((NP-NPA))
  
Line 340: Line 470:
 
umask 0077
 
umask 0077
  
−
# create a temporary directory with a random name in /scratch
+
# create a temporary directory with a random name in /scratch-small-local
−
TMP=/scratch/${PBS_JOBID}
+
TMP=/scratch-small-local/${PBS_JOBID}
 
mkdir -p ${TMP}
 
mkdir -p ${TMP}
 
echo "Temporary work dir: ${TMP}"
 
echo "Temporary work dir: ${TMP}"
Line 356: Line 486:
 
cd ${TMP}
 
cd ${TMP}
  
−
module load app/ansys162
+
module load app/ansys/20.1
  
 
# Start coupling program
 
# Start coupling program
Line 388: Line 518:
 
# delete my temporary files
 
# delete my temporary files
 
[ $? -eq 0 ] && /bin/rm -rf ${TMP}
 
[ $? -eq 0 ] && /bin/rm -rf ${TMP}
−
</pre>
+
</syntaxhighlight>
  
 
==== CFX ====
 
==== CFX ====
  
 
CFX script requesting 4 cores, on 1 node, -m selects to mail '''b'''egin and '''e'''nd messages and -M is the email address to send to. Requests 1 week walltime.
 
CFX script requesting 4 cores, on 1 node, -m selects to mail '''b'''egin and '''e'''nd messages and -M is the email address to send to. Requests 1 week walltime.
−
<pre>
+
<syntaxhighlight lang="bash">
 
#!/bin/bash
 
#!/bin/bash
 
#PBS -N JobName
 
#PBS -N JobName
−
#PBS -l select=1:ncpus=4:scratch=true
+
#PBS -l select=1:ncpus=4:mpiprocs=4
 
#PBS -l walltime=168:00:00
 
#PBS -l walltime=168:00:00
 
#PBS -m be
 
#PBS -m be
Line 408: Line 538:
 
# make sure I'm the only one that can read my output
 
# make sure I'm the only one that can read my output
 
umask 0077
 
umask 0077
−
TMP=/scratch/${PBS_JOBID}
+
TMP=/scratch-small-local/${PBS_JOBID}
 
mkdir -p ${TMP}
 
mkdir -p ${TMP}
  
Line 422: Line 552:
 
cd ${TMP}
 
cd ${TMP}
  
−
module load app/ansys162
+
module load app/ansys/16.2
  
 
# get list of processors
 
# get list of processors
Line 435: Line 565:
 
# delete my temporary files
 
# delete my temporary files
 
[ $? -eq 0 ] && /bin/rm -rf ${TMP}
 
[ $? -eq 0 ] && /bin/rm -rf ${TMP}
−
</pre>
+
</syntaxhighlight>
  
 
=== Abaqus ===
 
=== Abaqus ===
  
−
Abaqus script requesting 4 cores, on 1 node, -m selects to mail '''b'''egin and '''e'''nd messages and -M is the email address to send to. Uses system default walltime.
+
Abaqus script requesting 4 cores, on 1 node, -m selects to mail '''b'''egin and '''e'''nd messages and -M is the email address to send to.
−
<pre>
+
<syntaxhighlight lang="bash">
 
#!/bin/bash
 
#!/bin/bash
−
#PBS -l select=1:ncpus=4:scratch=true
+
#PBS -l select=1:ncpus=4:mpiprocs=4
 +
#PBS -l walltime=1:00:00
 
#PBS -m be
 
#PBS -m be
 
#PBS -M username@sun.ac.za
 
#PBS -M username@sun.ac.za
Line 451: Line 582:
 
# make sure I'm the only one that can read my output
 
# make sure I'm the only one that can read my output
 
umask 0077
 
umask 0077
−
TMP=/scratch/${PBS_JOBID}
+
TMP=/scratch-small-local/${PBS_JOBID}
 
mkdir -p ${TMP}
 
mkdir -p ${TMP}
  
Line 468: Line 599:
  
 
# Automatically calculate the number of processors
 
# Automatically calculate the number of processors
−
np=$(cat ${PBS_NODEFILE} | wc -l)
+
np=$(wc -l < ${PBS_NODEFILE})
  
 
abaqus job=${JOBNAME} input=${JOBNAME}.inp analysis cpus=${np} scratch=${TMP} interactive
 
abaqus job=${JOBNAME} input=${JOBNAME}.inp analysis cpus=${np} scratch=${TMP} interactive
Line 479: Line 610:
 
# delete my temporary files
 
# delete my temporary files
 
[ $? -eq 0 ] && /bin/rm -rf ${TMP}
 
[ $? -eq 0 ] && /bin/rm -rf ${TMP}
−
</pre>
+
</syntaxhighlight>
  
 
=== R ===
 
=== R ===
  
−
R script requesting 1 node in the 'intel' group, -m selects to mail '''b'''egin and '''e'''nd messages and -M is the email address to send to. Uses system default walltime.
+
R script requesting 1 node, -m selects to mail '''b'''egin and '''e'''nd messages and -M is the email address to send to.
  
−
<pre>
+
<syntaxhighlight lang="bash">
 
#!/bin/bash
 
#!/bin/bash
  
−
#PBS -l select=1:ncpus=1:intel=true
+
#PBS -l select=1:ncpus=1
 +
#PBS -l walltime=00:30:00
 
#PBS -M username@sun.ac.za
 
#PBS -M username@sun.ac.za
 
#PBS -m be
 
#PBS -m be
Line 494: Line 626:
 
cd ${PBS_O_WORKDIR}
 
cd ${PBS_O_WORKDIR}
  
−
module load app/R
+
module load app/R/4.0.2
  
 
R CMD BATCH script.R
 
R CMD BATCH script.R
−
</pre>
+
</syntaxhighlight>
  
 
=== CPMD ===
 
=== CPMD ===
  
−
CPMD script requesting 8 cores on 1 node, -N names the job 'cpmd', -m selects to mail '''e'''nd message and -M is the email address to send to. CPMD runs with MPI which needs to be told which nodes it may use. The list of nodes it may use is given in <code>$PBS_NODEFILE</code>. Uses system default walltime.
+
CPMD script requesting 8 cores on 1 node, -N names the job 'cpmd', -m selects to mail '''e'''nd message and -M is the email address to send to. CPMD runs with MPI which needs to be told which nodes it may use. The list of nodes it may use is given in <code>$PBS_NODEFILE</code>.
  
−
<pre>
+
<syntaxhighlight lang="bash">
 
#!/bin/bash
 
#!/bin/bash
 
#PBS -N cpmd
 
#PBS -N cpmd
−
#PBS -l select=1:mpiprocs=8
+
#PBS -l select=1:ncpus=8:mpiprocs=8
 +
#PBS -l walltime=1:00:00
 
#PBS -m e
 
#PBS -m e
 
#PBS -M username@sun.ac.za
 
#PBS -M username@sun.ac.za
Line 516: Line 649:
  
 
# Automatically calculate the number of processors
 
# Automatically calculate the number of processors
−
np=$(cat ${PBS_NODEFILE} | wc -l)
+
np=$(wc -l < ${PBS_NODEFILE})
  
 
mpirun -np ${np} --hostfile ${PBS_NODEFILE} /apps/CPMD/3.17.1/cpmd.x xyz.inp > xyz.out
 
mpirun -np ${np} --hostfile ${PBS_NODEFILE} /apps/CPMD/3.17.1/cpmd.x xyz.inp > xyz.out
−
</pre>
+
</syntaxhighlight>
  
 
=== Gaussian ===
 
=== Gaussian ===
Line 525: Line 658:
 
Gaussian has massive temporary files (.rwf file). Generally we don't care about this file afterward, so this script doesn't copy it from temporary storage after job completion. Requests 6 week walltime.
 
Gaussian has massive temporary files (.rwf file). Generally we don't care about this file afterward, so this script doesn't copy it from temporary storage after job completion. Requests 6 week walltime.
  
−
<pre>
+
<syntaxhighlight lang="bash">
 
#!/bin/bash
 
#!/bin/bash
 
#PBS -N SomeHecticallyChemicalName
 
#PBS -N SomeHecticallyChemicalName
−
#PBS -l select=1:ncpus=8:mem=16GB:scratch=true
+
#PBS -l select=1:ncpus=8:mpiprocs=8:mem=16GB:scratch=true
 
#PBS -l walltime=1008:00:00
 
#PBS -l walltime=1008:00:00
 
#PBS -m be
 
#PBS -m be
Line 536: Line 669:
 
# make sure I'm the only one that can read my output
 
# make sure I'm the only one that can read my output
 
umask 0077
 
umask 0077
−
TMP=/scratch/${PBS_JOBID}
+
TMP=/scratch-small-local/${PBS_JOBID}
−
TMP2=/scratch2/${PBS_JOBID}
+
TMP2=/scratch-large-network/${PBS_JOBID}
 
mkdir -p ${TMP} ${TMP2}
 
mkdir -p ${TMP} ${TMP2}
  
Line 546: Line 679:
  
 
if [ ! -d "${TMP2}" ]; then
 
if [ ! -d "${TMP2}" ]; then
−
echo "Cannot create overflow temporary directory. Disk probably full."
+
echo "Cannot create temporary directory. Disk probably full."
 
exit 1
 
exit 1
 
fi
 
fi
Line 559: Line 692:
  
 
# make sure input file has %RWF line for specifying temporary storage
 
# make sure input file has %RWF line for specifying temporary storage
−
if [ -z "`/bin/grep ^%RWF ${INPUT}`" ]; then
+
if [ -z "$(/bin/grep ^%RWF ${INPUT})" ]; then
 
/bin/sed -i '1s/^/%RWF\n/' ${INPUT}
 
/bin/sed -i '1s/^/%RWF\n/' ${INPUT}
 
fi
 
fi
  
−
# assign 100GB of local temporary storage for every 4 CPUs
+
# update input file to use temporary storage, split into 500GB files
−
MAXTMP=$(( $(/bin/cat ${PBS_NODEFILE} | /usr/bin/wc -l) * 100 / 4 ))
+
/bin/sed -i -E "s|%RWF(.*)|%RWF=${TMP}/1.rwf,200GB,${TMP2}/2.rwf,500GB,${TMP2}/3.rwf,500GB,${TMP2}/4.rwf,500GB,${TMP2}/,-1|g" ${TMP}/${INPUT}
−
 
 
−
# update input file to use local temporary storage
 
−
/bin/sed -i -E "s|%RWF(.*)|%RWF=${TMP}/,${MAXTMP}GB,${TMP2}/1.rwf,500GB,${TMP2}/2.rwf,500GB,${TMP2}/3.rwf,500GB,${TMP2}/4.rwf,500GB,${TMP2}/,-1|g" ${TMP}/${INPUT}
 
  
 
. /apps/Gaussian/09D/g09/bsd/g09.profile
 
. /apps/Gaussian/09D/g09/bsd/g09.profile
Line 578: Line 708:
  
 
# delete my temporary files
 
# delete my temporary files
−
[ $? -eq 0 ] && /bin/rm -rf ${TMP} ${TMP2}
+
[ $? -eq 0 ] && /bin/rm -rf ${TMP}
−
</pre>
+
</syntaxhighlight>
  
−
This script also requires that the input file contains a line starting with '''%RWF'''. This is so that the script can update the input file to specify that only the first part of the RWF be written to the compute node's local scratch space. Overflow is written to the scratch space on the storage server. Unfortunately the RWF files can grow in size to more than 1TB, and can fill the compute node's scratch space, choking out other jobs and dying itself.
+
This script also requires that the input file contains a line starting with '''%RWF'''. This is so that the script can update the input file to specify that only the first part of the RWF be written to the compute node's local scratch space.
  
 
=== pisoFOAM ===
 
=== pisoFOAM ===
Line 587: Line 717:
 
pisoFOAM generates a lot of output, not all of which is useful. In this example we use '''crontab''' to schedule the deletion of unwanted output while the job runs. Requests 3 week walltime.
 
pisoFOAM generates a lot of output, not all of which is useful. In this example we use '''crontab''' to schedule the deletion of unwanted output while the job runs. Requests 3 week walltime.
  
−
<pre>
+
<syntaxhighlight lang="bash">
 
#!/bin/bash
 
#!/bin/bash
−
#PBS -l select=1:ncpus=8:scratch=true
+
#PBS -l select=1:ncpus=8:mpiprocs=8:large-scratch=true
 
#PBS -l walltime=504:00:00
 
#PBS -l walltime=504:00:00
 
#PBS -m be
 
#PBS -m be
Line 595: Line 725:
 
# make sure I'm the only one that can read my output
 
# make sure I'm the only one that can read my output
 
umask 0077
 
umask 0077
−
# create a temporary directory in /scratch
+
# create a temporary directory in /scratch-small-local
−
TMP=/scratch/${PBS_JOBID}
+
TMP=/scratch-small-local/${PBS_JOBID}
 
/bin/mkdir ${TMP}
 
/bin/mkdir ${TMP}
 
echo "Temporary work dir: ${TMP}"
 
echo "Temporary work dir: ${TMP}"
Line 615: Line 745:
  
 
# Automatically calculate the number of processors
 
# Automatically calculate the number of processors
−
np=$(cat ${PBS_NODEFILE} | wc -l)
+
np=$(wc -l < ${PBS_NODEFILE})
  
 
module load compilers/gcc-4.8.2
 
module load compilers/gcc-4.8.2
Line 631: Line 761:
 
mpirun -np ${np} pisoFoam -parallel > ${PBS_O_WORKDIR}/output.log
 
mpirun -np ${np} pisoFoam -parallel > ${PBS_O_WORKDIR}/output.log
 
   
 
   
−
# remove crontab entry
+
# remove crontab entry (assumes I only have one on this node)
 
/usr/bin/crontab -r
 
/usr/bin/crontab -r
 
   
 
   
Line 640: Line 770:
 
# delete my temporary files
 
# delete my temporary files
 
[ $? -eq 0 ] && /bin/rm -rf ${TMP}
 
[ $? -eq 0 ] && /bin/rm -rf ${TMP}
−
</pre>
+
</syntaxhighlight>
  
 
=== MSC Marc ===
 
=== MSC Marc ===
  
−
Marc script requesting 1 core, on 1 node, -m selects to mail '''a'''bort and '''e'''nd messages and -M is the email address to send to. Uses system default walltime.
+
Marc script requesting 8 cores, on 1 node, -m selects to mail '''e'''nd message and -M is the email address to send to.
−
<pre>
+
<syntaxhighlight lang="bash">
 
#!/bin/bash
 
#!/bin/bash
 
#PBS -N JobName
 
#PBS -N JobName
−
#PBS -l nodes=1:ppn=1:scratch
+
#PBS -l select=1:ncpus=8:mpiprocs=8
−
#PBS -m abe
+
#PBS -l walltime=24:00:00
−
#PBS -e output.err
+
#PBS -l license_marc=8
−
#PBS -o output.out
+
#PBS -m e
−
#PBS -M username@sun.ac.za
 
  
 
INPUT=inputfile
 
INPUT=inputfile
Line 658: Line 787:
 
# make sure I'm the only one that can read my output
 
# make sure I'm the only one that can read my output
 
umask 0077
 
umask 0077
−
TMP=/scratch/$PBS_JOBID
+
TMP=/scratch-small-local/${PBS_JOBID}
−
mkdir -p $TMP
+
mkdir -p ${TMP}
  
−
if [ ! -d "$TMP" ]; then
+
if [ ! -d "${TMP}" ]; then
 
echo "Cannot create temporary directory. Disk probably full."
 
echo "Cannot create temporary directory. Disk probably full."
 
exit 1
 
exit 1
 
fi
 
fi
  
−
# copy the input files to $TMP
+
# copy the input files to ${TMP}
 
echo "Copying from ${PBS_O_WORKDIR}/ to ${TMP}/"
 
echo "Copying from ${PBS_O_WORKDIR}/ to ${TMP}/"
 
/usr/bin/rsync -vax "${PBS_O_WORKDIR}"/ ${TMP}/
 
/usr/bin/rsync -vax "${PBS_O_WORKDIR}"/ ${TMP}/
  
−
cd $TMP
+
cd ${TMP}
  
 
module load app/marc
 
module load app/marc
  
 
# get number of processors assigned
 
# get number of processors assigned
−
NPS=`/bin/cat $PBS_NODEFILE | /usr/bin/wc -l`
+
NPS=$(wc -l < ${PBS_NODEFILE})
−
HOSTS=hosts.$PBS_JOBID
+
HOSTS=hosts.${PBS_JOBID}
  
−
[ -f $HOSTS ] && /bin/rm $HOSTS
+
[ -f ${HOSTS} ] && /bin/rm ${HOSTS}
 
# create hosts file
 
# create hosts file
−
uniq -c $PBS_NODEFILE | while read np host; do
+
uniq -c ${PBS_NODEFILE} | while read np host; do
−
/bin/echo "${host} ${np}" >> $HOSTS
+
/bin/echo "${host} ${np}" >> ${HOSTS}
 
done
 
done
  
 
if [ ${NPS} -gt 1 ]; then
 
if [ ${NPS} -gt 1 ]; then
−
run_marc -j $INPUT -ver n -back n -ci n -cr n -nps $NPS -host $HOSTS
+
run_marc -j ${INPUT} -ver n -back n -ci n -cr n -nps ${NPS} -host ${HOSTS}
 
else
 
else
−
run_marc -j $INPUT -ver n -back n -ci n -cr n
+
run_marc -j ${INPUT} -ver n -back n -ci n -cr n
 
fi
 
fi
  
Line 696: Line 825:
 
# delete my temporary files
 
# delete my temporary files
 
[ $? -eq 0 ] && /bin/rm -rf ${TMP}
 
[ $? -eq 0 ] && /bin/rm -rf ${TMP}
−
</pre>
+
</syntaxhighlight>
  
 
=== mothur ===
 
=== mothur ===
Line 704: Line 833:
 
mothur's input can either be a file with all the commands to process listed, or the commands can be given on the commandline if prefixed with a #.
 
mothur's input can either be a file with all the commands to process listed, or the commands can be given on the commandline if prefixed with a #.
  
−
<pre>
+
<syntaxhighlight lang="bash">
 
#!/bin/bash
 
#!/bin/bash
  
−
#PBS -l nodes=1:ppn=1:scratch
+
#PBS -l select=1:ncpus=1:mpiprocs=1
−
#PBS -m ae
+
#PBS -l walltime=24:00:00
−
#PBS -M username@sun.ac.za
+
#PBS -m e
  
 
# make sure I'm the only one that can read my output
 
# make sure I'm the only one that can read my output
 
umask 0077
 
umask 0077
−
TMP=/scratch/$PBS_JOBID
+
TMP=/scratch-small-local/${PBS_JOBID}
−
mkdir -p $TMP
+
mkdir -p ${TMP}
  
−
if [ ! -d "$TMP" ]; then
+
if [ ! -d "${TMP}" ]; then
 
echo "Cannot create temporary directory. Disk probably full."
 
echo "Cannot create temporary directory. Disk probably full."
 
exit 1
 
exit 1
 
fi
 
fi
  
−
# copy the input files to $TMP
+
# copy the input files to ${TMP}
 
echo "Copying from ${PBS_O_WORKDIR}/ to ${TMP}/"
 
echo "Copying from ${PBS_O_WORKDIR}/ to ${TMP}/"
 
/usr/bin/rsync -vax "${PBS_O_WORKDIR}"/ ${TMP}/
 
/usr/bin/rsync -vax "${PBS_O_WORKDIR}"/ ${TMP}/
  
−
cd $TMP
+
cd ${TMP}
  
 
module load app/mothur
 
module load app/mothur
  
 
# Automatically calculate the number of processors
 
# Automatically calculate the number of processors
−
np=$(cat $PBS_NODEFILE | wc -l)
+
np=$(wc -l < ${PBS_NODEFILE})
  
 
mothur inputfile
 
mothur inputfile
  
 
# could also put the commands on the command line
 
# could also put the commands on the command line
−
#mothur "#cluster.split(column=file.dist, name=file.names, large=T, processors=$np)"
+
#mothur "#cluster.split(column=file.dist, name=file.names, large=T, processors=${np})"
  
 
# job done, copy everything back  
 
# job done, copy everything back  
Line 743: Line 872:
 
# delete my temporary files
 
# delete my temporary files
 
[ $? -eq 0 ] && /bin/rm -rf ${TMP}
 
[ $? -eq 0 ] && /bin/rm -rf ${TMP}
−
</pre>
+
</syntaxhighlight>
  
−
=== Hadoop ===
+
=== Altair FEKO ===
  
−
Hadoop is useful for sorting through massive amounts of data. In this example we read the input data into a distributed HDFS, and do a map/reduce. Upon completion the output is copied out of the HDFS to central storage. <code>amd</code> nodes are requested due to their large scratch space. The input and output data together should not exceed 1.5TB, so we request 1 node for every 750GB of input data. In this example we request 6 nodes for 4TB of input data.
+
<syntaxhighlight lang="bash">
 +
#!/bin/bash
  
−
'''Java example'''
+
#PBS -l select=1:ncpus=10:mpiprocs=10:scratch=true:mem=100GB
−
<pre>
+
#PBS -l walltime=48:00:00
−
#!/bin/bash
+
#PBS -m e
  
−
#PBS -V
+
INPUT=input.cfx
−
#PBS -m abe
 
−
#PBS -l nodes=6:ppn=1:amd:scratch
 
−
#PBS -N hadoopDedupe
 
−
#PBS -M username@sun.ac.za
 
−
#PBS -m abe
 
  
 
# make sure I'm the only one that can read my output
 
# make sure I'm the only one that can read my output
 
umask 0077
 
umask 0077
 +
TMP=/scratch-small-local/${PBS_JOBID}
 +
mkdir -p ${TMP}
  
−
# create a temporary directory in /scratch
+
if [ ! -d "${TMP}" ]; then
−
TMP=/scratch/$PBS_JOBID
+
echo "Cannot create temporary directory. Disk probably full."
−
mkdir -p $TMP/logs
+
exit 1
 +
fi
 +
 
 +
. /apps/altair/hyperworks/2017/altair/feko/bin/initfeko
 +
# override job directory
 +
export FEKO_USER_HOME=${PBS_O_WORKDIR}
 +
# override temporary directory
 +
export FEKO_TMPDIR=${TMP}
 +
# limit to the 100GB requested
 +
export FEKO_MAXALLOCM=$((1024 * 100))
 +
 
 +
runfeko ${INPUT} --use-job-scheduler
 +
 
 +
# delete my temporary files
 +
[ $? -eq 0 ] && /bin/rm -rf ${TMP}
 +
</syntaxhighlight>
  
−
JAR=dedupe.jar
+
=== Numeca ===
−
CLASS=za.ac.sun.hpc.dedupe
 
−
INPUT="${PBS_O_WORKDIR}/input"
 
−
OUTPUT="${PBS_O_WORKDIR}"
 
  
−
HADOOP_PREFIX=/apps/hadoop/2.4.1
+
This script assumes you have 4 Numeca files available for your project. If your project is named ''proj1'', the required files are ''proj1.iec'', ''proj1.igg'', ''proj1.bcs'' and ''proj1.cgns''.
−
JAVA_HOME=/usr/lib/jvm/java
 
−
HADOOP_CONF_DIR="${PBS_O_WORKDIR}/conf"
 
  
−
# copy the class to $TMP
+
Script requests 4 hours walltime, and uses 8 cores.
−
cp "${HADOOP_PREFIX}/common/$JAR" $TMP
 
  
−
# create Hadoop configs
+
<syntaxhighlight lang="bash">
−
cp -a $HADOOP_PREFIX/conf $HADOOP_CONF_DIR
+
#!/bin/bash
  
−
MASTER=`hostname`
+
#PBS -N proj1
−
uniq $PBS_NODEFILE > $HADOOP_CONF_DIR/slaves
+
#PBS -l select=1:ncpus=8:mpiprocs=8
−
echo $MASTER > $HADOOP_CONF_DIR/masters
+
#PBS -l walltime=04:00:00
  
−
sed -i "s|export JAVA_HOME=.*|export JAVA_HOME=$JAVA_HOME|g" $HADOOP_CONF_DIR/hadoop-env.sh
+
INPUT=proj1
−
sed -i "s|<value>/scratch/.*</value>|<value>/scratch/$PBS_JOBID</value>|g" $HADOOP_CONF_DIR/{hdfs,core}-site.xml
 
−
sed -i "s|<value>.*:50090</value>|<value>$MASTER:50090</value>|g" $HADOOP_CONF_DIR/{hdfs,core}-site.xml
 
−
sed -i "s|hdfs://.*:|hdfs://$MASTER:|g" $HADOOP_CONF_DIR/core-site.xml
 
−
sed -i "s|.*export HADOOP_LOG_DIR.*|export HADOOP_LOG_DIR=$TMP/logs|g" $HADOOP_CONF_DIR/hadoop-env.sh
 
−
sed -i "s|.*export HADOOP_PID_DIR.*|export HADOOP_PID_DIR=$TMP|g" $HADOOP_CONF_DIR/hadoop-env.sh
 
  
−
# setup Hadoop services
+
# make sure I'm the only one that can read my output
−
. $HADOOP_CONF_DIR/hadoop-env.sh
+
umask 0077
 +
# create a temporary directory with the job id as name in /scratch-small-local
 +
TMP=/scratch-small-local/${PBS_JOBID}
 +
mkdir -p ${TMP}
 +
echo "Temporary work dir: ${TMP}"
  
−
$HADOOP_PREFIX/bin/hdfs namenode -format
+
# copy the input files to ${TMP}
−
$HADOOP_PREFIX/sbin/start-dfs.sh
+
echo "Copying from ${PBS_O_WORKDIR}/ to ${TMP}/"
 +
/usr/bin/rsync -vax "${PBS_O_WORKDIR}/" ${TMP}/
  
−
# import data
+
cd ${TMP}
−
$HADOOP_PREFIX/bin/hdfs dfs -mkdir /user
 
−
$HADOOP_PREFIX/bin/hdfs dfs -mkdir /user/$USER
 
−
$HADOOP_PREFIX/bin/hdfs dfs -put $INPUT input
 
  
−
cd $TMP
+
NUMECA=/apps/numeca/bin
 +
VERSION=90_3
  
−
# run hadoop job
+
# create hosts list
−
$HADOOP_PREFIX/bin/hadoop jar $JAR $CLASS input output
+
TMPH=$(/bin/mktemp)
 +
/usr/bin/tail -n +2 ${PBS_NODEFILE} | /usr/bin/uniq -c | while read np host; do
 +
    /bin/echo "${host} ${np}" >> ${TMPH}
 +
done
 +
NHOSTS=$(wc -l < ${TMPH})
 +
LHOSTS=$(while read line; do echo -n ${line}; done < ${TMPH})
 +
/bin/rm ${TMPH}
  
−
# retrieve output from Hadoop
+
# Create .run file
−
mkdir -p "$OUTPUT"
+
${NUMECA}/fine -niversion ${VERSION} -batch ${INPUT}.iec ${INPUT}.igg ${PBS_JOBID}.run
−
$HADOOP_PREFIX/bin/hdfs dfs -get output "$OUTPUT"
 
  
−
# stop Hadoop services
+
# Set up parallel run
−
$HADOOP_PREFIX/sbin/stop-dfs.sh
+
${NUMECA}/fine -niversion ${VERSION} -batch -parallel ${PBS_JOBID}.run ${NHOSTS} ${LHOSTS}
  
−
# retrieve logs
+
# Start solver
−
cp -a $TMP/logs "$PBS_O_WORKDIR"
+
${NUMECA}/euranusTurbo_parallel ${PBS_JOBID}.run -steering ${PBS_JOBID}.steering -niversion ${VERSION} -p4pg ${PBS_JOBID}.p4pg
  
−
# clear HDFS directories on all slaves
+
# job done, copy everything back
−
cat $HADOOP_CONF_DIR/slaves | while read slave; do
+
echo "Copying from ${TMP}/ to ${PBS_O_WORKDIR}/"
−
    ssh -n $slave "rm -rf $TMP"
+
/usr/bin/rsync -vax ${TMP}/ "${PBS_O_WORKDIR}/"
−
done
 
  
 
# delete my temporary files
 
# delete my temporary files
−
[ $? -eq 0 ] && /bin/rm -rf $TMP
+
[ $? -eq 0 ] && /bin/rm -rf ${TMP}
−
</pre>
+
</syntaxhighlight>
 +
 
 +
This script assumes you have 3 Numeca files available for your project. If your project is named ''proj1'', the required files are ''proj1.iec'', ''proj1.trb'' and ''proj1.geomTurbo''.
 +
 
 +
Script requests 4 hours walltime, and uses 8 cores.
  
−
'''Third-party script example'''
+
<syntaxhighlight lang="bash">
−
<pre>
 
 
#!/bin/bash
 
#!/bin/bash
  
−
#PBS -V
+
#PBS -N proj1
−
#PBS -m abe
+
#PBS -l select=1:ncpus=8:mpiprocs=8
−
#PBS -l nodes=6:ppn=1:amd:scratch
+
#PBS -l walltime=04:00:00
−
#PBS -N hadoopDedupe
+
 
−
#PBS -M username@sun.ac.za
+
INPUT=proj1
−
#PBS -m abe
 
  
 
# make sure I'm the only one that can read my output
 
# make sure I'm the only one that can read my output
 
umask 0077
 
umask 0077
 +
# create a temporary directory with the job id as name in /scratch-small-local
 +
TMP=/scratch-small-local/${PBS_JOBID}
 +
mkdir -p ${TMP}
 +
echo "Temporary work dir: ${TMP}"
  
−
# create a temporary directory in /scratch
+
# copy the input files to ${TMP}
−
TMP=/scratch/$PBS_JOBID
+
echo "Copying from ${PBS_O_WORKDIR}/ to ${TMP}/"
−
mkdir -p $TMP/logs
+
/usr/bin/rsync -vax "${PBS_O_WORKDIR}/" ${TMP}/
  
−
INPUT="${PBS_O_WORKDIR}/input"
+
# Automatically calculate the number of processors - 1 less than requested, used for master process
−
OUTPUT="${PBS_O_WORKDIR}"
+
NP=$(( $(wc -l < ${PBS_NODEFILE}) - 1))
−
MAPPER="mapper.py"
 
−
REDUCER="reducer.py"
 
  
−
# copy the mapper and reducer to $TMP
+
cd ${TMP}
−
cp "${PBS_O_WORKDIR}/$MAPPER" "${PBS_O_WORKDIR}/$REDUCER" $TMP
 
  
−
HADOOP_PREFIX=/apps/hadoop/2.4.1
+
NUMECA=/apps/numeca/bin
−
JAVA_HOME=/usr/lib/jvm/java
+
VERSION=101
−
HADOOP_CONF_DIR="${PBS_O_WORKDIR}/conf"
 
  
−
# create Hadoop configs
+
# inputs .trb, .geomTurbo
−
cp -a $HADOOP_PREFIX/conf $HADOOP_CONF_DIR
+
# outputs .bcs, .cgns, .igg
 +
/usr/bin/xvfb-run -d ${NUMECA}/igg -print -batch -niversion ${VERSION} -autogrid5 -trb ${TMP}/${INPUT}.trb -geomTurbo ${TMP}/${INPUT}.geomTurbo -mesh ${TMP}/${PBS_JOBID}.igg
  
−
MASTER=`hostname`
+
# inputs .iec, .igg
−
uniq $PBS_NODEFILE > $HADOOP_CONF_DIR/slaves
+
# outputs .run
−
echo $MASTER > $HADOOP_CONF_DIR/masters
+
/usr/bin/xvfb-run -d ${NUMECA}/fine -print -batch -niversion ${VERSION} -project ${TMP}/${INPUT}.iec -mesh ${TMP}/${PBS_JOBID}.igg -computation ${TMP}/${PBS_JOBID}.run
  
−
sed -i "s|export JAVA_HOME=.*|export JAVA_HOME=$JAVA_HOME|g" $HADOOP_CONF_DIR/hadoop-env.sh
+
# inputs .run
−
sed -i "s|<value>/scratch/.*</value>|<value>/scratch/$PBS_JOBID</value>|g" $HADOOP_CONF_DIR/{hdfs,core}-site.xml
+
# outputs .p4pg, .batch
−
sed -i "s|<value>.*:50090</value>|<value>$MASTER:50090</value>|g" $HADOOP_CONF_DIR/{hdfs,core}-site.xml
+
/usr/bin/xvfb-run -d ${NUMECA}/fine -print -batch -niversion ${VERSION} -parallel -computation ${TMP}/${PBS_JOBID}.run -nproc ${NP} -nbint 128 -nbreal 128
−
sed -i "s|hdfs://.*:|hdfs://$MASTER:|g" $HADOOP_CONF_DIR/core-site.xml
 
−
sed -i "s|.*export HADOOP_LOG_DIR.*|export HADOOP_LOG_DIR=$TMP/logs|g" $HADOOP_CONF_DIR/hadoop-env.sh
 
−
sed -i "s|.*export HADOOP_PID_DIR.*|export HADOOP_PID_DIR=$TMP|g" $HADOOP_CONF_DIR/hadoop-env.sh
 
  
−
# setup Hadoop services
+
# inputs .run, .p4pg
−
. $HADOOP_CONF_DIR/hadoop-env.sh
+
${NUMECA}/euranusTurbo_parallel${VERSION} ${TMP}/${PBS_JOBID}.run -steering ${TMP}/${PBS_JOBID}.steering -p4pg ${TMP}/${PBS_JOBID}.p4pg
  
−
$HADOOP_PREFIX/bin/hdfs namenode -format
+
# job done, copy everything back
−
$HADOOP_PREFIX/sbin/start-dfs.sh
+
echo "Copying from ${TMP}/ to ${PBS_O_WORKDIR}/"
 +
/usr/bin/rsync -vax ${TMP}/ "${PBS_O_WORKDIR}/"
 +
 
 +
# delete my temporary files
 +
[ $? -eq 0 ] && /bin/rm -rf ${TMP}
 +
</syntaxhighlight>
 +
 
 +
== Programs that handle job submission differently ==
  
−
# import data
+
=== MATLAB ===
−
$HADOOP_PREFIX/bin/hdfs dfs -mkdir /user
 
−
$HADOOP_PREFIX/bin/hdfs dfs -mkdir /user/$USER
 
−
$HADOOP_PREFIX/bin/hdfs dfs -put $INPUT input
 
  
−
cd $TMP
+
Matlab R2023a can be used on the hpc cluster with a VPN connection.  Enquire gerhardv@sun.ac.za
  
−
# run hadoop job
+
The MATLAB campus license allows all students to install MATLAB on their own computers. See [http://rga.sun.ac.za/firga/matlab.html this link] for details. The HPC currently has MATLAB R2023a, R2022b and R2021a installed.
−
STREAM=$HADOOP_PREFIX/share/hadoop/tools/lib/hadoop-streaming-2.4.1.jar
 
−
$HADOOP_PREFIX/bin/hadoop jar $STREAM $OPTIONS -files $MAPPER,$REDUCER -mapper $MAPPER -reducer $REDUCER -input input -output output
 
  
−
# retrieve output from Hadoop
+
Furthermore, the license includes a MATLAB Parallel Server license that means MATLAB jobs can run on the high performance cluster (hpc1). Prior to R2019a, MATLAB Parallel Server was called MATLAB Distributed Computing Server.  With MATLAB's Parallel Computing Toolbox (now freely available on the campus license), it's possible to submit your MATLAB code directly from your desktop to the HPC without writing submit scripts and submitting the job manually.  It is also possible to compile MATLAB executables and run these on the HPC with a bash script.  The advantage of doing this is that it does not consume worker licenses so freeing up the development environment for code writing and testing.
−
mkdir -p "$OUTPUT"
 
−
$HADOOP_PREFIX/bin/hdfs dfs -get output "$OUTPUT"
 
  
−
# stop Hadoop services
+
==== Setup ====
−
$HADOOP_PREFIX/sbin/stop-dfs.sh
 
  
−
# retrieve logs
+
#Download and run the [[:File:parallel_pbs.zip|integration script]]
−
cp -a $TMP/logs "$PBS_O_WORKDIR"
+
#The script (parallel_pbs) should be unzipped and placed in the MATLAB path. <code>C:\Users\me\Documents\MATLAB\</code> is a good place.
 +
#These files will be used to configure and establish the connection with the MATLAB parallel server on the HPC.
  
−
# clear HDFS directories on all slaves
+
==== Validate ====
−
cat $HADOOP_CONF_DIR/slaves | while read slave; do
+
 
−
    ssh -n $slave "rm -rf $TMP"
+
If your PCT is installed and licensed correctly, you should see a dropdown named Parallel in your toolbar.
−
done
+
 
 +
# Open the ''Parallel'' dropdown, and select ''Manage Cluster Profiles...'' ([[Media:MATLABPCT15.png|screenshot]])
 +
#: 'hpc1 remote R2023a' should be in the list of Cluster Profiles ([[Media:MATLABPCT16.png|screenshot]])
 +
# Select the 'Validation' tab
 +
#* Set the 'Number of workers to use' to '''4''' and click ''Validate'' ([[Media:MATLABPCT17.png|screenshot]])
 +
#: You will be prompted for your HPC username. When prompted for an identity file, select '''No''' if you don't know what it is.
 +
#: Depending on how busy the HPC is, the testing should complete in 10 to 30 minutes. ([[Media:MATLABPCT18.png|screenshot]])
 +
#; If the last step fails, your hostname is most probably incorrectly set up.
 +
 
 +
=== GPU ===
  
−
# delete my temporary files
+
To use the GPU's specify the ngpus= parameter in a qsub -I for interactive session, or in the bash script - add it to the line that normally has the mem= and ncpus= parameters.  Specify the queue with -q ee and Qlist=ee.  See below for an example for an interactive session.  The mem= ncpus= values can be changed within reason.  The bash script structure and other aspects of HPC use is described on https://www0.sun.ac.za/hpc
−
[ $? -eq 0 ] && /bin/rm -rf $TMP
 
−
</pre>
 
  
−
=== Numeca ===
+
'''If you want to use the GPUs message gerhardv@sun.ac.za for inclusion on the GPU queue.'''
  
−
This script assumes you have 4 Numeca files available for your project. If your project is named ''proj1'', the required files are ''proj1.iec'', ''proj1.igg'', ''proj1.bcs'' and ''proj1.cgns''.
+
<syntaxhighlight lang="bash">
 +
[gerhardv@hpc1 ~]$ qsub -I -l walltime=2:00:00 -q ee -l select=1:ncpus=4:mem=4GB:ngpus=1:Qlist=ee
 +
qsub: waiting for job 253270.hpc1.hpc to start
 +
qsub: job 253270.hpc1.hpc ready
 +
[gerhardv@comp048 ~]$
 +
</syntaxhighlight>
  
−
Script requests 4 hours walltime, and uses 8 cores on a host with scratch space.
+
The GPUs are commonly used via python with tensorflow or pytorch.  Other GPU bindings are possible.  Software sometimes automatically uses GPUs if they are present on the system, and R and MATLAB can also be configured to invoke GPU.  
 +
In bash script form it would look something like this:
  
−
<pre>
+
<syntaxhighlight lang="bash">
 
#!/bin/bash
 
#!/bin/bash
 +
#PBS -N JobName
  
−
#PBS -N proj1
+
#PBS -l select=1:ncpus=2:mem=16GB:ngpus=1:Qlist=ee
−
#PBS -l nodes=1:ppn=8:scratch
+
#PBS -l walltime=16:00:00
−
#PBS -l walltime=00:04:00:00
+
#PBS -m be
 +
#PBS -e output.err
 +
#PBS -o output.out
 +
#PBS -M username@sun.ac.za
  
−
INPUT=proj1
+
INPUT=inputfile
  
 
# make sure I'm the only one that can read my output
 
# make sure I'm the only one that can read my output
 
umask 0077
 
umask 0077
−
# create a temporary directory with the job id as name in /scratch
+
TMP=/scratch-small-local/${PBS_JOBID}
−
TMP=/scratch/${PBS_JOBID}
 
 
mkdir -p ${TMP}
 
mkdir -p ${TMP}
−
echo "Temporary work dir: ${TMP}"
 
  
−
# copy the input files to $TMP
+
if [ ! -d "${TMP}" ]; then
 +
echo "Cannot create temporary directory. Disk probably full."
 +
exit 1
 +
fi
 +
 
 +
# copy the input files to ${TMP}
 
echo "Copying from ${PBS_O_WORKDIR}/ to ${TMP}/"
 
echo "Copying from ${PBS_O_WORKDIR}/ to ${TMP}/"
−
/usr/bin/rsync -vax "${PBS_O_WORKDIR}/" ${TMP}/
+
/usr/bin/rsync -vax "${PBS_O_WORKDIR}"/ ${TMP}/
  
 
cd ${TMP}
 
cd ${TMP}
  
−
NUMECA=/apps/numeca/bin
+
# choose version of python that has tensorflow/pytorch
−
VERSION=90_3
+
module load python/3.8.1
−
 
 
−
# create hosts list
 
−
TMPH=$(/bin/mktemp)
 
−
/usr/bin/tail -n +2 ${PBS_NODEFILE} | /usr/bin/uniq -c | while read np host; do
 
−
    /bin/echo "${host} ${np}" >> ${TMPH}
 
−
done
 
−
NHOSTS=$(/bin/cat ${TMPH} | /usr/bin/wc -l)
 
−
LHOSTS=$(while read line; do echo -n ${line}; done < ${TMPH})
 
−
/bin/rm ${TMPH}
 
−
 
 
−
# Create .run file
 
−
${NUMECA}/fine -niversion $VERSION -batch ${INPUT}.iec ${INPUT}.igg ${PBS_JOBID}.run
 
  
−
# Set up parallel run
+
python3 yourcode.py
−
${NUMECA}/fine -niversion $VERSION -batch -parallel ${PBS_JOBID}.run ${NHOSTS} ${LHOSTS}
 
  
−
# Start solver
+
# job done, copy everything back  
−
${NUMECA}/euranusTurbo_parallel ${PBS_JOBID}.run -steering ${PBS_JOBID}.steering -niversion ${VERSION} -p4pg ${PBS_JOBID}.p4pg
 
−
 
 
−
# job done, copy everything back
 
 
echo "Copying from ${TMP}/ to ${PBS_O_WORKDIR}/"
 
echo "Copying from ${TMP}/ to ${PBS_O_WORKDIR}/"
 
/usr/bin/rsync -vax ${TMP}/ "${PBS_O_WORKDIR}/"
 
/usr/bin/rsync -vax ${TMP}/ "${PBS_O_WORKDIR}/"
Line 963: Line 1,105:
 
# delete my temporary files
 
# delete my temporary files
 
[ $? -eq 0 ] && /bin/rm -rf ${TMP}
 
[ $? -eq 0 ] && /bin/rm -rf ${TMP}
−
</pre>
 
  
−
== Programs that handle job submission differently ==
+
</syntaxhighlight>
  
−
=== MATLAB ===
+
Typically the code would be developed offline on a notebook or with an interactive session and then benchmarked on the HPC with a bash script to optimise walltime and memory.  The python code would then be run as a scheduler job using a bash script after development and debugging in interactive mode.
 +
Interactive session is launched as above with qsub -I.
  
−
With MATLAB's Parallel Computing Toolbox (PCT), it's possible to submit your MATLAB code directly from your desktop to the HPC without writing submit scripts and submitting the job manually. See [http://www.mathworks.com/products/parallel-computing/ MathWorks] for further details.
+
<syntaxhighlight lang="bash">
 +
[gerhardv@comp050 ~]$ module load python/3.8.1
 +
[gerhardv@comp050 ~]$ python3
 +
Python 3.8.1 (default, Jun 11 2021, 05:57:44)
 +
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
 +
Type "help", "copyright", "credits" or "license" for more information.
 +
>>> import tensorflow as tp
 +
>>> import torch as tt
 +
>>>
 +
</syntaxhighlight>
  
−
The HPC has a license to allow the use of 16 cores by MATLAB. MATLAB R2015a and all standard toolboxes are installed on the HPC, and any MATLAB product you are licensed for will be able to run on the HPC.
+
There is a conda environment for tensorflow and python 3.9.13
  
−
To be able to use the HPC for MATLAB, you will require a Parallel Computing Toolbox license on your desktop.
+
<syntaxhighlight lang="bash">
 +
[gerhardv@comp050 ~]$ module load app/tensorflow/2.9.2
 +
(tf) [gerhardv@comp049 ~]$ python3
 +
Python 3.9.13 (main, Aug 25 2022, 23:26:10)
 +
[GCC 11.2.0] :: Anaconda, Inc. on linux
 +
Type "help", "copyright", "credits" or "license" for more information.
 +
>>> import tensorflow as tf
 +
2022-09-08 12:29:15.954105: I tensorflow/core/util/util.cc:169] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
 +
>>> print(tf.__version__)
 +
2.9.2
 +
>>>
 +
>>> print(tf.config.list_physical_devices('GPU'))
 +
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:1', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:2', device_type='GPU')]
  
−
==== Setup ====
 
  
−
# Install the required scripts for a generic PBS cluster
+
</syntaxhighlight>
−
## Copy all the files from <code>''MATLABROOT''\toolbox\distcomp\examples\integration\pbs\nonshared</code> to <code>''MATLABROOT''\toolbox\local</code>. ''MATLABROOT'' is the location where you installed MATLAB on your machine, most probably in <code>C:\Program Files\MATLAB\R2015a</code> or <code>/usr/local/MATLAB/R2015a</code>.
 
−
## Edit <code>''MATLABROOT''\toolbox\local\independentSubmitFcn.m</code>
 
−
##* Change line 122 by adding <code>-l walltime=1:00:00:00</code><br/><code>additionalSubmitArgs = '-l walltime=1:00:00:00';</code>
 
−
## Edit <code>''MATLABROOT''\toolbox\local\communicatingSubmitFcn.m</code>
 
−
##* Change line 117 by adding <code>-l walltime=1:00:00:00</code><br/><code>additionalSubmitArgs = sprintf('-l nodes=%d:ppn=%d -l walltime=1:00:00:00', numberOfNodes, procsPerNode);</code>
 
−
##: The two changes are required to increase the default [[Main_Page#Job_priorities|walltime]] on the HPC.
 
−
# Force the use of FQDN or IP in hostname lookup for ''parpool''
 
−
## Create (or edit if it already exists) <code>''MATLABROOT''\toolbox\local\startup.m</code>, and add
 
−
##:<code>pctconfig('hostname', ''''IP or hostname'''');</code>
 
−
##:: replace '''IP or hostname''' with your machine's IP or hostname (<code>ip addr list</code> on Linux, <code>ipconfig</code> in a ''Command Prompt'' on Windows)
 
−
## Restart MATLAB to apply the change, or run it manually in the ''Command Window''
 
−
# Create a cluster profile
 
−
#: If your PCT is installed and licensed correctly, you should see a dropdown named Parallel in your toolbar.
 
−
## Open the ''Parallel'' dropdown, and select ''Manage Cluster Profiles...'' ([[Media:MATLABPCT1.png|screenshot]])
 
−
## Add a new '''Generic''' custom 3rd party cluster profile ([[Media:MATLABPCT2.png|screenshot]])
 
−
## Rename the new cluster profile to 'HPC1' by right-clicking on it
 
−
## Set the following values ([[Media:MATLABPCT3.png|screenshot]], [[Media:MATLABPCT4.png|screenshot]], [[Media:MATLABPCT5.png|screenshot]])
 
−
##* '''Description''': HPC1
 
−
##* '''NumWorkers''': 16
 
−
##* '''ClusterMatlabRoot''': /apps/MATLAB/R2015a
 
−
##* '''IndependentSubmitFcn''': {@independentSubmitFcn, 'hpc1.sun.ac.za', '/scratch2/'''user''''}
 
−
##*: replace '''user''' with your own username
 
−
##* '''CommunicationSubmitFcn''': {@communicatingSubmitFcn, 'hpc1.sun.ac.za', '/scratch2/'''user''''}
 
−
##*: replace '''user''' with your own username
 
−
##* '''OperatingSystem''': unix
 
−
##* '''HasSharedFilesystem''': false
 
−
##* '''GetJobStateFcn''': @getJobStateFcn
 
−
##* '''DeleteJobFcn''': @deleteJobFcn
 
−
##; All other values can be left at their default (or empty) values.
 
−
## Select the 'Validation Results' tab and click ''Validate'' ([[Media:MATLABPCT6.png|screenshot]])
 
−
##: You will be prompted for your HPC username. When prompted for a identity file, select '''No''' if you don't know what it is.
 
−
##: Depending on how busy the HPC is, the testing should complete in 10 to 30 minutes.
 
−
##; If the last step fails, your hostname is most probably incorrectly set up.
 

Latest revision as of 15:36, 15 August 2024

Jobs

The HPC implements a batch queueing system called PBS. Any work you want to do needs to be packaged as a job that the system can do without any interaction from you. Because it's a queue, and the queue might be long, your job might only start a couple of hours from now. It's important to have your script completely self-contained so that it can run without supervision, otherwise, it might just error and all that time waiting in the queue will be wasted.

Submitting jobs

PBS comes with very complete man pages. Therefore, for complete documentation of PBS commands you are encouraged to type man pbs and go from there. Jobs are submitted using the qsub command. Type man qsub for information on the plethora of options that it offers.

Let's say I have an executable called "myprog". Let me try and submit it to PBS: 

[username@hpc1 ~]$ qsub myprog
qsub:  file must be an ascii script

Oops... That didn't work because qsub expects a shell script. Any shell should work, so use your favorite one. So I write a simple script called "myscript.sh" 

#!/bin/bash
cd $PBS_O_WORKDIR
./myprog argument1 argument2

and then I submit it: 

[username@hpc1 ~]$ qsub myscript.sh
qsub: Job has no walltime requested. Request walltime with '-l walltime=HH:MM:SS'.

PBS wants to know how long every job is expected to run. This is so that it can schedule jobs optimally. Updating the script to include the PBS directive '-l walltime', we get the script 

#!/bin/bash
#PBS -l walltime=00:05:00
cd $PBS_O_WORKDIR
./myprog argument1 argument2

which we then submit: 

[username@hpc1 ~]$ qsub myscript.sh
16.hpc1.hpc

That worked! Note the use of the $PBS_O_WORKDIR environment variable. This is important since by default PBS on our cluster will start executing the commands in your shell script from your home directory. To go to the directory in which you executed qsub, cd to $PBS_O_WORKDIR. There are several other useful PBS environment variables that we will encounter later.

In this script, we also informed PBS that the job will be running for a maximum of 5 minutes.

Editing files

Editing files on the cluster can be done through a couple of different methods...

Native Editors

  • vim - The visual editor (vi) is the traditional Unix editor. However, it is not necessarily the most intuitive editor. That being the case, if you are unfamiliar with it, there is a vi tutorial, vimtutor.
  • pico - While pico is not installed on the system, nano is installed, and is a pico work-a-like.
  • nano - Nano has a good bit of on-screen help to make it easier to use.

External Editors

You can also use your favourite editor on your local machine and then transfer the files over to the HPC afterwards. One caveat to this is that files created on Windows machines usually contain unprintable characters which may be misinterpreted by Linux command interpreters (shells). If this happens, there is a utility called dos2unix that you can use to convert the text file from DOS/Windows formatting to Linux formatting. 

$ dos2unix script.sub
dos2unix: converting file script.sub to UNIX format ...

If you're using MobaXterm's default text editor, make sure your file is in the correct format by selecting UNIX in the Format menu, or checking that the penguin icon in the button bar is selected.

Specifying job parameters

By default, any script you submit will run on a single processor with 1GB of memory. The name of the job will be the name of the script, and it will not email you when it starts, finishes, or is interrupted. stdout and stderr are collected into separate files named after the job number. You can affect the default behaviour of PBS by passing it parameters. These parameters can be specified on the command line or inside the shell script itself. For example, let's say I want to send stdout and stderr to a file that is different from the default: 

[username@hpc1 ~]$ qsub -e myprog.err -o myprog.out myscript.sh

Alternatively, I can actually edit myscript.sh to include these parameters. I can specify any PBS command line parameter I want in a line that begins with "#PBS": 

#!/bin/bash
#PBS -l walltime=00:05:00
#PBS -e myprog.err
#PBS -o myprog.out
cd $PBS_O_WORKDIR
./myprog argument1 argument2

Now I just submit my modified script with no command-line arguments 

[username@hpc1 ~]$ qsub myscript.sh

Useful PBS parameters

Here is an example of a more involved script that requests only 1 hour of execution time, renames the job to 'My-Program', and sends email when the job begins, ends, or aborts: 

#!/bin/bash
 
# Name of my job:
#PBS -N My-Program
 
# Run for 1 hour:
#PBS -l walltime=1:00:00
 
# Where to write stderr:
#PBS -e myprog.err
 
# Where to write stdout: 
#PBS -o myprog.out
 
# Send me email when my job aborts, begins, or ends
#PBS -m abe
 
# This command switched to the directory from which the "qsub" command was run:
cd $PBS_O_WORKDIR
 
#  Now run my program
./myprog argument1 argument2
 
echo Done!

Some more useful PBS parameters:

  • -M: Specify your email address (defaults to campus email).
  • -j oe: merge standard output and standard error into standard output file.
  • -V: export all your environment variables to the batch job.
  • -I: run an interactive job (see below).

Once again, you are encouraged to consult the qsub manpage for more options.

Interactive jobs

Normally when you call qsub PBS grabs your script and goes away. You get the prompt returned to you almost immediately and whatever output is produced from your job goes into a designated file. But what if you want to run an interactive job, say to compile something? This is not, in general, the right way to use an HPC, because interactive jobs waste CPU time, mostly because humans are so slow. 

[username@hpc1 ~]$ qsub -I -l walltime=2:00:00 
qsub: waiting for job 1695.hpc1.hpc to start
qsub: job 1695.hpc1.hpc ready

[username@comp007 ~]$ logout

qsub: job 1695.hpc1.hpc completed
[username@hpc1 ~]$

In the example above I requested an interactive job, using the defaults of 1 core and 1GB memory and requesting it for 2 hours. If the system can honour the request, I'm immediately given an SSH session to the compute node assigned to my job. Closing the SSH session ends the job, and I'm returned to the login node.

You can also use the qsubi script, which will request an interactive session for 24 hours on a node with 4 cores and 14GB RAM. 

[username@hpc1 ~]$ qsubi
Requesting interactive session with 4 cores, 14GB RAM, for 24 hours
qsub: waiting for job 1696.hpc1.hpc to start
qsub: job 1696.hpc1.hpc ready

[username@comp008 ~]$ logout

qsub: job 1696.hpc1.hpc completed
[username@hpc1 ~]$

Jobs with large output files

If your job makes frequent writes to disk, it may benefit from using scratch space instead of working directly from the home directory.

Instead of a job submission like this: 

#!/bin/bash
#PBS -N massiveJob
#PBS -l walltime=01:00:00

cd ${PBS_O_WORKDIR}
myprogram -i /home/username/inputfile -o /home/username/outputfile

change it to something like this: 

#!/bin/bash
#PBS -l select=1:ncpus=1
#PBS -l walltime=1:00:00
#PBS -N massiveJob

# make sure I'm the only one that can read my output
umask 0077
# create a temporary directory with the job ID as name in /scratch-small-local
TMP=/scratch-small-local/${PBS_JOBID}
mkdir -p ${TMP}
echo "Temporary work dir: ${TMP}"

# copy the input files to ${TMP}
echo "Copying from ${PBS_O_WORKDIR}/ to ${TMP}/"
/usr/bin/rsync -vax "${PBS_O_WORKDIR}/" ${TMP}/

cd ${TMP}

# write my output to my new temporary work directory
myprogram -i inputfile -o outputfile

# job done, copy everything back
echo "Copying from ${TMP}/ to ${PBS_O_WORKDIR}/"
/usr/bin/rsync -vax ${TMP}/ "${PBS_O_WORKDIR}/"

# if the copy back succeeded, delete my temporary files
[ $? -eq 0 ] && /bin/rm -rf ${TMP}

Any job that has to write massive amounts of data will benefit from the above. If your job doesn't complete successfully or you don't clean up your temporary files, your temporary files will be moved to the /orphans directory. If you can't find your output, go look for it there.

There are two scratch spaces. One is the per-node /scratch-small-local directory, the other is the global /scratch-large-network directory. If you need to write more than 200GB of output, it's probably better to use the larger, network space.

Running OpenMP multi-threaded programs

By default, PBS assigns you 1 core on 1 node. You can, however, run your job on up to 64 cores per node. Therefore, if you want to run an OpenMP program, you must specify the number of processors per node. This is done with the flag -l select=1:ncpus=<cores> where <cores> is the number of OpenMP threads you wish to use. Keep in mind that you still must set the OMP_NUM_THREADS environment variable within your script, e.g.: 

#!/bin/bash
#PBS -N My-OpenMP-Script
#PBS -l select=1:ncpus=8
#PBS -l walltime=1:00:00
 
cd ${PBS_O_WORKDIR}
export OMP_NUM_THREADS=8
./MyOpenMPProgram

Using the PBS_NODEFILE for multi-threaded jobs

In a serial job, your PBS script will automatically be executed on the target node assigned by the scheduler. If you asked for more than one node, however, your script will only execute on the first node of the set of nodes allocated to you. To access the remainder of the nodes, you must either use MPI or manually launch threads. But which nodes to run on? PBS gives you a list of nodes in a file at the location pointed to by the PBS_NODEFILE environment variable. In your shell script, you may thereby ascertain the nodes on which your job can run by looking at the file in the location specified by this variable: 

#!/bin/bash
#PBS -l walltime=1:00:00
#PBS -l select=2:ncpus=8:mpiprocs=8

echo "The nodefile for this job is stored at ${PBS_NODEFILE}"
cat ${PBS_NODEFILE}
np=$(wc -l < ${PBS_NODEFILE})
echo "Cores assigned: ${np}"

When you run this job, you should then get output similar to: 

The nodefile for this job is stored at /var/spool/PBS/aux/33.hpc1.hpc
comp001.hpc
comp001.hpc
comp001.hpc
comp001.hpc
comp001.hpc
comp001.hpc
comp001.hpc
comp001.hpc
comp002.hpc
comp002.hpc
comp002.hpc
comp002.hpc
comp002.hpc
comp002.hpc
comp002.hpc
comp002.hpc
Cores assigned: 16

If you have an application that manually forks processes onto the nodes of your job, you are responsible for parsing the PBS_NODEFILE to determine which nodes those are.

Some MPI implementations require you to feed the PBS_NODEFILE to mpirun, e.g. for OpenMPI one may pass -hostfile ${PBS_NODEFILE}.

Be sure to include the mpiprocs parameter in your resource request. The default is 1, and will cause the PBS_NODEFILE to contain only one line per node if not specified correctly.

Selecting different nodes in one job

Using the above information, one may allocate multiple nodes of the same type, e.g. multiple 48-core nodes. In order to mix multiple different resources, one may use the PBS' "+" notation. For example in order to mix one 48-core node and two 8-core nodes in one PBS job, one may pass: 

[username@hpc1 ~]$ qsub -l select=1:ncpus=48:mpiprocs=48+2:ncpus=8:mpiprocs=8 -l walltime=1:00:00 myscript.sh

Guidelines / Rules

  • Create a temporary working directory in /scratch-small-local or /scratch-large-network, not /tmp
    • /tmp is reserved for use by the operating system, and is only 5GB in size.
    • Preferably specify /scratch-small-local/${PBS_JOBID} in your submit script so that it's easy to associate scratch directories with their jobs.
    • Copy your input files to your scratch space and work on the data there. Avoid using your home directory as much as possible.
    • Copy only your results back to your home directory. Input files that haven't changed don't need to be copied.
    • Erase your temporary working directory when you're done.
  • Secure your work from accidental deletion or contamination by disallowing other users access to your scratch directories
    • umask 0077 disallows access by all other users on newly created files and directories

Examples

ADF

ADF generates run files which are scripts which contain your data. Make sure to convert it to a UNIX file first using dos2unix, and remember to make it executable with chmod +x.

ADF script requesting 4 cores, on 1 node, -m selects to mail begin and end messages and -M is the email address to send to. Requests 1 week walltime. 

#!/bin/bash
#PBS -N JobName
#PBS -l select=1:ncpus=4
#PBS -l walltime=168:00:00
#PBS -m be
#PBS -M username@sun.ac.za

INPUT=inputfile.run

# make sure I'm the only one that can read my output
umask 0077
TMP=/scratch-small-local/${PBS_JOBID}
mkdir -p ${TMP}

if [ ! -d "${TMP}" ]; then
	echo "Cannot create temporary directory. Disk probably full."
	exit 1
fi

cd ${TMP}

module load app/adf/2019.103

# override ADF's scratch directory
export SCM_TMPDIR=${TMP}

# override log file
export SCM_LOGFILE="${TMP}/${PBS_JOBID}.logfile"

# Submit job
${PBS_O_WORKDIR}/${INPUT}

# job done, copy everything back 
echo "Copying from ${TMP}/ to ${PBS_O_WORKDIR}/"
/usr/bin/rsync -vax ${TMP}/ "${PBS_O_WORKDIR}/"

# delete my temporary files
[ $? -eq 0 ] && /bin/rm -rf ${TMP}

ANSYS

Fluent

Fluent script requesting 4 cores, on 1 node, -m selects to mail begin and end messages and -M is the email address to send to. Requests 1 week walltime. 

#!/bin/bash
#PBS -N JobName
#PBS -l select=1:ncpus=4:mpiprocs=4:mem=16GB
#PBS -l license_fluent=4
#PBS -l walltime=168:00:00
#PBS -m be
#PBS -e output.err
#PBS -o output.out
#PBS -M username@sun.ac.za

INPUT=inputfile.jou

# make sure I'm the only one that can read my output
umask 0077
TMP=/scratch-small-local/${PBS_JOBID}
mkdir -p ${TMP}

if [ ! -d "${TMP}" ]; then
	echo "Cannot create temporary directory. Disk probably full."
	exit 1
fi

# copy the input files to ${TMP}
echo "Copying from ${PBS_O_WORKDIR}/ to ${TMP}/"
/usr/bin/rsync -vax "${PBS_O_WORKDIR}"/ ${TMP}/

cd ${TMP}

# choose version of FLUENT
# Use module avail to see which versions of ansys are available 
module load app/ansys/20.1

# Automatically calculate the number of processors
np=$(wc -l < ${PBS_NODEFILE})

fluent 3d -pdefault -cnf=${PBS_NODEFILE} -mpi=intel -g -t${np} -ssh -i ${INPUT}

# job done, copy everything back 
echo "Copying from ${TMP}/ to ${PBS_O_WORKDIR}/"
/usr/bin/rsync -vax ${TMP}/ "${PBS_O_WORKDIR}/"

# delete my temporary files
[ $? -eq 0 ] && /bin/rm -rf ${TMP}

MPI example using 3 nodes and 3 x 18 cores. 

#!/bin/bash

#PBS -l select=3:ncpus=18:mpiprocs=18:mem=28GB
##PBS -l license_fluent=54
#PBS -l walltime=24:00:00
#PBS -l place=scatter
#PBS -N 46krpm-grad

#PBS -e fluent.err
#PBS -o fluent.out
s
INPUT=/home/gerhardv/46krpm-grad/journal-1.jou
TMP=/scratch2
mkdir -p ${TMP}

if [ ! -d "${TMP}" ]; then
        echo "Cannot create temporary directory. Disk probably full."
        exit 1
fi

# copy the input files to ${TMP}
echo "Copying from ${PBS_O_WORKDIR}/ to ${TMP}/"
/usr/bin/rsync -vax "${PBS_O_WORKDIR}"/ ${TMP}/

cd ${TMP}

module load app/ansys/20.1
module load intel/mpi

np=$(wc -l < ${PBS_NODEFILE})
fluent 3ddp -pdefault -cnf=${PBS_NODEFILE} -mpi=intel -g -t${np} -ssh -i ${INPUT}

# job done, copy everything back
echo "Copying from ${TMP}/ to ${PBS_O_WORKDIR}/"
/usr/bin/rsync -vax ${TMP}/ "${PBS_O_WORKDIR}/"

# delete my temporary files
[ $? -eq 0 ] && /bin/rm -rf ${TMP}

Fluid-Structure Interaction

You need the following 5 files:

  • coupling (.sci) file
  • structural data (.dat) file
  • case (.cas.gz) file
  • journal (.jnl) file
  • submit script (.sh)

The coupling file should contain two participants. The names of these participants should not have spaces in them. In the example below, Solution 4 should be renamed to something like Solution4. Make sure to replace all instances of the name in the file. 

<SystemCoupling Ver="1">
  <Participants Count="2">
    <Participant Ver="1" Type="0">
      <Name PropType="string">Solution 4</Name>
      <DisplayName PropType="string">0012 V2</DisplayName>
      <SupportsCouplingIterations PropType="bool">True</SupportsCouplingIterations>
      <UnitSystem PropType="string">MKS_STANDARD</UnitSystem>
      <Regions Count="1">
<--- snip --->

The journal file should contain (replace the filename on the ‘rc’ line with your case file): 

file/start-transcript Solution.trn
file set-batch-options , yes ,
rc FFF-1.1-1-00047.cas.gz
solve/initialize/initialize-flow
(sc-solve)
wcd FluentRestart.cas.gz
exit
ok

The job script is given below. Update the COUPLING, STRUCTURALDATA, JOURNAL and NPA variables to reflect your case. 

#!/bin/bash

#PBS -N fsi
#PBS -l select=1:ncpus=48:mpiprocs=48:mem=90GB
#PBS -l license_fluent=48
#PBS -l walltime=24:00:00

COUPLING=coupling.sci
STRUCTURALDATA=ds.dat
JOURNAL=fluent.journal

# number of processors for Ansys
NPA=8
# Automatically calculate the number of processors left over for Fluent
NP=$(wc -l < ${PBS_NODEFILE})
NPF=$((NP-NPA))

# make sure I'm the only one that can read my output
umask 0077

# create a temporary directory with a random name in /scratch-small-local
TMP=/scratch-small-local/${PBS_JOBID}
mkdir -p ${TMP}
echo "Temporary work dir: ${TMP}"

if [ ! -d "${TMP}" ]; then
	echo "Cannot create temporary directory. Disk probably full."
	exit 1
fi

# copy the input files to ${TMP}
echo "Copying from ${PBS_O_WORKDIR}/ to ${TMP}/"
/usr/bin/rsync -vax "${PBS_O_WORKDIR}/" ${TMP}/

cd ${TMP}

module load app/ansys/20.1

# Start coupling program
/apps/ansys_inc/v162/aisol/.workbench -cmd ansys.services.systemcoupling.exe -inputFile ${COUPLING} &

# Wait until scServer.scs is created
TIMEOUT=60
while [ ! -f scServer.scs -a $TIMEOUT -gt 0 ]; do
	TIMEOUT=$((TIMEOUT-1))
	sleep 2
done

if [ -f scServer.scs ]; then

	# Parse the data in scServer.scs
	readarray JOB < scServer.scs
	HOSTPORT=(${JOB[0]//@/ })

	# Run Fluent
	fluent 3ddp -g -t${NPF} -driver null -ssh -scport=${HOSTPORT[0]} -schost=${HOSTPORT[1]} -scname="${JOB[4]}" < ${JOURNAL} > output.FLUENT &

	# Run Ansys
	ansys162 -b -scport=${HOSTPORT[0]} -schost=${HOSTPORT[1]} -scname="${JOB[2]}" -i ${STRUCTURALDATA} -o output.ANSYS -np ${NPA}

	# job done, copy everything back
	echo "Copying from ${TMP}/ to ${PBS_O_WORKDIR}/"
	/usr/bin/rsync -vax ${TMP}/ "${PBS_O_WORKDIR}/"

fi

# delete my temporary files
[ $? -eq 0 ] && /bin/rm -rf ${TMP}

CFX

CFX script requesting 4 cores, on 1 node, -m selects to mail begin and end messages and -M is the email address to send to. Requests 1 week walltime. 

#!/bin/bash
#PBS -N JobName
#PBS -l select=1:ncpus=4:mpiprocs=4
#PBS -l walltime=168:00:00
#PBS -m be
#PBS -e output.err
#PBS -o output.out
#PBS -M username@sun.ac.za

DEF=inputfile.def
INI=inputfile.ini

# make sure I'm the only one that can read my output
umask 0077
TMP=/scratch-small-local/${PBS_JOBID}
mkdir -p ${TMP}

if [ ! -d "${TMP}" ]; then
	echo "Cannot create temporary directory. Disk probably full."
	exit 1
fi

# copy the input files to ${TMP}
echo "Copying from ${PBS_O_WORKDIR}/ to ${TMP}/"
/usr/bin/rsync -vax "${PBS_O_WORKDIR}"/ ${TMP}/

cd ${TMP}

module load app/ansys/16.2

# get list of processors
PAR=$(sed -e '{:q;N;s/\n/,/g;t q}' ${PBS_NODEFILE})

cfx5solve -def ${DEF} -ini ${INI} -par-dist ${PAR}

# job done, copy everything back 
echo "Copying from ${TMP}/ to ${PBS_O_WORKDIR}/"
/usr/bin/rsync -vax ${TMP}/ "${PBS_O_WORKDIR}/"

# delete my temporary files
[ $? -eq 0 ] && /bin/rm -rf ${TMP}

Abaqus

Abaqus script requesting 4 cores, on 1 node, -m selects to mail begin and end messages and -M is the email address to send to. 

#!/bin/bash
#PBS -l select=1:ncpus=4:mpiprocs=4
#PBS -l walltime=1:00:00
#PBS -m be
#PBS -M username@sun.ac.za

# the input file without the .inp extension
JOBNAME=xyz

# make sure I'm the only one that can read my output
umask 0077
TMP=/scratch-small-local/${PBS_JOBID}
mkdir -p ${TMP}

if [ ! -d "${TMP}" ]; then
	echo "Cannot create temporary directory. Disk probably full."
	exit 1
fi

# copy the input files to ${TMP}
echo "Copying from ${PBS_O_WORKDIR}/ to ${TMP}/"
/usr/bin/rsync -vax "${PBS_O_WORKDIR}"/ ${TMP}/

cd ${TMP}

module load app/abaqus

# Automatically calculate the number of processors
np=$(wc -l < ${PBS_NODEFILE})

abaqus job=${JOBNAME} input=${JOBNAME}.inp analysis cpus=${np} scratch=${TMP} interactive
wait

# job done, copy everything back 
echo "Copying from ${TMP}/ to ${PBS_O_WORKDIR}/"
/usr/bin/rsync -vax ${TMP}/ "${PBS_O_WORKDIR}/"

# delete my temporary files
[ $? -eq 0 ] && /bin/rm -rf ${TMP}

R

R script requesting 1 node, -m selects to mail begin and end messages and -M is the email address to send to. 

#!/bin/bash

#PBS -l select=1:ncpus=1
#PBS -l walltime=00:30:00
#PBS -M username@sun.ac.za
#PBS -m be

cd ${PBS_O_WORKDIR}

module load app/R/4.0.2

R CMD BATCH script.R

CPMD

CPMD script requesting 8 cores on 1 node, -N names the job 'cpmd', -m selects to mail end message and -M is the email address to send to. CPMD runs with MPI which needs to be told which nodes it may use. The list of nodes it may use is given in $PBS_NODEFILE. 

#!/bin/bash
#PBS -N cpmd
#PBS -l select=1:ncpus=8:mpiprocs=8
#PBS -l walltime=1:00:00
#PBS -m e
#PBS -M username@sun.ac.za

module load compilers/gcc-4.8.2
module load openmpi-x86_64

cd ${PBS_O_WORKDIR}

# Automatically calculate the number of processors
np=$(wc -l < ${PBS_NODEFILE})

mpirun -np ${np} --hostfile ${PBS_NODEFILE} /apps/CPMD/3.17.1/cpmd.x xyz.inp > xyz.out

Gaussian

Gaussian has massive temporary files (.rwf file). Generally we don't care about this file afterward, so this script doesn't copy it from temporary storage after job completion. Requests 6 week walltime. 

#!/bin/bash
#PBS -N SomeHecticallyChemicalName
#PBS -l select=1:ncpus=8:mpiprocs=8:mem=16GB:scratch=true
#PBS -l walltime=1008:00:00
#PBS -m be

INPUT=input.cor

# make sure I'm the only one that can read my output
umask 0077
TMP=/scratch-small-local/${PBS_JOBID}
TMP2=/scratch-large-network/${PBS_JOBID}
mkdir -p ${TMP} ${TMP2}

if [ ! -d "${TMP}" ]; then
	echo "Cannot create temporary directory. Disk probably full."
	exit 1
fi

if [ ! -d "${TMP2}" ]; then
	echo "Cannot create temporary directory. Disk probably full."
	exit 1
fi

export GAUSS_SCRDIR=${TMP}

# copy the input files to ${TMP}
echo "Copying from ${PBS_O_WORKDIR}/ to ${TMP}/"
/usr/bin/rsync -vax "${PBS_O_WORKDIR}"/ ${TMP}/

cd ${TMP}

# make sure input file has %RWF line for specifying temporary storage
if [ -z "$(/bin/grep ^%RWF ${INPUT})" ]; then
	/bin/sed -i '1s/^/%RWF\n/' ${INPUT}
fi

# update input file to use temporary storage, split into 500GB files
/bin/sed -i -E "s|%RWF(.*)|%RWF=${TMP}/1.rwf,200GB,${TMP2}/2.rwf,500GB,${TMP2}/3.rwf,500GB,${TMP2}/4.rwf,500GB,${TMP2}/,-1|g" ${TMP}/${INPUT}

. /apps/Gaussian/09D/g09/bsd/g09.profile

/apps/Gaussian/09D/g09/g09 ${INPUT} > output.log

# job done, copy everything except .rwf back 
echo "Copying from ${TMP}/ to ${PBS_O_WORKDIR}/"
/usr/bin/rsync -vax --exclude=*.rwf ${TMP}/ "${PBS_O_WORKDIR}/"

# delete my temporary files
[ $? -eq 0 ] && /bin/rm -rf ${TMP}

This script also requires that the input file contains a line starting with %RWF. This is so that the script can update the input file to specify that only the first part of the RWF be written to the compute node's local scratch space.

pisoFOAM

pisoFOAM generates a lot of output, not all of which is useful. In this example we use crontab to schedule the deletion of unwanted output while the job runs. Requests 3 week walltime. 

#!/bin/bash
#PBS -l select=1:ncpus=8:mpiprocs=8:large-scratch=true
#PBS -l walltime=504:00:00
#PBS -m be
 
# make sure I'm the only one that can read my output
umask 0077
# create a temporary directory in /scratch-small-local
TMP=/scratch-small-local/${PBS_JOBID}
/bin/mkdir ${TMP}
echo "Temporary work dir: ${TMP}"

if [ ! -d "${TMP}" ]; then
	echo "Cannot create temporary directory. Disk probably full."
	exit 1
fi

# copy the input files to ${TMP}
echo "Copying from ${PBS_O_WORKDIR}/ to ${TMP}/"
/usr/bin/rsync -vax "${PBS_O_WORKDIR}/" ${TMP}/

cd ${TMP}
 
# start crontab, delete unwanted files every 6 hours
/bin/echo "0 */6 * * * /bin/find ${TMP} -regextype posix-egrep -regex '(${TMP}/processor[0-9]+)/([^/]*)/((uniform/.*)|ddt.*|phi.*|.*_0.*)' -exec rm {} \\;" | /usr/bin/crontab

# Automatically calculate the number of processors
np=$(wc -l < ${PBS_NODEFILE})

module load compilers/gcc-4.8.2
module load openmpi/1.6.5

export MPI_BUFFER_SIZE=200000000
 
export FOAM_INST_DIR=/apps/OpenFOAM
foamDotFile=${FOAM_INST_DIR}/OpenFOAM-2.2.2/etc/bashrc
[ -f ${foamDotFile} ] && . ${foamDotFile}

blockMesh
decomposePar
 
mpirun -np ${np} pisoFoam -parallel > ${PBS_O_WORKDIR}/output.log
 
# remove crontab entry (assumes I only have one on this node)
/usr/bin/crontab -r
 
# job done, copy everything back
echo "Copying from ${TMP}/ to ${PBS_O_WORKDIR}/"
/usr/bin/rsync -vax --exclude "*_0.gz" --exclude "phi*.gz" --exclude "ddt*.gz" ${TMP}/ "${PBS_O_WORKDIR}/"
 
# delete my temporary files
[ $? -eq 0 ] && /bin/rm -rf ${TMP}

MSC Marc

Marc script requesting 8 cores, on 1 node, -m selects to mail end message and -M is the email address to send to. 

#!/bin/bash
#PBS -N JobName
#PBS -l select=1:ncpus=8:mpiprocs=8
#PBS -l walltime=24:00:00
#PBS -l license_marc=8
#PBS -m e

INPUT=inputfile

# make sure I'm the only one that can read my output
umask 0077
TMP=/scratch-small-local/${PBS_JOBID}
mkdir -p ${TMP}

if [ ! -d "${TMP}" ]; then
	echo "Cannot create temporary directory. Disk probably full."
	exit 1
fi

# copy the input files to ${TMP}
echo "Copying from ${PBS_O_WORKDIR}/ to ${TMP}/"
/usr/bin/rsync -vax "${PBS_O_WORKDIR}"/ ${TMP}/

cd ${TMP}

module load app/marc

# get number of processors assigned
NPS=$(wc -l < ${PBS_NODEFILE})
HOSTS=hosts.${PBS_JOBID}

[ -f ${HOSTS} ] && /bin/rm ${HOSTS}
# create hosts file
uniq -c ${PBS_NODEFILE} | while read np host; do
	/bin/echo "${host} ${np}" >> ${HOSTS}
done

if [ ${NPS} -gt 1 ]; then
	run_marc -j ${INPUT} -ver n -back n -ci n -cr n -nps ${NPS} -host ${HOSTS}
else
	run_marc -j ${INPUT} -ver n -back n -ci n -cr n
fi

# job done, copy everything back 
echo "Copying from ${TMP}/ to ${PBS_O_WORKDIR}/"
/usr/bin/rsync -vax ${TMP}/ "${PBS_O_WORKDIR}/"

# delete my temporary files
[ $? -eq 0 ] && /bin/rm -rf ${TMP}

mothur

mothur has massive data volumes, and therefore has to use local scratch space to avoid killing the file server. Requests 1 core on 1 node.

mothur's input can either be a file with all the commands to process listed, or the commands can be given on the commandline if prefixed with a #. 

#!/bin/bash

#PBS -l select=1:ncpus=1:mpiprocs=1
#PBS -l walltime=24:00:00
#PBS -m e

# make sure I'm the only one that can read my output
umask 0077
TMP=/scratch-small-local/${PBS_JOBID}
mkdir -p ${TMP}

if [ ! -d "${TMP}" ]; then
	echo "Cannot create temporary directory. Disk probably full."
	exit 1
fi

# copy the input files to ${TMP}
echo "Copying from ${PBS_O_WORKDIR}/ to ${TMP}/"
/usr/bin/rsync -vax "${PBS_O_WORKDIR}"/ ${TMP}/

cd ${TMP}

module load app/mothur

# Automatically calculate the number of processors
np=$(wc -l < ${PBS_NODEFILE})

mothur inputfile

# could also put the commands on the command line
#mothur "#cluster.split(column=file.dist, name=file.names, large=T, processors=${np})"

# job done, copy everything back 
echo "Copying from ${TMP}/ to ${PBS_O_WORKDIR}/"
/usr/bin/rsync -vax ${TMP}/ "${PBS_O_WORKDIR}/"

# delete my temporary files
[ $? -eq 0 ] && /bin/rm -rf ${TMP}

Altair FEKO

#!/bin/bash

#PBS -l select=1:ncpus=10:mpiprocs=10:scratch=true:mem=100GB
#PBS -l walltime=48:00:00
#PBS -m e

INPUT=input.cfx

# make sure I'm the only one that can read my output
umask 0077
TMP=/scratch-small-local/${PBS_JOBID}
mkdir -p ${TMP}

if [ ! -d "${TMP}" ]; then
	echo "Cannot create temporary directory. Disk probably full."
	exit 1
fi

. /apps/altair/hyperworks/2017/altair/feko/bin/initfeko
# override job directory
export FEKO_USER_HOME=${PBS_O_WORKDIR}
# override temporary directory
export FEKO_TMPDIR=${TMP}
# limit to the 100GB requested
export FEKO_MAXALLOCM=$((1024 * 100))

runfeko ${INPUT} --use-job-scheduler

# delete my temporary files
[ $? -eq 0 ] && /bin/rm -rf ${TMP}

Numeca

This script assumes you have 4 Numeca files available for your project. If your project is named proj1, the required files are proj1.iec, proj1.igg, proj1.bcs and proj1.cgns.

Script requests 4 hours walltime, and uses 8 cores. 

#!/bin/bash

#PBS -N proj1
#PBS -l select=1:ncpus=8:mpiprocs=8
#PBS -l walltime=04:00:00

INPUT=proj1

# make sure I'm the only one that can read my output
umask 0077
# create a temporary directory with the job id as name in /scratch-small-local
TMP=/scratch-small-local/${PBS_JOBID}
mkdir -p ${TMP}
echo "Temporary work dir: ${TMP}"

# copy the input files to ${TMP}
echo "Copying from ${PBS_O_WORKDIR}/ to ${TMP}/"
/usr/bin/rsync -vax "${PBS_O_WORKDIR}/" ${TMP}/

cd ${TMP}

NUMECA=/apps/numeca/bin
VERSION=90_3

# create hosts list
TMPH=$(/bin/mktemp)
/usr/bin/tail -n +2 ${PBS_NODEFILE} | /usr/bin/uniq -c | while read np host; do
    /bin/echo "${host} ${np}" >> ${TMPH}
done
NHOSTS=$(wc -l < ${TMPH})
LHOSTS=$(while read line; do echo -n ${line}; done < ${TMPH})
/bin/rm ${TMPH}

# Create .run file
${NUMECA}/fine -niversion ${VERSION} -batch ${INPUT}.iec ${INPUT}.igg ${PBS_JOBID}.run

# Set up parallel run
${NUMECA}/fine -niversion ${VERSION} -batch -parallel ${PBS_JOBID}.run ${NHOSTS} ${LHOSTS}

# Start solver
${NUMECA}/euranusTurbo_parallel ${PBS_JOBID}.run -steering ${PBS_JOBID}.steering -niversion ${VERSION} -p4pg ${PBS_JOBID}.p4pg

# job done, copy everything back
echo "Copying from ${TMP}/ to ${PBS_O_WORKDIR}/"
/usr/bin/rsync -vax ${TMP}/ "${PBS_O_WORKDIR}/"

# delete my temporary files
[ $? -eq 0 ] && /bin/rm -rf ${TMP}

This script assumes you have 3 Numeca files available for your project. If your project is named proj1, the required files are proj1.iec, proj1.trb and proj1.geomTurbo.

Script requests 4 hours walltime, and uses 8 cores. 

#!/bin/bash

#PBS -N proj1
#PBS -l select=1:ncpus=8:mpiprocs=8
#PBS -l walltime=04:00:00

INPUT=proj1

# make sure I'm the only one that can read my output
umask 0077
# create a temporary directory with the job id as name in /scratch-small-local
TMP=/scratch-small-local/${PBS_JOBID}
mkdir -p ${TMP}
echo "Temporary work dir: ${TMP}"

# copy the input files to ${TMP}
echo "Copying from ${PBS_O_WORKDIR}/ to ${TMP}/"
/usr/bin/rsync -vax "${PBS_O_WORKDIR}/" ${TMP}/

# Automatically calculate the number of processors - 1 less than requested, used for master process
NP=$(( $(wc -l < ${PBS_NODEFILE}) - 1))

cd ${TMP}

NUMECA=/apps/numeca/bin
VERSION=101

# inputs .trb, .geomTurbo
# outputs .bcs, .cgns, .igg
/usr/bin/xvfb-run -d ${NUMECA}/igg -print -batch -niversion ${VERSION} -autogrid5 -trb ${TMP}/${INPUT}.trb -geomTurbo ${TMP}/${INPUT}.geomTurbo -mesh ${TMP}/${PBS_JOBID}.igg

# inputs .iec, .igg
# outputs .run
/usr/bin/xvfb-run -d ${NUMECA}/fine -print -batch -niversion ${VERSION} -project ${TMP}/${INPUT}.iec -mesh ${TMP}/${PBS_JOBID}.igg -computation ${TMP}/${PBS_JOBID}.run

# inputs .run
# outputs .p4pg, .batch
/usr/bin/xvfb-run -d ${NUMECA}/fine -print -batch -niversion ${VERSION} -parallel -computation ${TMP}/${PBS_JOBID}.run -nproc ${NP} -nbint 128 -nbreal 128

# inputs .run, .p4pg
${NUMECA}/euranusTurbo_parallel${VERSION} ${TMP}/${PBS_JOBID}.run -steering ${TMP}/${PBS_JOBID}.steering -p4pg ${TMP}/${PBS_JOBID}.p4pg

# job done, copy everything back
echo "Copying from ${TMP}/ to ${PBS_O_WORKDIR}/"
/usr/bin/rsync -vax ${TMP}/ "${PBS_O_WORKDIR}/"

# delete my temporary files
[ $? -eq 0 ] && /bin/rm -rf ${TMP}

Programs that handle job submission differently

MATLAB

Matlab R2023a can be used on the hpc cluster with a VPN connection. Enquire gerhardv@sun.ac.za

The MATLAB campus license allows all students to install MATLAB on their own computers. See this link for details. The HPC currently has MATLAB R2023a, R2022b and R2021a installed.

Furthermore, the license includes a MATLAB Parallel Server license that means MATLAB jobs can run on the high performance cluster (hpc1). Prior to R2019a, MATLAB Parallel Server was called MATLAB Distributed Computing Server. With MATLAB's Parallel Computing Toolbox (now freely available on the campus license), it's possible to submit your MATLAB code directly from your desktop to the HPC without writing submit scripts and submitting the job manually. It is also possible to compile MATLAB executables and run these on the HPC with a bash script. The advantage of doing this is that it does not consume worker licenses so freeing up the development environment for code writing and testing.

Setup

  1. Download and run the integration script
  2. The script (parallel_pbs) should be unzipped and placed in the MATLAB path. C:\Users\me\Documents\MATLAB\ is a good place.
  3. These files will be used to configure and establish the connection with the MATLAB parallel server on the HPC.

Validate

If your PCT is installed and licensed correctly, you should see a dropdown named Parallel in your toolbar.

  1. Open the Parallel dropdown, and select Manage Cluster Profiles... (screenshot)
    'hpc1 remote R2023a' should be in the list of Cluster Profiles (screenshot)
  2. Select the 'Validation' tab
    • Set the 'Number of workers to use' to 4 and click Validate (screenshot)
    You will be prompted for your HPC username. When prompted for an identity file, select No if you don't know what it is.
    Depending on how busy the HPC is, the testing should complete in 10 to 30 minutes. (screenshot)
    If the last step fails, your hostname is most probably incorrectly set up.

GPU

To use the GPU's specify the ngpus= parameter in a qsub -I for interactive session, or in the bash script - add it to the line that normally has the mem= and ncpus= parameters. Specify the queue with -q ee and Qlist=ee. See below for an example for an interactive session. The mem= ncpus= values can be changed within reason. The bash script structure and other aspects of HPC use is described on https://www0.sun.ac.za/hpc

If you want to use the GPUs message gerhardv@sun.ac.za for inclusion on the GPU queue. 

[gerhardv@hpc1 ~]$ qsub -I -l walltime=2:00:00 -q ee -l select=1:ncpus=4:mem=4GB:ngpus=1:Qlist=ee
qsub: waiting for job 253270.hpc1.hpc to start
qsub: job 253270.hpc1.hpc ready
[gerhardv@comp048 ~]$

The GPUs are commonly used via python with tensorflow or pytorch. Other GPU bindings are possible. Software sometimes automatically uses GPUs if they are present on the system, and R and MATLAB can also be configured to invoke GPU. In bash script form it would look something like this: 

#!/bin/bash
#PBS -N JobName

#PBS -l select=1:ncpus=2:mem=16GB:ngpus=1:Qlist=ee
#PBS -l walltime=16:00:00
#PBS -m be
#PBS -e output.err
#PBS -o output.out
#PBS -M username@sun.ac.za

INPUT=inputfile

# make sure I'm the only one that can read my output
umask 0077
TMP=/scratch-small-local/${PBS_JOBID}
mkdir -p ${TMP}

if [ ! -d "${TMP}" ]; then
	echo "Cannot create temporary directory. Disk probably full."
	exit 1
fi

# copy the input files to ${TMP}
echo "Copying from ${PBS_O_WORKDIR}/ to ${TMP}/"
/usr/bin/rsync -vax "${PBS_O_WORKDIR}"/ ${TMP}/

cd ${TMP}

# choose version of python that has tensorflow/pytorch
module load python/3.8.1

python3 yourcode.py

# job done, copy everything back 
echo "Copying from ${TMP}/ to ${PBS_O_WORKDIR}/"
/usr/bin/rsync -vax ${TMP}/ "${PBS_O_WORKDIR}/"

# delete my temporary files
[ $? -eq 0 ] && /bin/rm -rf ${TMP}

Typically the code would be developed offline on a notebook or with an interactive session and then benchmarked on the HPC with a bash script to optimise walltime and memory. The python code would then be run as a scheduler job using a bash script after development and debugging in interactive mode. Interactive session is launched as above with qsub -I. 

[gerhardv@comp050 ~]$ module load python/3.8.1
[gerhardv@comp050 ~]$ python3
Python 3.8.1 (default, Jun 11 2021, 05:57:44)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tp
>>> import torch as tt
>>>

There is a conda environment for tensorflow and python 3.9.13 

[gerhardv@comp050 ~]$ module load app/tensorflow/2.9.2
(tf) [gerhardv@comp049 ~]$ python3
Python 3.9.13 (main, Aug 25 2022, 23:26:10)
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
2022-09-08 12:29:15.954105: I tensorflow/core/util/util.cc:169] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
>>> print(tf.__version__)
2.9.2
>>>
>>> print(tf.config.list_physical_devices('GPU'))
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:1', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:2', device_type='GPU')]
Retrieved from "https://www0.sun.ac.za/hpc/index.php?title=HOWTO_submit_jobs&oldid=830"