In this exercise, we submit our first job, that only prints a message. This is to illustrate the concept of job submission.

When you connect to a cluster via ssh, you connect to a login node. A login node shall be used exclusively to submit jobs to compute nodes, organise your files and data, and transfer them to your local machine or other servers. You should absolutely not perform computations on a login node because they are already slow and you will make them even slower for everybody else.
So instead of running your script on the login node, you have to use the sbatch command to send the instruction to run the script to a compute node. The arguments of the sbatch command are used to specify the specifications you need for the compute node (i.e. the amount of cpu cores, GPUs, RAM, …). You could pass all arguments and the instruction to the sbatch command inline, something like :

sbatch --account=rrg-pbellec --time=1:0:0 --cpus-per-task=8 --mem=32G submit.sh
 

But it is usually more convenient to define these arguments in the job script with the #SBATCH prefix at the beginning of the line.

Instructions :

• Job options must be set with “#SBATCH …” lines on top of the script, but below the “#!/bin/bash” line.
• Modify submit.sh to specify your account, e.g. #SBATCH --account=rrg-pbellec.
• Modify submit.sh to specify 1 node with 1 processor for 2 minutes.
• Modify submit.sh to specify the job name “ex1”.
• Submit the job with the following command:

   sbatch submit.sh
 

• Please note the job ID.
• Verify the status of your job with:

   squeue -u $USER
 

• Verify the result of your job in slurm-JOB_ID.out

Additional information:

• Alliance Canada users are specifying accounts in the form of: TYPE-NAME, where TYPE can be def, rrg, rpp or ctb, and NAME is the PI’s username. def accounts are the defaults accounts with the default priority, so if your PI has a non default account like rrg, rpp or ctb it is almost always preferable to use the non default account.
• The file slurm-JOB_ID.out constains both standard and error outputs (stdout and stderr) of the job. If it failed, you should check in this file. In the case of this exercise, you should see “Bonjour”.

In this exercise, we want to apply the “grayscale” filter to one picture.

==== Instructions ====

We will use the filterImage.exe application to convert one picture to grayscale.

• Modify submit.sh in order to load required modules with module load gcc boost and call filterImage.exe with the “grayscale” filter and one picture file. Use ../filterImage.exe --help to find the arguments to use.
• Submit the job with the following command:

   sbatch submit.sh
 

• Verify that the job has generated one image file in the current folder. Note: you may download this file to your local computer with any SCP client. The visual result can show errors on some lines of pixels. The goal of this exercise is only to practice job submission.

Additional information:
The help message of filterImage.exe being in French, here is a translation :

Apply filters on a series of images
 Options:
   -h [ --help ]         Print help messages
   -i [ --srcdir ] arg   Source directory ?
   -o [ --dstdir ] arg   Destination directory ?
   --files arg           File list ?
   --filters arg         Filter(s) to apply : grayscale, edges, emboss,
                         negate, solarize, flip, flop, monochrome, add_noise
   --combined arg (=0)  Should the filters be combined
                        (applied on after the other on each image) ?
 

Like in exercise #2, we want to apply the “grayscale” filter to multiple pictures. In order to use multiple cores on a compute node, we will run multiple instances of the executable simultaneously.

This time, we will use a tool called GNU-Parallel, which is more flexible while having a compact syntax.

==== GNU parallel ====

GNU parallel is a powerful tool to execute parallel tasks. It supports two main input formats:

• Lists of values on the command line.
• Lists of values in a file.

Please check the Alliance Canada wiki for basic options, and the official documentation for advanced options.

For this exercise, we will reuse the output of the “ls” command with $( ). For example, to display the content of “../photos” in parallel, we would use GNU parallel the following way:

   parallel echo {1} ::: $(ls ../photos)
 

In this example, {1} refers to the first variable in the command template. The operator “:::” separates the template from values for the first variable. We could define multiple variables ({1}, {2}, etc.) by adding more “:::” operators at the end of the parallel command.

==== Instructions ====

We will use the parallel command to convert all pictures in “../photos”:

• Request 2 cores with the –cpus-per-task option in the job submission script header.
• Use the parallel command on filterImage.exe to run it on the list of files in “../photos” (you can mimic the example line above, replacing echo by the adequate command).
• Submit the job with the command.

   sbatch submit.sh
 

• Verify that the job generates multiple images in the current folder.

==== Advanced Instructions ====

Modify your job script so that it also creates images applying the “negate” filters (it shouldn’t apply both filters at the same time but for each input image, it should create two output files, one with the grayscale filter and one with the negate filter). It should still do that in parallel, so you have to define a second variable to your command template that corresponds to the filter.

• Verify that the job generates twice as many files in the local folder.

==== No need to specify the number of simultaneous tasks? ====

By default, GNU parallel will use one core per task, and it will launch as many tasks as there are cores on the system. As soon as a task is completed, the next one will start automatically.

You may change the default behavious with the “-j” option (see the man page with the man parallel command).

Like in exercise #3, we want to apply one filter to a set of pictures. But, this time, we want to apply different filters, one specific filter per job. In this exercise, we will use a double strategy for parallel computing:

• For each job, we will use GNU parallel to process a set of pictures.
• We will submit a job array in order to run multiple instances of the same kind of job.

==== GNU parallel ====

Please check ../3-gnu-parallel/README.en for the description of this tool.

==== Job Arrays ====

Job arrays are a parallel mechanism offered by the scheduler. The global job array is separated in multiple job instances that are started independently. Each job instance will be identified by a different value in the SLURM_ARRAY_TASK_ID environment variable. The possible values are defined by the job array.

The syntax is the following:

  #SBATCH --array=<start>-<end>:<step>
 

For example, for a job array of 5 jobs, where SLURM_ARRAY_TASK_ID would be 1, 3, 5, 7 and 9, we would request:

  #SBATCH --array=1-9:2
 

Alternatively, you can define only specific values by listing them separated by commas, for example if you only want values 2, 5, 8 and 22, you would use :

  #SBATCH --array=2,5,8,22
 

==== An Array in Bash ====

Bash supports arrays. A Bash array can be declared the following way:

  MY_ARRAY=(value1 value2 value3 value4)
 

To access a specific element in the array:

  ${MY_ARRAY[$i]}
 

where $i would be a variable having values from 0 through 3. That means ‘value1’ is at position 0, and ‘value4’ is at position 3.

==== Instructions ====

We will use the parallel command to transform all pictures. We will also use a job array to apply a different filter for each job.

• Modify submit.sh to define a job array such that the index will go from 0 to 8 inclusive.
• Use the value of SLURM_ARRAY_TASK_ID to get the proper filter in FILTERS.
• Submit the job array with the following command: sbatch submit.sh
• Verify that the job array is running fine. Many(!) files should have been created.

So far, we have used filterImage.exe in serial mode, but this is in fact an executable that can run in parallel on multiple nodes with MPI.

==== MPI ====

“MPI” means “Message Passing Interface”. An MPI application can split the workload on multiple compute nodes, and each part of the task is computed in parallel in different processes. Coding an MPI application is relatively complex. On the other hand, using such an application remains simple. All one needs to do is to use the “mpiexec” command. For example:

  mpiexec ../filterImage.exe ....
 

Note: if some application does not use MPI to split the workload, all created processes will do the same workload, and all the workload many times. For example: mpiexec hostname.

Note: mpiexec is smart enough to get your SLURM environment and determine how many processes must be started on each node.

==== Instructions ====

The filterImage.exe application uses MPI to process multiple images simultaneously on multiple nodes. For this exercise, we will process all pictures with 4 processors, i.e. two nodes and two cores per node.

• Modify submit.sh to request 2 nodes, 2 tasks per node and 1 core per task.
• Use mpiexec with filterImage.exe.
• Submit the job with the following command: sbatch submit.sh
• Verify that the task is running properly. You should get new images in the current directory.

==== Bonus ====

By default, when filterImage.exe receives a list of filters, it will apply each filter separately on each image. This allows an easy parallelism. The executable also accepts an option “–combined true”, which allows to combine listed filters and apply them in order. In other words, the executable applies the first filter on the original image, then the second filter on the result of the first filter, and so on. Unfortunately, combining filters reduces the granularity of the parallelism because each filter depends on the previous result.

Nevertheless, combining filters may generate interesting results. By using one pair of cores (to not waste resources), combine both filters “add_noise” and “monochrome”. Compare the resulting image to the monochrome-only one. What do you see? Can you explain it?