Difference between revisions of "Submitting CPU Jobs"

From UFAL AIC
(→‎How to read output epilog)
(→‎Selected submit options)
 
(37 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 +
The CPU jobs should be submitted to <code>cpu</code> partition.
 +
 +
You can submit a non-interactive job using the '''sbatch''' command.
 +
To submit an interactive job, use the '''srun''' command:
 +
 +
srun --pty bash
 +
 
== Resource specification ==
 
== Resource specification ==
 +
 +
You should specify the memory and CPU requirements (if higher than the defaults) and don't exceed them.
 +
If your job needs more than one CPU (thread) (on a single machine) for most of the time, reserve the given number of CPU threads with the <code>--cpus-per-task</code> and memory with the <code>--mem</code> options. 
 +
 +
srun -p cpu --cpus-per-task=4 --mem=8G --pty bash
 +
 +
This will give you an interactive shell with 4 threads and 8G RAM on the ''cpu'' partition.
 +
 
== Monitoring and interaction ==
 
== Monitoring and interaction ==
  
−
== Job monitoring ==
+
=== Job monitoring ===
−
We should be able to see what is going on when we run a job. Following examples shows typical usage of the command <code>qstat</code>:
+
We should be able to see what is going on when we run a job. Following examples shows usage of some typical commands:
−
* <code>qstat</code> - this way we inspect all our jobs (both waiting in the queue and scheduled, i.e. running).
+
* <code>squeue -a</code> - this shows the jobs in all partitions.
−
* <code>qstat -u '*' | less</code> - this shows the jobs of all users.
+
* <code>squeue -u user</code> - print a list of running/waiting jobs of a given user
−
* <code>qstat -j 121144</code> - this shows detailed info about the job with this number (if it is still running).
+
* <code>squeue -j<JOB_ID></code> - this shows detailed info about the job with given JOB_ID (if it is still running).
 +
* <code>sinfo</code> - print available/total resources
 +
 
 +
=== Job interaction ===
 +
* <code>scontrol show job JOBID</code> - this shows details of running job with JOBID
 +
* <code>scancel JOBID</code> - delete job from the queue
 +
 
 +
=== Selected submit options ===
 +
The complete list of available options for the commands <code>srun</code> and <code>sbatch</code> can be found in [https://slurm.schedmd.com/man_index.html SLURM documentation]. Most of the options listed here can be entered as a command parameters or as an SBATCH directive inside of a script.
 +
 
 +
  -J helloWorld        # name of job
 +
  --chdir /job/path/    # path where the job will be executed
 +
  -p gpu                # name of partition or queue (if not specified default partition is used)
 +
  -q normal            # QOS level (sets priority of the job)
 +
  -c 4                  # reserve 4 CPU threads
 +
  --gres=gpu:1          # reserve 1 GPU card
 +
  -o script.out        # name of output file for the job
 +
  -e script.err        # name of error file for the job
 +
 
 +
== Array jobs ==
 +
If you need to submit rather large number of jobs which are similar (i.e. processing a large number of input files) you should consider launching an ''array job''.
 +
 
 +
For example, one might need to process 1000 files named <code>file_N.txt</code> (where N is a number between 1-1000).
 +
A program that can process one file is called <code>crunchFile</code> and it takes only one argument - the name of the file to process. Instead of calling 1000x:
 +
  sbatch crunchFile file_N.txt
 +
 
 +
we can write a wrapper script <code>crunchScript.sh</code> referring to the SLURM variable <code>SLURM_ARRAY_TASK_ID</code>:
  
−
== Output monitoring ==
+
  #!/bin/bash
−
If we need to see output produced by our job (suppose the ID is 121144), we can inspect the job's output (in our case stored in <code>job_script.sh.o121144</code>) with:<br>
+
  #SBATCH -p CPU
−
<code>less job_script.sh.o*</code><br>
+
  #SBATCH --mem 2G
−
''Hint:'' if the job is still running, press '''F''' in <code>less</code> to simulate <code>tail -f</code>.
+
 
 +
  crunchFile name_${SLURM_ARRAY_TASK_ID}.txt
  
−
=== How to read output epilog ===
+
and submit all the jobs at once as an ''array job'':
−
The epilog section contains some interesting pieces of information. However this it can get confusing sometimes.
 
  
−
======= EPILOG: Tue Jun 4 12:41:07 CEST 2019
+
   sbatch --array=1-1000%20 crunchScript.sh
−
== Limits:    
 
−
== Usage:    cpu=00:00:00, mem=0.00000 GB s, io=0.00000 GB, vmem=N/A, maxvmem=N/A
 
−
== Duration: 00:00:00 (0 s)
 
−
== Server name: cpu-node13
 
  
−
* ''Limits'' - on this line you can see job limits specified through <code>qsub</code> options
+
Where the option <code>--array 1-1000%20</code> means that we want SLURM to:
−
* ''Usage'' - resource usage during computation
+
* launch 1000 instances of <code>crunchScript.sh</code>
−
** ''cpu=HH:MM:SS'' - the accumulated CPU time usage
+
* each instance will be launched with <code>SLURM_ARRAY_TASK_ID</code> set to a number in the specified range
−
** ''mem=XY GB'' - gigabytes of RAM used times the duration of the job in seconds, so don't be afraid XY is usually a very high number (unlike in this toy example)
+
* there will be at most 20 parallel tasks running at once. This is useful for a larger number of tasks - this way we ensure that we do not flood the cluster with requests.
−
** ''io=XY GB'' - the amount of data transferred in input/output operations in GB
 
−
** ''vmem=XY'' - actual virtual memory consumption when the job finished
 
−
** ''maxvmem=XY'' - peak virtual memory consumption
 
−
* ''Duration'' - total execution time
 
−
* ''Server name'' - name of the executing server
 
  
−
== Output ==
+
You can read more about ''array jobs'' from the [https://slurm.schedmd.com/job_array.html SLURM documentation].
−
== Logs ==
 

Latest revision as of 16:49, 23 April 2024

The CPU jobs should be submitted to cpu partition.

You can submit a non-interactive job using the sbatch command. To submit an interactive job, use the srun command: 

srun --pty bash

Resource specification

You should specify the memory and CPU requirements (if higher than the defaults) and don't exceed them. If your job needs more than one CPU (thread) (on a single machine) for most of the time, reserve the given number of CPU threads with the --cpus-per-task and memory with the --mem options. 

srun -p cpu --cpus-per-task=4 --mem=8G --pty bash

This will give you an interactive shell with 4 threads and 8G RAM on the cpu partition.

Monitoring and interaction

Job monitoring

We should be able to see what is going on when we run a job. Following examples shows usage of some typical commands:

  • squeue -a - this shows the jobs in all partitions.
  • squeue -u user - print a list of running/waiting jobs of a given user
  • squeue -j<JOB_ID> - this shows detailed info about the job with given JOB_ID (if it is still running).
  • sinfo - print available/total resources

Job interaction

  • scontrol show job JOBID - this shows details of running job with JOBID
  • scancel JOBID - delete job from the queue

Selected submit options

The complete list of available options for the commands srun and sbatch can be found in SLURM documentation. Most of the options listed here can be entered as a command parameters or as an SBATCH directive inside of a script. 

 -J helloWorld         # name of job
 --chdir /job/path/    # path where the job will be executed
 -p gpu                # name of partition or queue (if not specified default partition is used)
 -q normal             # QOS level (sets priority of the job)
 -c 4                  # reserve 4 CPU threads
 --gres=gpu:1          # reserve 1 GPU card
 -o script.out         # name of output file for the job 
 -e script.err         # name of error file for the job

Array jobs

If you need to submit rather large number of jobs which are similar (i.e. processing a large number of input files) you should consider launching an array job.

For example, one might need to process 1000 files named file_N.txt (where N is a number between 1-1000). A program that can process one file is called crunchFile and it takes only one argument - the name of the file to process. Instead of calling 1000x: 

  sbatch crunchFile file_N.txt

we can write a wrapper script crunchScript.sh referring to the SLURM variable SLURM_ARRAY_TASK_ID: 

  #!/bin/bash
  #SBATCH -p CPU
  #SBATCH --mem 2G
  
  crunchFile name_${SLURM_ARRAY_TASK_ID}.txt

and submit all the jobs at once as an array job: 

 sbatch --array=1-1000%20 crunchScript.sh

Where the option --array 1-1000%20 means that we want SLURM to:

  • launch 1000 instances of crunchScript.sh
  • each instance will be launched with SLURM_ARRAY_TASK_ID set to a number in the specified range
  • there will be at most 20 parallel tasks running at once. This is useful for a larger number of tasks - this way we ensure that we do not flood the cluster with requests.

You can read more about array jobs from the SLURM documentation.

Retrieved from "https://aic.ufal.mff.cuni.cz/index.php?title=Submitting_CPU_Jobs&oldid=120"