UFAL AIC - User contributions [en]

Submitting GPU Jobs

2025-01-24T11:04:47Z

Admin: /* List of installed GPUs */

Start by reading [[Submitting CPU Jobs]] page.

The GPU jobs are submitted to <code>gpu</code> partition.

To ask for one GPU card, use <code>#SBATCH -G 1</code> directive or <code>-G 1</code> option on the command line. The submitted job has <code>CUDA_VISIBLE_DEVICES</code> set appropriately, so all CUDA applications should use only the allocated GPUs.

== Rules ==

* Always use GPUs via ''sbatch'' (or ''srun''), never via ''ssh''. You can ssh to any machine e.g. to run ''nvidia-smi'' or ''htop'', but not to start computing on GPU.
* Don't forget to specify you RAM requirements with e.g. ''--mem=10G''.
* Always specify the number of GPU cards (e.g. ''-G 1''). Thus e.g. <code>srun -p gpu --mem=64G -G 2 --pty bash</code>
* For interactive jobs, you can use ''srun'', but make sure to end your job as soon as you don't need the GPU (so don't use srun for long training).
* In general: don't reserve a GPU (as described above) without actually using it for longer time, e.g., try separating steps which need GPU and steps which do not and execute those separately on our GPU resp. CPU cluster.
* If you know an approximate runtime of your job, please specify it with ''-t <time>''. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds".

== CUDA and cuDNN ==

Available CUDA versions are in
/lnet/aic/opt/cuda/
and as of Apr 2023, available versions as 10.1, 10.2, 11.2, 11.7, 11.8.

The cuDNN library is also available in the subdirectory <code>cudnn/VERSION/lib64</code> of the respective CUDA directories.

Therefore, to use CUDA 11.2 with cuDNN 8.1.1, you should add the following to your <code>.profile</code>:
export PATH="/lnet/aic/opt/cuda/cuda-11.2/bin:$PATH"
export LD_LIBRARY_PATH="/lnet/aic/opt/cuda/cuda-11.2/lib64:/lnet/aic/opt/cuda/cuda-11.2/cudnn/8.1.1/lib64:/lnet/aic/opt/cuda/cuda-11.2/extras/CUPTI/lib64:$LD_LIBRARY_PATH"
export XLA_FLAGS=--xla_gpu_cuda_data_dir=/lnet/aic/opt/cuda/cuda-11.2 # XLA configuration if you are using TensorFlow

=== CUDA modules ===
CUDA 11.2 and later can be also loaded as modules. This will set various environment variables for you so you should be able to use CUDA easily.

On a GPU node, you can do the following:
# list available modules with: <code>module avail</code>
# load the version you need (possibly specifying the version of CuDNN): <code>module load <modulename></code>
# you can unload the module with: <code>module unload <modulename></code>

As of Apr 2023, the available modules are
cuda/11.2
cuda/11.2-cudnn8.1
cuda/11.7
cuda/11.7-cudnn8.5
cuda/11.8
cuda/11.8-cudnn8.5
cuda/11.8-cudnn8.6
cuda/11.8-cudnn8.9

=== List of installed GPUs ===

===== GPU types and memory size =====
* 2080 - 11G GPU RAM
* A4000 - 16G GPU RAM
* 3090 - 24G GPU RAM

root@gpu-node1:~# nvidia-smi -L
GPU 0: NVIDIA RTX A4000 (UUID: GPU-5b111b2e-ff0d-25f7-2e08-f4065c510832)
GPU 1: NVIDIA RTX A4000 (UUID: GPU-9e4fa6ca-e3fa-d404-eac2-026295fbd076)
GPU 2: NVIDIA RTX A4000 (UUID: GPU-189c4e93-0ebe-2c7b-aa61-270d08db5a9c)
GPU 3: NVIDIA RTX A4000 (UUID: GPU-2f06bc8b-0ef4-6bd9-4385-69c76d73daae)
GPU 4: NVIDIA RTX A4000 (UUID: GPU-818b6a31-6d23-39a5-2139-c2c6c8a1174e)
GPU 5: NVIDIA GeForce RTX 3090 (UUID: GPU-ba293e60-32f9-6907-705b-e053d1bf453b)
GPU 6: NVIDIA RTX A4000 (UUID: GPU-edbbd8a2-f618-070b-8fce-b9a5fa10ccb2)
GPU 7: NVIDIA RTX A4000 (UUID: GPU-82956c50-ec17-6fb7-7898-d3c920c1b1f7)
GPU 8: NVIDIA RTX A4000 (UUID: GPU-ae27887e-5198-ebc6-fa9c-1f4b66d91b46)

root@gpu-node2:~# nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-15b17780-d818-bcd2-566c-564aa1dfc38e)
GPU 1: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-e184b0d4-7147-af43-041b-caa7f597363a)
GPU 2: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-ac1a453e-1c30-3fe0-e246-dd07c7645066)
GPU 3: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-4d19d859-d044-fdc8-17e0-e84fef4a8a13)
GPU 4: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-8035e3f3-76c9-124f-c5ea-d1dd4369f2a8)
GPU 5: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-670d0788-a048-8eef-ad1b-1eb77b18980b)
GPU 6: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-18d030c6-5956-f45f-7d15-ab53cffa813e)
GPU 7: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-f7940219-84a7-8c9c-386f-14e4043c9884)

Submitting GPU Jobs

2025-01-24T11:04:11Z

Admin: /* List of installed GPUs */

Start by reading [[Submitting CPU Jobs]] page.

The GPU jobs are submitted to <code>gpu</code> partition.

To ask for one GPU card, use <code>#SBATCH -G 1</code> directive or <code>-G 1</code> option on the command line. The submitted job has <code>CUDA_VISIBLE_DEVICES</code> set appropriately, so all CUDA applications should use only the allocated GPUs.

== Rules ==

* Always use GPUs via ''sbatch'' (or ''srun''), never via ''ssh''. You can ssh to any machine e.g. to run ''nvidia-smi'' or ''htop'', but not to start computing on GPU.
* Don't forget to specify you RAM requirements with e.g. ''--mem=10G''.
* Always specify the number of GPU cards (e.g. ''-G 1''). Thus e.g. <code>srun -p gpu --mem=64G -G 2 --pty bash</code>
* For interactive jobs, you can use ''srun'', but make sure to end your job as soon as you don't need the GPU (so don't use srun for long training).
* In general: don't reserve a GPU (as described above) without actually using it for longer time, e.g., try separating steps which need GPU and steps which do not and execute those separately on our GPU resp. CPU cluster.
* If you know an approximate runtime of your job, please specify it with ''-t <time>''. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds".

== CUDA and cuDNN ==

Available CUDA versions are in
/lnet/aic/opt/cuda/
and as of Apr 2023, available versions as 10.1, 10.2, 11.2, 11.7, 11.8.

The cuDNN library is also available in the subdirectory <code>cudnn/VERSION/lib64</code> of the respective CUDA directories.

Therefore, to use CUDA 11.2 with cuDNN 8.1.1, you should add the following to your <code>.profile</code>:
export PATH="/lnet/aic/opt/cuda/cuda-11.2/bin:$PATH"
export LD_LIBRARY_PATH="/lnet/aic/opt/cuda/cuda-11.2/lib64:/lnet/aic/opt/cuda/cuda-11.2/cudnn/8.1.1/lib64:/lnet/aic/opt/cuda/cuda-11.2/extras/CUPTI/lib64:$LD_LIBRARY_PATH"
export XLA_FLAGS=--xla_gpu_cuda_data_dir=/lnet/aic/opt/cuda/cuda-11.2 # XLA configuration if you are using TensorFlow

=== CUDA modules ===
CUDA 11.2 and later can be also loaded as modules. This will set various environment variables for you so you should be able to use CUDA easily.

On a GPU node, you can do the following:
# list available modules with: <code>module avail</code>
# load the version you need (possibly specifying the version of CuDNN): <code>module load <modulename></code>
# you can unload the module with: <code>module unload <modulename></code>

As of Apr 2023, the available modules are
cuda/11.2
cuda/11.2-cudnn8.1
cuda/11.7
cuda/11.7-cudnn8.5
cuda/11.8
cuda/11.8-cudnn8.5
cuda/11.8-cudnn8.6
cuda/11.8-cudnn8.9

=== List of installed GPUs ===

==== GPU types and memory size ====
* 2080 - 11G GPU RAM
* A4000 - 16G GPU RAM
* 3090 - 24G GPU RAM

root@gpu-node1:~# nvidia-smi -L
GPU 0: NVIDIA RTX A4000 (UUID: GPU-5b111b2e-ff0d-25f7-2e08-f4065c510832)
GPU 1: NVIDIA RTX A4000 (UUID: GPU-9e4fa6ca-e3fa-d404-eac2-026295fbd076)
GPU 2: NVIDIA RTX A4000 (UUID: GPU-189c4e93-0ebe-2c7b-aa61-270d08db5a9c)
GPU 3: NVIDIA RTX A4000 (UUID: GPU-2f06bc8b-0ef4-6bd9-4385-69c76d73daae)
GPU 4: NVIDIA RTX A4000 (UUID: GPU-818b6a31-6d23-39a5-2139-c2c6c8a1174e)
GPU 5: NVIDIA GeForce RTX 3090 (UUID: GPU-ba293e60-32f9-6907-705b-e053d1bf453b)
GPU 6: NVIDIA RTX A4000 (UUID: GPU-edbbd8a2-f618-070b-8fce-b9a5fa10ccb2)
GPU 7: NVIDIA RTX A4000 (UUID: GPU-82956c50-ec17-6fb7-7898-d3c920c1b1f7)
GPU 8: NVIDIA RTX A4000 (UUID: GPU-ae27887e-5198-ebc6-fa9c-1f4b66d91b46)

root@gpu-node2:~# nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-15b17780-d818-bcd2-566c-564aa1dfc38e)
GPU 1: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-e184b0d4-7147-af43-041b-caa7f597363a)
GPU 2: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-ac1a453e-1c30-3fe0-e246-dd07c7645066)
GPU 3: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-4d19d859-d044-fdc8-17e0-e84fef4a8a13)
GPU 4: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-8035e3f3-76c9-124f-c5ea-d1dd4369f2a8)
GPU 5: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-670d0788-a048-8eef-ad1b-1eb77b18980b)
GPU 6: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-18d030c6-5956-f45f-7d15-ab53cffa813e)
GPU 7: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-f7940219-84a7-8c9c-386f-14e4043c9884)

Submitting GPU Jobs

2025-01-24T10:59:27Z

Admin: /* List of installed GPUs */

Start by reading [[Submitting CPU Jobs]] page.

The GPU jobs are submitted to <code>gpu</code> partition.

To ask for one GPU card, use <code>#SBATCH -G 1</code> directive or <code>-G 1</code> option on the command line. The submitted job has <code>CUDA_VISIBLE_DEVICES</code> set appropriately, so all CUDA applications should use only the allocated GPUs.

== Rules ==

* Always use GPUs via ''sbatch'' (or ''srun''), never via ''ssh''. You can ssh to any machine e.g. to run ''nvidia-smi'' or ''htop'', but not to start computing on GPU.
* Don't forget to specify you RAM requirements with e.g. ''--mem=10G''.
* Always specify the number of GPU cards (e.g. ''-G 1''). Thus e.g. <code>srun -p gpu --mem=64G -G 2 --pty bash</code>
* For interactive jobs, you can use ''srun'', but make sure to end your job as soon as you don't need the GPU (so don't use srun for long training).
* In general: don't reserve a GPU (as described above) without actually using it for longer time, e.g., try separating steps which need GPU and steps which do not and execute those separately on our GPU resp. CPU cluster.
* If you know an approximate runtime of your job, please specify it with ''-t <time>''. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds".

== CUDA and cuDNN ==

Available CUDA versions are in
/lnet/aic/opt/cuda/
and as of Apr 2023, available versions as 10.1, 10.2, 11.2, 11.7, 11.8.

The cuDNN library is also available in the subdirectory <code>cudnn/VERSION/lib64</code> of the respective CUDA directories.

Therefore, to use CUDA 11.2 with cuDNN 8.1.1, you should add the following to your <code>.profile</code>:
export PATH="/lnet/aic/opt/cuda/cuda-11.2/bin:$PATH"
export LD_LIBRARY_PATH="/lnet/aic/opt/cuda/cuda-11.2/lib64:/lnet/aic/opt/cuda/cuda-11.2/cudnn/8.1.1/lib64:/lnet/aic/opt/cuda/cuda-11.2/extras/CUPTI/lib64:$LD_LIBRARY_PATH"
export XLA_FLAGS=--xla_gpu_cuda_data_dir=/lnet/aic/opt/cuda/cuda-11.2 # XLA configuration if you are using TensorFlow

=== CUDA modules ===
CUDA 11.2 and later can be also loaded as modules. This will set various environment variables for you so you should be able to use CUDA easily.

On a GPU node, you can do the following:
# list available modules with: <code>module avail</code>
# load the version you need (possibly specifying the version of CuDNN): <code>module load <modulename></code>
# you can unload the module with: <code>module unload <modulename></code>

As of Apr 2023, the available modules are
cuda/11.2
cuda/11.2-cudnn8.1
cuda/11.7
cuda/11.7-cudnn8.5
cuda/11.8
cuda/11.8-cudnn8.5
cuda/11.8-cudnn8.6
cuda/11.8-cudnn8.9

=== List of installed GPUs ===
root@gpu-node1:~# nvidia-smi -L
GPU 0: NVIDIA RTX A4000 (UUID: GPU-5b111b2e-ff0d-25f7-2e08-f4065c510832)
GPU 1: NVIDIA RTX A4000 (UUID: GPU-9e4fa6ca-e3fa-d404-eac2-026295fbd076)
GPU 2: NVIDIA RTX A4000 (UUID: GPU-189c4e93-0ebe-2c7b-aa61-270d08db5a9c)
GPU 3: NVIDIA RTX A4000 (UUID: GPU-2f06bc8b-0ef4-6bd9-4385-69c76d73daae)
GPU 4: NVIDIA RTX A4000 (UUID: GPU-818b6a31-6d23-39a5-2139-c2c6c8a1174e)
GPU 5: NVIDIA GeForce RTX 3090 (UUID: GPU-ba293e60-32f9-6907-705b-e053d1bf453b)
GPU 6: NVIDIA RTX A4000 (UUID: GPU-edbbd8a2-f618-070b-8fce-b9a5fa10ccb2)
GPU 7: NVIDIA RTX A4000 (UUID: GPU-82956c50-ec17-6fb7-7898-d3c920c1b1f7)
GPU 8: NVIDIA RTX A4000 (UUID: GPU-ae27887e-5198-ebc6-fa9c-1f4b66d91b46)

root@gpu-node2:~# nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-15b17780-d818-bcd2-566c-564aa1dfc38e)
GPU 1: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-e184b0d4-7147-af43-041b-caa7f597363a)
GPU 2: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-ac1a453e-1c30-3fe0-e246-dd07c7645066)
GPU 3: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-4d19d859-d044-fdc8-17e0-e84fef4a8a13)
GPU 4: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-8035e3f3-76c9-124f-c5ea-d1dd4369f2a8)
GPU 5: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-670d0788-a048-8eef-ad1b-1eb77b18980b)
GPU 6: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-18d030c6-5956-f45f-7d15-ab53cffa813e)
GPU 7: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-f7940219-84a7-8c9c-386f-14e4043c9884)

Submitting CPU Jobs

2024-04-23T14:49:45Z

Admin: /* Selected submit options */

The CPU jobs should be submitted to <code>cpu</code> partition.

You can submit a non-interactive job using the '''sbatch''' command.
To submit an interactive job, use the '''srun''' command:

srun --pty bash

== Resource specification ==

You should specify the memory and CPU requirements (if higher than the defaults) and don't exceed them.
If your job needs more than one CPU (thread) (on a single machine) for most of the time, reserve the given number of CPU threads with the <code>--cpus-per-task</code> and memory with the <code>--mem</code> options.

srun -p cpu --cpus-per-task=4 --mem=8G --pty bash

This will give you an interactive shell with 4 threads and 8G RAM on the ''cpu'' partition.

== Monitoring and interaction ==

=== Job monitoring ===
We should be able to see what is going on when we run a job. Following examples shows usage of some typical commands:
* <code>squeue -a</code> - this shows the jobs in all partitions.
* <code>squeue -u user</code> - print a list of running/waiting jobs of a given user
* <code>squeue -j<JOB_ID></code> - this shows detailed info about the job with given JOB_ID (if it is still running).
* <code>sinfo</code> - print available/total resources

=== Job interaction ===
* <code>scontrol show job JOBID</code> - this shows details of running job with JOBID
* <code>scancel JOBID</code> - delete job from the queue

=== Selected submit options ===
The complete list of available options for the commands <code>srun</code> and <code>sbatch</code> can be found in [https://slurm.schedmd.com/man_index.html SLURM documentation]. Most of the options listed here can be entered as a command parameters or as an SBATCH directive inside of a script.

-J helloWorld # name of job
--chdir /job/path/ # path where the job will be executed
-p gpu # name of partition or queue (if not specified default partition is used)
-q normal # QOS level (sets priority of the job)
-c 4 # reserve 4 CPU threads
--gres=gpu:1 # reserve 1 GPU card
-o script.out # name of output file for the job
-e script.err # name of error file for the job

== Array jobs ==
If you need to submit rather large number of jobs which are similar (i.e. processing a large number of input files) you should consider launching an ''array job''.

For example, one might need to process 1000 files named <code>file_N.txt</code> (where N is a number between 1-1000).
A program that can process one file is called <code>crunchFile</code> and it takes only one argument - the name of the file to process. Instead of calling 1000x:
sbatch crunchFile file_N.txt

we can write a wrapper script <code>crunchScript.sh</code> referring to the SLURM variable <code>SLURM_ARRAY_TASK_ID</code>:

#!/bin/bash
#SBATCH -p CPU
#SBATCH --mem 2G

crunchFile name_${SLURM_ARRAY_TASK_ID}.txt

and submit all the jobs at once as an ''array job'':

sbatch --array=1-1000%20 crunchScript.sh

Where the option <code>--array 1-1000%20</code> means that we want SLURM to:
* launch 1000 instances of <code>crunchScript.sh</code>
* each instance will be launched with <code>SLURM_ARRAY_TASK_ID</code> set to a number in the specified range
* there will be at most 20 parallel tasks running at once. This is useful for a larger number of tasks - this way we ensure that we do not flood the cluster with requests.

You can read more about ''array jobs'' from the [https://slurm.schedmd.com/job_array.html SLURM documentation].

Submitting CPU Jobs

2024-04-23T14:49:04Z

Admin: /* Selected submit options */

The CPU jobs should be submitted to <code>cpu</code> partition.

You can submit a non-interactive job using the '''sbatch''' command.
To submit an interactive job, use the '''srun''' command:

srun --pty bash

== Resource specification ==

You should specify the memory and CPU requirements (if higher than the defaults) and don't exceed them.
If your job needs more than one CPU (thread) (on a single machine) for most of the time, reserve the given number of CPU threads with the <code>--cpus-per-task</code> and memory with the <code>--mem</code> options.

srun -p cpu --cpus-per-task=4 --mem=8G --pty bash

This will give you an interactive shell with 4 threads and 8G RAM on the ''cpu'' partition.

== Monitoring and interaction ==

=== Job monitoring ===
We should be able to see what is going on when we run a job. Following examples shows usage of some typical commands:
* <code>squeue -a</code> - this shows the jobs in all partitions.
* <code>squeue -u user</code> - print a list of running/waiting jobs of a given user
* <code>squeue -j<JOB_ID></code> - this shows detailed info about the job with given JOB_ID (if it is still running).
* <code>sinfo</code> - print available/total resources

=== Job interaction ===
* <code>scontrol show job JOBID</code> - this shows details of running job with JOBID
* <code>scancel JOBID</code> - delete job from the queue

=== Selected submit options ===
The complete list of available options for the commands <code>srun</code> and <code>sbatch</code> can be found in [https://slurm.schedmd.com/man_index.html SLURM documentation]. Most of the options listed here can be entered as a command parameters or as an SBATCH directive inside of a script.

-J helloWorld # name of job
--chdir /path/where/the/job/will/be/executed
-p gpu # name of partition or queue (if not specified default partition is used)
-q normal # QOS level (sets priority of the job)
-c 4 # reserve 4 CPU threads
--gres=gpu:1 # reserve 1 GPU card
-o script.out # name of output file for the job
-e script.err # name of error file for the job

== Array jobs ==
If you need to submit rather large number of jobs which are similar (i.e. processing a large number of input files) you should consider launching an ''array job''.

For example, one might need to process 1000 files named <code>file_N.txt</code> (where N is a number between 1-1000).
A program that can process one file is called <code>crunchFile</code> and it takes only one argument - the name of the file to process. Instead of calling 1000x:
sbatch crunchFile file_N.txt

we can write a wrapper script <code>crunchScript.sh</code> referring to the SLURM variable <code>SLURM_ARRAY_TASK_ID</code>:

#!/bin/bash
#SBATCH -p CPU
#SBATCH --mem 2G

crunchFile name_${SLURM_ARRAY_TASK_ID}.txt

and submit all the jobs at once as an ''array job'':

sbatch --array=1-1000%20 crunchScript.sh

Where the option <code>--array 1-1000%20</code> means that we want SLURM to:
* launch 1000 instances of <code>crunchScript.sh</code>
* each instance will be launched with <code>SLURM_ARRAY_TASK_ID</code> set to a number in the specified range
* there will be at most 20 parallel tasks running at once. This is useful for a larger number of tasks - this way we ensure that we do not flood the cluster with requests.

You can read more about ''array jobs'' from the [https://slurm.schedmd.com/job_array.html SLURM documentation].

Submitting CPU Jobs

2024-04-23T14:22:48Z

Admin:

The CPU jobs should be submitted to <code>cpu</code> partition.

You can submit a non-interactive job using the '''sbatch''' command.
To submit an interactive job, use the '''srun''' command:

srun --pty bash

== Resource specification ==

You should specify the memory and CPU requirements (if higher than the defaults) and don't exceed them.
If your job needs more than one CPU (thread) (on a single machine) for most of the time, reserve the given number of CPU threads with the <code>--cpus-per-task</code> and memory with the <code>--mem</code> options.

srun -p cpu --cpus-per-task=4 --mem=8G --pty bash

This will give you an interactive shell with 4 threads and 8G RAM on the ''cpu'' partition.

== Monitoring and interaction ==

=== Job monitoring ===
We should be able to see what is going on when we run a job. Following examples shows usage of some typical commands:
* <code>squeue -a</code> - this shows the jobs in all partitions.
* <code>squeue -u user</code> - print a list of running/waiting jobs of a given user
* <code>squeue -j<JOB_ID></code> - this shows detailed info about the job with given JOB_ID (if it is still running).
* <code>sinfo</code> - print available/total resources

=== Job interaction ===
* <code>scontrol show job JOBID</code> - this shows details of running job with JOBID
* <code>scancel JOBID</code> - delete job from the queue

=== Selected submit options ===
The complete list of available options for the commands <code>srun</code> and <code>sbatch</code> can be found in [https://slurm.schedmd.com/man_index.html SLURM documentation]. Most of the options listed here can be entered as a command parameters or as an SBATCH directive inside of a script.

-J helloWorld # name of job
-p gpu # name of partition or queue (if not specified default partition is used)
-q normal # QOS level (sets priority of the job)
-c 4 # reserve 4 CPU threads
--gres=gpu:1 # reserve 1 GPU card
-o script.out # name of output file for the job
-e script.err # name of error file for the job

== Array jobs ==
If you need to submit rather large number of jobs which are similar (i.e. processing a large number of input files) you should consider launching an ''array job''.

For example, one might need to process 1000 files named <code>file_N.txt</code> (where N is a number between 1-1000).
A program that can process one file is called <code>crunchFile</code> and it takes only one argument - the name of the file to process. Instead of calling 1000x:
sbatch crunchFile file_N.txt

we can write a wrapper script <code>crunchScript.sh</code> referring to the SLURM variable <code>SLURM_ARRAY_TASK_ID</code>:

#!/bin/bash
#SBATCH -p CPU
#SBATCH --mem 2G

crunchFile name_${SLURM_ARRAY_TASK_ID}.txt

and submit all the jobs at once as an ''array job'':

sbatch --array=1-1000%20 crunchScript.sh

Where the option <code>--array 1-1000%20</code> means that we want SLURM to:
* launch 1000 instances of <code>crunchScript.sh</code>
* each instance will be launched with <code>SLURM_ARRAY_TASK_ID</code> set to a number in the specified range
* there will be at most 20 parallel tasks running at once. This is useful for a larger number of tasks - this way we ensure that we do not flood the cluster with requests.

You can read more about ''array jobs'' from the [https://slurm.schedmd.com/job_array.html SLURM documentation].

Environment

2024-04-09T13:34:19Z

Admin:

=== Bash ===

There are several useful environment variables provided by SLURM when submitting jobs:
SLURM_JOB_ID - Job ID.
SLURM_JOB_NAME - Job name.
CUDA_VISIBLE_DEVICES - Specifies the GPU devices for the job allocation.
TMPDIR - Local temporary directory reserved for the job. This should be used instead of /tmp.

Refer to [https://slurm.schedmd.com/prolog_epilog.html SLURM docs] for complete list.

=== Python ===

If you need to use a different version of python than the default one installed with node OS you can use pre-installed versions found at:

/opt/python

If you need some specific version not found in this directory please contact the [mailto:it@ufal.mff.cuni.cz AIC administrators].

=== Virtual environment ===
The recommended way is to use python through a virtual environment.
You need to decide which PYTHON_VERSION you want to use and where do you want to store your virtual environment (VENV_PATH). Then you can create it:

/opt/python/<PYTHON_VERSION>/bin/python3 -m venv <VENV_PATH>

You need to activate the environment before you actually use it:

source <VENV_PATH>/bin/activate

Then you should be able to use the python version of your choice.
More about virtual environments can be found [https://docs.python.org/3/library/venv.html here]

Main Page

2024-03-19T15:06:41Z

Admin:

<div style='text-align: center;'>CZ.02.2.69/0.0/0.0/17_044/0008562</div>
<div style='text-align: center;'>Podpora rozvoje studijního prostředí na Univerzitě Karlově - VRR</div>
[[File:OP_VVV_logo.jpg|frameless|center|upright=2.5]]

== Welcome to AIC ==

AIC (Artificial Intelligence Cluster) is a computational grid with sufficient computational capacity for research in the field of [https://en.wikipedia.org/wiki/Deep_learning deep learning] using both CPU and GPU. It was built on top of [https://slurm.schedmd.com/ SLURM] scheduling system. MFF students of Bc. and Mgr. degrees can use it to run their experiments and learn the proper ways of grid computing in the process.

=== Access ===
AIC is dedicated to UFAL students who will get an account if requested by authorized lector.

To change the password, this link is available: https://aic.ufal.mff.cuni.cz/pw-manager

There is a restriction on resources allocated by one user in group '''students''' at a given time.
By default, this is set to a maximum of 4 CPU and 1 GPU.

=== Jupyterlab ===
AIC provides also Jupyterlab portal on top of your AIC account and HOME directory. It can be found at https://aic.ufal.mff.cuni.cz/jlab . Pre-installed extensions: R, ipython, Rstudio (community), Slurm Queue Manager.

=== Connecting to the Cluster (directly) ===
Use SSH to connect to the cluster:
ssh LOGIN@aic.ufal.mff.cuni.cz

=== Basic HOWTO ===

Following HOWTO is meant to provide only a simplified overview of the cluster usage. It is strongly recommended to read some further documentation ([[Submitting_CPU_Jobs|CPU]] or [[Submitting_GPU_Jobs|GPU]]) before running some serious experiments.
More serious experiments tend to take more resources. In order to avoid unexpected failures please make sure your [[Quotas|quota]] is not exceeded.

'''Rule 0: NEVER RUN JOBS DIRECTLY ON aic.ufal.mff.cuni.cz HEADNODE. Use <code>srun</code> to get computational node shell!'''

Suppose we want to run some computations described by a script called <code>job_script.sh</code>:

#!/bin/bash
#SBATCH -J helloWorld # name of job
#SBATCH -p cpu # name of partition or queue (default is cpu)
#SBATCH -o helloWorld.out # name of output file for this submission script
#SBATCH -e helloWorld.err # name of error file for this submission script
# run my job (some executable)
sleep 5
echo "Hello I am running on cluster!"

We need to ''submit'' the job to the cluster which is done by logging on the submit host <code>aic.ufal.mff.cuni.cz</code> and issuing the command:<br>
<code>sbatch job_script.sh</code>

This will enqueue our ''job'' to the default ''partition'' (or ''queue'') which is <code>cpu</code>. The scheduler decides which particular machine in the specified queue has ''resources'' needed to run the job. Typically we will see a message which tells us the ID of our job (3 in this example):

Submitted batch job 3

The options used in this example are specified inside the script using the ''#SBATCH'' directive. Any option can be specified either in the script or as a command line parameter (see ''man sbatch'' for details).

We can specify custom arguments '''before''' the name of the script:

sbatch --export=ARG1='firstArg',ARG2='secondArg' job_script.sh

These can be accessed in the job script as <code>$ARG1</code> and <code>$ARG2</code>.

Main Page

2024-03-19T15:06:18Z

Admin:

<div style='text-align: center;'>CZ.02.2.69/0.0/0.0/17_044/0008562</div>
<div style='text-align: center;'>Podpora rozvoje studijního prostředí na Univerzitě Karlově - VRR</div>
[[File:OP_VVV_logo.jpg|frameless|center|upright=2.5]]

== Welcome to AIC ==

AIC (Artificial Intelligence Cluster) is a computational grid with sufficient computational capacity for research in the field of [https://en.wikipedia.org/wiki/Deep_learning deep learning] using both CPU and GPU. It was built on top of [https://slurm.schedmd.com/ SLURM] scheduling system. MFF students of Bc. and Mgr. degrees can use it to run their experiments and learn the proper ways of grid computing in the process.

=== Access ===
AIC is dedicated to UFAL students who will get an account if requested by authorized lector.
To change the password, this link is available: https://aic.ufal.mff.cuni.cz/pw-manager

There is a restriction on resources allocated by one user in group '''students''' at a given time.
By default, this is set to a maximum of 4 CPU and 1 GPU.

=== Jupyterlab ===
AIC provides also Jupyterlab portal on top of your AIC account and HOME directory. It can be found at https://aic.ufal.mff.cuni.cz/jlab . Pre-installed extensions: R, ipython, Rstudio (community), Slurm Queue Manager.

=== Connecting to the Cluster (directly) ===
Use SSH to connect to the cluster:
ssh LOGIN@aic.ufal.mff.cuni.cz

=== Basic HOWTO ===

Following HOWTO is meant to provide only a simplified overview of the cluster usage. It is strongly recommended to read some further documentation ([[Submitting_CPU_Jobs|CPU]] or [[Submitting_GPU_Jobs|GPU]]) before running some serious experiments.
More serious experiments tend to take more resources. In order to avoid unexpected failures please make sure your [[Quotas|quota]] is not exceeded.

'''Rule 0: NEVER RUN JOBS DIRECTLY ON aic.ufal.mff.cuni.cz HEADNODE. Use <code>srun</code> to get computational node shell!'''

Suppose we want to run some computations described by a script called <code>job_script.sh</code>:

#!/bin/bash
#SBATCH -J helloWorld # name of job
#SBATCH -p cpu # name of partition or queue (default is cpu)
#SBATCH -o helloWorld.out # name of output file for this submission script
#SBATCH -e helloWorld.err # name of error file for this submission script
# run my job (some executable)
sleep 5
echo "Hello I am running on cluster!"

We need to ''submit'' the job to the cluster which is done by logging on the submit host <code>aic.ufal.mff.cuni.cz</code> and issuing the command:<br>
<code>sbatch job_script.sh</code>

This will enqueue our ''job'' to the default ''partition'' (or ''queue'') which is <code>cpu</code>. The scheduler decides which particular machine in the specified queue has ''resources'' needed to run the job. Typically we will see a message which tells us the ID of our job (3 in this example):

Submitted batch job 3

The options used in this example are specified inside the script using the ''#SBATCH'' directive. Any option can be specified either in the script or as a command line parameter (see ''man sbatch'' for details).

We can specify custom arguments '''before''' the name of the script:

sbatch --export=ARG1='firstArg',ARG2='secondArg' job_script.sh

These can be accessed in the job script as <code>$ARG1</code> and <code>$ARG2</code>.

Main Page

2024-03-19T15:01:43Z

Admin:

<div style='text-align: center;'>CZ.02.2.69/0.0/0.0/17_044/0008562</div>
<div style='text-align: center;'>Podpora rozvoje studijního prostředí na Univerzitě Karlově - VRR</div>
[[File:OP_VVV_logo.jpg|frameless|center|upright=2.5]]

== Welcome to AIC ==

AIC (Artificial Intelligence Cluster) is a computational grid with sufficient computational capacity for research in the field of [https://en.wikipedia.org/wiki/Deep_learning deep learning] using both CPU and GPU. It was built on top of [https://slurm.schedmd.com/ SLURM] scheduling system. MFF students of Bc. and Mgr. degrees can use it to run their experiments and learn the proper ways of grid computing in the process.

=== Access ===
AIC is dedicated to UFAL students who will get an account if requested by authorized lector.
For changing AIC password, this link is avialible: https://aic.ufal.mff.cuni.cz/pw-manager

There is a restriction on resources allocated by one user in group '''students''' at a given time.
By default, this is set to a maximum of 4 CPU and 1 GPU.

=== Jupyterlab ===
AIC provides also Jupyterlab portal on top of your AIC account and HOME directory. It can be found at https://aic.ufal.mff.cuni.cz/jlab . Pre-installed extensions: R, ipython, Rstudio (community), Slurm Queue Manager.

=== Connecting to the Cluster (directly) ===
Use SSH to connect to the cluster:
ssh LOGIN@aic.ufal.mff.cuni.cz

=== Basic HOWTO ===

Following HOWTO is meant to provide only a simplified overview of the cluster usage. It is strongly recommended to read some further documentation ([[Submitting_CPU_Jobs|CPU]] or [[Submitting_GPU_Jobs|GPU]]) before running some serious experiments.
More serious experiments tend to take more resources. In order to avoid unexpected failures please make sure your [[Quotas|quota]] is not exceeded.

'''Rule 0: NEVER RUN JOBS DIRECTLY ON aic.ufal.mff.cuni.cz HEADNODE. Use <code>srun</code> to get computational node shell!'''

Suppose we want to run some computations described by a script called <code>job_script.sh</code>:

#!/bin/bash
#SBATCH -J helloWorld # name of job
#SBATCH -p cpu # name of partition or queue (default is cpu)
#SBATCH -o helloWorld.out # name of output file for this submission script
#SBATCH -e helloWorld.err # name of error file for this submission script
# run my job (some executable)
sleep 5
echo "Hello I am running on cluster!"

We need to ''submit'' the job to the cluster which is done by logging on the submit host <code>aic.ufal.mff.cuni.cz</code> and issuing the command:<br>
<code>sbatch job_script.sh</code>

This will enqueue our ''job'' to the default ''partition'' (or ''queue'') which is <code>cpu</code>. The scheduler decides which particular machine in the specified queue has ''resources'' needed to run the job. Typically we will see a message which tells us the ID of our job (3 in this example):

Submitted batch job 3

The options used in this example are specified inside the script using the ''#SBATCH'' directive. Any option can be specified either in the script or as a command line parameter (see ''man sbatch'' for details).

We can specify custom arguments '''before''' the name of the script:

sbatch --export=ARG1='firstArg',ARG2='secondArg' job_script.sh

These can be accessed in the job script as <code>$ARG1</code> and <code>$ARG2</code>.

Submitting GPU Jobs

2023-12-07T10:37:40Z

Admin:

Start by reading [[Submitting CPU Jobs]] page.

The GPU jobs are submitted to <code>gpu</code> partition.

To ask for one GPU card, use <code>#SBATCH -G 1</code> directive or <code>-G 1</code> option on the command line. The submitted job has <code>CUDA_VISIBLE_DEVICES</code> set appropriately, so all CUDA applications should use only the allocated GPUs.

== Rules ==

* Always use GPUs via ''sbatch'' (or ''srun''), never via ''ssh''. You can ssh to any machine e.g. to run ''nvidia-smi'' or ''htop'', but not to start computing on GPU.
* Don't forget to specify you RAM requirements with e.g. ''--mem=10G''.
* Always specify the number of GPU cards (e.g. ''-G 1''). Thus e.g. <code>srun -p gpu --mem=64G -G 2 --pty bash</code>
* For interactive jobs, you can use ''srun'', but make sure to end your job as soon as you don't need the GPU (so don't use srun for long training).
* In general: don't reserve a GPU (as described above) without actually using it for longer time, e.g., try separating steps which need GPU and steps which do not and execute those separately on our GPU resp. CPU cluster.
* If you know an approximate runtime of your job, please specify it with ''-t <time>''. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds".

== CUDA and cuDNN ==

Available CUDA versions are in
/lnet/aic/opt/cuda/
and as of Apr 2023, available versions as 10.1, 10.2, 11.2, 11.7, 11.8.

The cuDNN library is also available in the subdirectory <code>cudnn/VERSION/lib64</code> of the respective CUDA directories.

Therefore, to use CUDA 11.2 with cuDNN 8.1.1, you should add the following to your <code>.profile</code>:
export PATH="/lnet/aic/opt/cuda/cuda-11.2/bin:$PATH"
export LD_LIBRARY_PATH="/lnet/aic/opt/cuda/cuda-11.2/lib64:/lnet/aic/opt/cuda/cuda-11.2/cudnn/8.1.1/lib64:/lnet/aic/opt/cuda/cuda-11.2/extras/CUPTI/lib64:$LD_LIBRARY_PATH"
export XLA_FLAGS=--xla_gpu_cuda_data_dir=/lnet/aic/opt/cuda/cuda-11.2 # XLA configuration if you are using TensorFlow

=== CUDA modules ===
CUDA 11.2 and later can be also loaded as modules. This will set various environment variables for you so you should be able to use CUDA easily.

On a GPU node, you can do the following:
# list available modules with: <code>module avail</code>
# load the version you need (possibly specifying the version of CuDNN): <code>module load <modulename></code>
# you can unload the module with: <code>module unload <modulename></code>

As of Apr 2023, the available modules are
cuda/11.2
cuda/11.2-cudnn8.1
cuda/11.7
cuda/11.7-cudnn8.5
cuda/11.8
cuda/11.8-cudnn8.5
cuda/11.8-cudnn8.6
cuda/11.8-cudnn8.9

=== List of installed GPUs ===
root@gpu-node1:~# nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 3090 (UUID: GPU-ba293e60-32f9-6907-705b-e053d1bf453b)
GPU 1: NVIDIA GeForce GTX 1080 Ti (UUID: GPU-b29fe79f-6192-5ece-6f91-e59d97ab304e)
GPU 2: NVIDIA GeForce GTX 1080 Ti (UUID: GPU-fbcba5d0-61bd-cc4c-810e-c80cbd9cd563)
GPU 3: NVIDIA GeForce GTX 1080 Ti (UUID: GPU-76ae3ae7-0a2d-ea68-3070-94c919f40169)
GPU 4: NVIDIA GeForce GTX 1080 Ti (UUID: GPU-18174817-13c3-b930-1d68-37c47b41dc0b)
GPU 5: NVIDIA GeForce GTX 1080 Ti (UUID: GPU-3af8a5c5-9e07-9468-e9dc-e1259f3e7890)
GPU 6: NVIDIA GeForce GTX 1080 Ti (UUID: GPU-51e376db-189b-b11d-bd27-bbbb6470ff26)

root@gpu-node2:~# nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-15b17780-d818-bcd2-566c-564aa1dfc38e)
GPU 1: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-e184b0d4-7147-af43-041b-caa7f597363a)
GPU 2: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-ac1a453e-1c30-3fe0-e246-dd07c7645066)
GPU 3: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-4d19d859-d044-fdc8-17e0-e84fef4a8a13)
GPU 4: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-8035e3f3-76c9-124f-c5ea-d1dd4369f2a8)
GPU 5: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-670d0788-a048-8eef-ad1b-1eb77b18980b)
GPU 6: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-18d030c6-5956-f45f-7d15-ab53cffa813e)
GPU 7: NVIDIA GeForce RTX 2080 Ti (UUID: GPU-f7940219-84a7-8c9c-386f-14e4043c9884)

Main Page

2023-10-25T10:37:03Z

Admin:

<div style='text-align: center;'>CZ.02.2.69/0.0/0.0/17_044/0008562</div>
<div style='text-align: center;'>Podpora rozvoje studijního prostředí na Univerzitě Karlově - VRR</div>
[[File:OP_VVV_logo.jpg|frameless|center|upright=2.5]]

== Welcome to AIC ==

AIC (Artificial Intelligence Cluster) is a computational grid with sufficient computational capacity for research in the field of [https://en.wikipedia.org/wiki/Deep_learning deep learning] using both CPU and GPU. It was built on top of [https://slurm.schedmd.com/ SLURM] scheduling system. MFF students of Bc. and Mgr. degrees can use it to run their experiments and learn the proper ways of grid computing in the process.

=== Access ===
AIC is dedicated to UFAL students who will get an account if requested by authorized lector.

There is a restriction on resources allocated by one user in group '''students''' at a given time.
By default, this is set to a maximum of 4 CPU and 1 GPU.

=== Jupyterlab ===
AIC provides also Jupyterlab portal on top of your AIC account and HOME directory. It can be found at https://aic.ufal.mff.cuni.cz/jlab . Pre-installed extensions: R, ipython, Rstudio (community), Slurm Queue Manager.

=== Connecting to the Cluster (directly) ===
Use SSH to connect to the cluster:
ssh LOGIN@aic.ufal.mff.cuni.cz

=== Basic HOWTO ===

Following HOWTO is meant to provide only a simplified overview of the cluster usage. It is strongly recommended to read some further documentation ([[Submitting_CPU_Jobs|CPU]] or [[Submitting_GPU_Jobs|GPU]]) before running some serious experiments.
More serious experiments tend to take more resources. In order to avoid unexpected failures please make sure your [[Quotas|quota]] is not exceeded.

'''Rule 0: NEVER RUN JOBS DIRECTLY ON aic.ufal.mff.cuni.cz HEADNODE. Use <code>srun</code> to get computational node shell!'''

Suppose we want to run some computations described by a script called <code>job_script.sh</code>:

#!/bin/bash
#SBATCH -J helloWorld # name of job
#SBATCH -p cpu # name of partition or queue (default is cpu)
#SBATCH -o helloWorld.out # name of output file for this submission script
#SBATCH -e helloWorld.err # name of error file for this submission script
# run my job (some executable)
sleep 5
echo "Hello I am running on cluster!"

We need to ''submit'' the job to the cluster which is done by logging on the submit host <code>aic.ufal.mff.cuni.cz</code> and issuing the command:<br>
<code>sbatch job_script.sh</code>

This will enqueue our ''job'' to the default ''partition'' (or ''queue'') which is <code>cpu</code>. The scheduler decides which particular machine in the specified queue has ''resources'' needed to run the job. Typically we will see a message which tells us the ID of our job (3 in this example):

Submitted batch job 3

The options used in this example are specified inside the script using the ''#SBATCH'' directive. Any option can be specified either in the script or as a command line parameter (see ''man sbatch'' for details).

We can specify custom arguments '''before''' the name of the script:

sbatch --export=ARG1='firstArg',ARG2='secondArg' job_script.sh

These can be accessed in the job script as <code>$ARG1</code> and <code>$ARG2</code>.

Main Page

2023-10-25T10:36:42Z

Admin:

<div style='text-align: center;'>CZ.02.2.69/0.0/0.0/17_044/0008562</div>
<div style='text-align: center;'>Podpora rozvoje studijního prostředí na Univerzitě Karlově - VRR</div>
[[File:OP_VVV_logo.jpg|frameless|center|upright=2.5]]

== Welcome to AIC ==

AIC (Artificial Intelligence Cluster) is a computational grid with sufficient computational capacity for research in the field of [https://en.wikipedia.org/wiki/Deep_learning deep learning] using both CPU and GPU. It was built on top of [https://slurm.schedmd.com/ SLURM] scheduling system. MFF students of Bc. and Mgr. degrees can use it to run their experiments and learn the proper ways of grid computing in the process.

=== Access ===
AIC is dedicated to UFAL students who will get an account if requested by authorized lector.

There is a restriction on resources allocated by one user in group '''students''' at a given time.
By default, this is set to a maximum of 4 CPU and 1 GPU.

=== Jupyterlab ===
AIC provides also Jupyterlab portal on top of your AIC account and HOME directory. It can be found at https://aic.ufal.mff.cuni.cz/jlab . Pre-installed extensions: R, ipython, Rstudio (community), Slurm Queue Manager.

=== Connecting to the Cluster. ===
Use SSH to connect to the cluster:
ssh LOGIN@aic.ufal.mff.cuni.cz

=== Basic HOWTO ===

Following HOWTO is meant to provide only a simplified overview of the cluster usage. It is strongly recommended to read some further documentation ([[Submitting_CPU_Jobs|CPU]] or [[Submitting_GPU_Jobs|GPU]]) before running some serious experiments.
More serious experiments tend to take more resources. In order to avoid unexpected failures please make sure your [[Quotas|quota]] is not exceeded.

'''Rule 0: NEVER RUN JOBS DIRECTLY ON aic.ufal.mff.cuni.cz HEADNODE. Use <code>srun</code> to get computational node shell!'''

Suppose we want to run some computations described by a script called <code>job_script.sh</code>:

#!/bin/bash
#SBATCH -J helloWorld # name of job
#SBATCH -p cpu # name of partition or queue (default is cpu)
#SBATCH -o helloWorld.out # name of output file for this submission script
#SBATCH -e helloWorld.err # name of error file for this submission script
# run my job (some executable)
sleep 5
echo "Hello I am running on cluster!"

We need to ''submit'' the job to the cluster which is done by logging on the submit host <code>aic.ufal.mff.cuni.cz</code> and issuing the command:<br>
<code>sbatch job_script.sh</code>

This will enqueue our ''job'' to the default ''partition'' (or ''queue'') which is <code>cpu</code>. The scheduler decides which particular machine in the specified queue has ''resources'' needed to run the job. Typically we will see a message which tells us the ID of our job (3 in this example):

Submitted batch job 3

The options used in this example are specified inside the script using the ''#SBATCH'' directive. Any option can be specified either in the script or as a command line parameter (see ''man sbatch'' for details).

We can specify custom arguments '''before''' the name of the script:

sbatch --export=ARG1='firstArg',ARG2='secondArg' job_script.sh

These can be accessed in the job script as <code>$ARG1</code> and <code>$ARG2</code>.

Main Page

2023-10-25T10:35:34Z

Admin:

<div style='text-align: center;'>CZ.02.2.69/0.0/0.0/17_044/0008562</div>
<div style='text-align: center;'>Podpora rozvoje studijního prostředí na Univerzitě Karlově - VRR</div>
[[File:OP_VVV_logo.jpg|frameless|center|upright=2.5]]

== Welcome to AIC ==

AIC (Artificial Intelligence Cluster) is a computational grid with sufficient computational capacity for research in the field of [https://en.wikipedia.org/wiki/Deep_learning deep learning] using both CPU and GPU. It was built on top of [https://slurm.schedmd.com/ SLURM] scheduling system. MFF students of Bc. and Mgr. degrees can use it to run their experiments and learn the proper ways of grid computing in the process.

=== Access ===
AIC is dedicated to UFAL students who will get an account if requested by authorized lector.

There is a restriction on resources allocated by one user in group '''students''' at a given time.
By default, this is set to a maximum of 4 CPU and 1 GPU.

=== Connecting to the Cluster. ===
Use SSH to connect to the cluster:
ssh LOGIN@aic.ufal.mff.cuni.cz

=== Jupyterlab ===
AIC provides also Jupyterlab portal on top of your AIC account and HOME directory. It can be found at https://aic.ufal.mff.cuni.cz/jlab . Pre-installed extensions: R, ipython, Rstudio (community), Slurm Queue Manager.

=== Basic HOWTO ===

Following HOWTO is meant to provide only a simplified overview of the cluster usage. It is strongly recommended to read some further documentation ([[Submitting_CPU_Jobs|CPU]] or [[Submitting_GPU_Jobs|GPU]]) before running some serious experiments.
More serious experiments tend to take more resources. In order to avoid unexpected failures please make sure your [[Quotas|quota]] is not exceeded.

'''Rule 0: NEVER RUN JOBS DIRECTLY ON aic.ufal.mff.cuni.cz HEADNODE. Use <code>srun</code> to get computational node shell!'''

Suppose we want to run some computations described by a script called <code>job_script.sh</code>:

#!/bin/bash
#SBATCH -J helloWorld # name of job
#SBATCH -p cpu # name of partition or queue (default is cpu)
#SBATCH -o helloWorld.out # name of output file for this submission script
#SBATCH -e helloWorld.err # name of error file for this submission script
# run my job (some executable)
sleep 5
echo "Hello I am running on cluster!"

We need to ''submit'' the job to the cluster which is done by logging on the submit host <code>aic.ufal.mff.cuni.cz</code> and issuing the command:<br>
<code>sbatch job_script.sh</code>

This will enqueue our ''job'' to the default ''partition'' (or ''queue'') which is <code>cpu</code>. The scheduler decides which particular machine in the specified queue has ''resources'' needed to run the job. Typically we will see a message which tells us the ID of our job (3 in this example):

Submitted batch job 3

The options used in this example are specified inside the script using the ''#SBATCH'' directive. Any option can be specified either in the script or as a command line parameter (see ''man sbatch'' for details).

We can specify custom arguments '''before''' the name of the script:

sbatch --export=ARG1='firstArg',ARG2='secondArg' job_script.sh

These can be accessed in the job script as <code>$ARG1</code> and <code>$ARG2</code>.

Submitting GPU Jobs

2022-12-02T09:55:13Z

Admin: /* CUDA modules */

Start by reading [[Submitting CPU Jobs]] page.

The GPU jobs are submitted to <code>gpu</code> partition.

To ask for one GPU card, use <code>#SBATCH --gres=gpu:1</code> directive or <code>--gres=gpu:1</code> option on the command line. The submitted job has <code>CUDA_VISIBLE_DEVICES</code> set appropriately, so all CUDA applications should use only the allocated GPUs.

== Rules ==

* Always use GPUs via ''sbatch'' (or ''srun''), never via ''ssh''. You can ssh to any machine e.g. to run ''nvidia-smi'' or ''htop'', but not to start computing on GPU.
* Don't forget to specify you RAM requirements with e.g. ''--mem=10G''.
* Always specify the number of GPU cards (e.g. ''--gres=gpu:1''). Thus e.g. <code>srun -p gpu --mem=64G --gres=gpu:2 --pty bash</code>
* For interactive jobs, you can use ''srun'', but make sure to end your job as soon as you don't need the GPU (so don't use srun for long training).
* In general: don't reserve a GPU (as described above) without actually using it for longer time, e.g., try separating steps which need GPU and steps which do not and execute those separately on our GPU resp. CPU cluster.
* If you know an approximate runtime of your job, please specify it with ''-t <time>''. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds".

== CUDA and cuDNN ==

Available CUDA versions are in
/opt/cuda

=== CUDA modules ===
You can load late versions of CUDA as modules. This will set various environment variables for you so you should be able to use CUDA easily.

# list available modules with: <code>module avail</code>
# load the version you need (possibly specifying the version of CuDNN): <code>module load <modulename></code>
# you can unload the module with: <code>module unload <modulename></code>

Submitting GPU Jobs

2022-12-02T09:51:46Z

Admin: /* CUDA modules */

Start by reading [[Submitting CPU Jobs]] page.

The GPU jobs are submitted to <code>gpu</code> partition.

To ask for one GPU card, use <code>#SBATCH --gres=gpu:1</code> directive or <code>--gres=gpu:1</code> option on the command line. The submitted job has <code>CUDA_VISIBLE_DEVICES</code> set appropriately, so all CUDA applications should use only the allocated GPUs.

== Rules ==

* Always use GPUs via ''sbatch'' (or ''srun''), never via ''ssh''. You can ssh to any machine e.g. to run ''nvidia-smi'' or ''htop'', but not to start computing on GPU.
* Don't forget to specify you RAM requirements with e.g. ''--mem=10G''.
* Always specify the number of GPU cards (e.g. ''--gres=gpu:1''). Thus e.g. <code>srun -p gpu --mem=64G --gres=gpu:2 --pty bash</code>
* For interactive jobs, you can use ''srun'', but make sure to end your job as soon as you don't need the GPU (so don't use srun for long training).
* In general: don't reserve a GPU (as described above) without actually using it for longer time, e.g., try separating steps which need GPU and steps which do not and execute those separately on our GPU resp. CPU cluster.
* If you know an approximate runtime of your job, please specify it with ''-t <time>''. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds".

== CUDA and cuDNN ==

Available CUDA versions are in
/opt/cuda

=== CUDA modules ===
You can load late versions of CUDA as modules. This will set various environment variables for you so you should be able to use CUDA easily.

# list available modules with:
module avail
<li value=2> load the version you need (possibly specifying the version of CuDNN):
module load <modulename>
<li value=3> you can unload the module with:
module unload <modulename>

Submitting GPU Jobs

2022-12-02T09:49:27Z

Admin: /* CUDA and cuDNN */

Start by reading [[Submitting CPU Jobs]] page.

The GPU jobs are submitted to <code>gpu</code> partition.

To ask for one GPU card, use <code>#SBATCH --gres=gpu:1</code> directive or <code>--gres=gpu:1</code> option on the command line. The submitted job has <code>CUDA_VISIBLE_DEVICES</code> set appropriately, so all CUDA applications should use only the allocated GPUs.

== Rules ==

* Always use GPUs via ''sbatch'' (or ''srun''), never via ''ssh''. You can ssh to any machine e.g. to run ''nvidia-smi'' or ''htop'', but not to start computing on GPU.
* Don't forget to specify you RAM requirements with e.g. ''--mem=10G''.
* Always specify the number of GPU cards (e.g. ''--gres=gpu:1''). Thus e.g. <code>srun -p gpu --mem=64G --gres=gpu:2 --pty bash</code>
* For interactive jobs, you can use ''srun'', but make sure to end your job as soon as you don't need the GPU (so don't use srun for long training).
* In general: don't reserve a GPU (as described above) without actually using it for longer time, e.g., try separating steps which need GPU and steps which do not and execute those separately on our GPU resp. CPU cluster.
* If you know an approximate runtime of your job, please specify it with ''-t <time>''. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds".

== CUDA and cuDNN ==

Available CUDA versions are in
/opt/cuda

=== CUDA modules ===
You can load late versions of CUDA as modules. This will set various environment variables for you so you should be able to use CUDA easily.

# list available modules with:
module avail
# load the version you need (possibly specifying the version of CuDNN):
module load <modulename>
# you can unload the module with:
module unload <modulename>

Submitting CPU Jobs

2022-12-01T13:17:25Z

Admin: /* Monitoring and interaction */

Submitting CPU Jobs

2022-12-01T13:09:30Z

Admin: /* Selected submit options */

Submitting CPU Jobs

2022-12-01T12:48:13Z

Admin: /* Selected submit options */

Submitting CPU Jobs

2022-12-01T12:46:01Z

Admin:

Main Page

2022-12-01T12:40:32Z

Admin: /* Access */

<div style='text-align: center;'>CZ.02.2.69/0.0/0.0/17_044/0008562</div>
<div style='text-align: center;'>Podpora rozvoje studijního prostředí na Univerzitě Karlově - VRR</div>
[[File:OP_VVV_logo.jpg|frameless|center|upright=2.5]]

== Welcome to AIC ==

AIC (Artificial Intelligence Cluster) is a computational grid with sufficient computational capacity for research in the field of [https://en.wikipedia.org/wiki/Deep_learning deep learning] using both CPU and GPU. It was built on top of [https://slurm.schedmd.com/ SLURM] scheduling system. MFF students of Bc. and Mgr. degrees can use it to run their experiments and learn the proper ways of grid computing in the process.

=== Access ===
AIC is dedicated to UFAL students who will get an account if requested by authorized lector.

There is a restriction on resources allocated by one user in group '''students''' at a given time.
By default, this is set to a maximum of 4 CPU and 1 GPU.

=== Connecting to the Cluster. ===
Use SSH to connect to the cluster:
ssh LOGIN@aic.ufal.mff.cuni.cz

=== Basic HOWTO ===

Following HOWTO is meant to provide only a simplified overview of the cluster usage. It is strongly recommended to read some further documentation ([[Submitting_CPU_Jobs|CPU]] or [[Submitting_GPU_Jobs|GPU]]) before running some serious experiments.
More serious experiments tend to take more resources. In order to avoid unexpected failures please make sure your [[Quotas|quota]] is not exceeded.

'''Rule 0: NEVER RUN JOBS DIRECTLY ON aic.ufal.mff.cuni.cz HEADNODE. Use <code>srun</code> to get computational node shell!'''

Suppose we want to run some computations described by a script called <code>job_script.sh</code>:

#!/bin/bash
#SBATCH -J helloWorld # name of job
#SBATCH -p cpu # name of partition or queue (default is cpu)
#SBATCH -o helloWorld.out # name of output file for this submission script
#SBATCH -e helloWorld.err # name of error file for this submission script
# run my job (some executable)
sleep 5
echo "Hello I am running on cluster!"

We need to ''submit'' the job to the cluster which is done by logging on the submit host <code>aic.ufal.mff.cuni.cz</code> and issuing the command:<br>
<code>sbatch job_script.sh</code>

This will enqueue our ''job'' to the default ''partition'' (or ''queue'') which is <code>cpu</code>. The scheduler decides which particular machine in the specified queue has ''resources'' needed to run the job. Typically we will see a message which tells us the ID of our job (3 in this example):

Submitted batch job 3

The options used in this example are specified inside the script using the ''#SBATCH'' directive. Any option can be specified either in the script or as a command line parameter (see ''man sbatch'' for details).

We can specify custom arguments '''before''' the name of the script:

sbatch --export=ARG1='firstArg',ARG2='secondArg' job_script.sh

These can be accessed in the job script as <code>$ARG1</code> and <code>$ARG2</code>.

Main Page

2022-12-01T12:22:59Z

Admin: /* Basic HOWTO */

<div style='text-align: center;'>CZ.02.2.69/0.0/0.0/17_044/0008562</div>
<div style='text-align: center;'>Podpora rozvoje studijního prostředí na Univerzitě Karlově - VRR</div>
[[File:OP_VVV_logo.jpg|frameless|center|upright=2.5]]

== Welcome to AIC ==

AIC (Artificial Intelligence Cluster) is a computational grid with sufficient computational capacity for research in the field of [https://en.wikipedia.org/wiki/Deep_learning deep learning] using both CPU and GPU. It was built on top of [https://slurm.schedmd.com/ SLURM] scheduling system. MFF students of Bc. and Mgr. degrees can use it to run their experiments and learn the proper ways of grid computing in the process.

=== Access ===
AIC is dedicated to UFAL students who will get an account if requested by authorized lector.

=== Connecting to the Cluster. ===
Use SSH to connect to the cluster:
ssh LOGIN@aic.ufal.mff.cuni.cz

=== Basic HOWTO ===

Following HOWTO is meant to provide only a simplified overview of the cluster usage. It is strongly recommended to read some further documentation ([[Submitting_CPU_Jobs|CPU]] or [[Submitting_GPU_Jobs|GPU]]) before running some serious experiments.
More serious experiments tend to take more resources. In order to avoid unexpected failures please make sure your [[Quotas|quota]] is not exceeded.

'''Rule 0: NEVER RUN JOBS DIRECTLY ON aic.ufal.mff.cuni.cz HEADNODE. Use <code>srun</code> to get computational node shell!'''

Suppose we want to run some computations described by a script called <code>job_script.sh</code>:

#!/bin/bash
#SBATCH -J helloWorld # name of job
#SBATCH -p cpu # name of partition or queue (default is cpu)
#SBATCH -o helloWorld.out # name of output file for this submission script
#SBATCH -e helloWorld.err # name of error file for this submission script
# run my job (some executable)
sleep 5
echo "Hello I am running on cluster!"

We need to ''submit'' the job to the cluster which is done by logging on the submit host <code>aic.ufal.mff.cuni.cz</code> and issuing the command:<br>
<code>sbatch job_script.sh</code>

This will enqueue our ''job'' to the default ''partition'' (or ''queue'') which is <code>cpu</code>. The scheduler decides which particular machine in the specified queue has ''resources'' needed to run the job. Typically we will see a message which tells us the ID of our job (3 in this example):

Submitted batch job 3

The options used in this example are specified inside the script using the ''#SBATCH'' directive. Any option can be specified either in the script or as a command line parameter (see ''man sbatch'' for details).

We can specify custom arguments '''before''' the name of the script:

sbatch --export=ARG1='firstArg',ARG2='secondArg' job_script.sh

These can be accessed in the job script as <code>$ARG1</code> and <code>$ARG2</code>.

Main Page

2022-12-01T12:22:21Z

Admin: /* Basic HOWTO */

<div style='text-align: center;'>CZ.02.2.69/0.0/0.0/17_044/0008562</div>
<div style='text-align: center;'>Podpora rozvoje studijního prostředí na Univerzitě Karlově - VRR</div>
[[File:OP_VVV_logo.jpg|frameless|center|upright=2.5]]

== Welcome to AIC ==

AIC (Artificial Intelligence Cluster) is a computational grid with sufficient computational capacity for research in the field of [https://en.wikipedia.org/wiki/Deep_learning deep learning] using both CPU and GPU. It was built on top of [https://slurm.schedmd.com/ SLURM] scheduling system. MFF students of Bc. and Mgr. degrees can use it to run their experiments and learn the proper ways of grid computing in the process.

=== Access ===
AIC is dedicated to UFAL students who will get an account if requested by authorized lector.

=== Connecting to the Cluster. ===
Use SSH to connect to the cluster:
ssh LOGIN@aic.ufal.mff.cuni.cz

=== Basic HOWTO ===

Following HOWTO is meant to provide only a simplified overview of the cluster usage. It is strongly recommended to read some further documentation ([[Submitting_CPU_Jobs|CPU]] or [[Submitting_GPU_Jobs|GPU]]) before running some serious experiments.
More serious experiments tend to take more resources. In order to avoid unexpected failures please make sure your [[Quotas|quota]] is not exceeded.

'''Rule 0: NEVER RUN JOBS DIRECTLY ON aic.ufal.mff.cuni.cz HEADNODE. Use '''srun''' to get computational node shell!'''

Suppose we want to run some computations described by a script called <code>job_script.sh</code>:

#!/bin/bash
#SBATCH -J helloWorld # name of job
#SBATCH -p cpu # name of partition or queue (default is cpu)
#SBATCH -o helloWorld.out # name of output file for this submission script
#SBATCH -e helloWorld.err # name of error file for this submission script
# run my job (some executable)
sleep 5
echo "Hello I am running on cluster!"

We need to ''submit'' the job to the cluster which is done by logging on the submit host <code>aic.ufal.mff.cuni.cz</code> and issuing the command:<br>
<code>sbatch job_script.sh</code>

This will enqueue our ''job'' to the default ''partition'' (or ''queue'') which is <code>cpu</code>. The scheduler decides which particular machine in the specified queue has ''resources'' needed to run the job. Typically we will see a message which tells us the ID of our job (3 in this example):

Submitted batch job 3

The options used in this example are specified inside the script using the ''#SBATCH'' directive. Any option can be specified either in the script or as a command line parameter (see ''man sbatch'' for details).

We can specify custom arguments '''before''' the name of the script:

sbatch --export=ARG1='firstArg',ARG2='secondArg' job_script.sh

These can be accessed in the job script as <code>$ARG1</code> and <code>$ARG2</code>.

Main Page

2022-11-22T10:34:43Z

Admin: /* Welcome to AIC */

<div style='text-align: center;'>CZ.02.2.69/0.0/0.0/17_044/0008562</div>
<div style='text-align: center;'>Podpora rozvoje studijního prostředí na Univerzitě Karlově - VRR</div>
[[File:OP_VVV_logo.jpg|frameless|center|upright=2.5]]

== Welcome to AIC ==

AIC (Artificial Intelligence Cluster) is a computational grid with sufficient computational capacity for research in the field of [https://en.wikipedia.org/wiki/Deep_learning deep learning] using both CPU and GPU. It was built on top of [https://slurm.schedmd.com/ SLURM] scheduling system. MFF students of Bc. and Mgr. degrees can use it to run their experiments and learn the proper ways of grid computing in the process.

=== Access ===
AIC is dedicated to UFAL students who will get an account if requested by authorized lector.

=== Connecting to the Cluster. ===
Use SSH to connect to the cluster:
ssh LOGIN@aic.ufal.mff.cuni.cz

=== Basic HOWTO ===

Following HOWTO is meant to provide only a simplified overview of the cluster usage. It is strongly recommended to read some further documentation ([[Submitting_CPU_Jobs|CPU]] or [[Submitting_GPU_Jobs|GPU]]) before running some serious experiments.
More serious experiments tend to take more resources. In order to avoid unexpected failures please make sure your [[Quotas|quota]] is not exceeded.

'''Rule 0: NEVER RUN JOBS DIRECTLY ON aic.ufal.mff.cuni.cz HEADNODE. Use *srun* to get computational node shell!'''

Suppose we want to run some computations described by a script called <code>job_script.sh</code>:

#!/bin/bash
#SBATCH -J helloWorld # name of job
#SBATCH -p cpu # name of partition or queue (default is cpu)
#SBATCH -o helloWorld.out # name of output file for this submission script
#SBATCH -e helloWorld.err # name of error file for this submission script
# run my job (some executable)
sleep 5
echo "Hello I am running on cluster!"

We need to ''submit'' the job to the cluster which is done by logging on the submit host <code>aic.ufal.mff.cuni.cz</code> and issuing the command:<br>
<code>sbatch job_script.sh</code>

This will enqueue our ''job'' to the default ''partition'' (or ''queue'') which is <code>cpu</code>. The scheduler decides which particular machine in the specified queue has ''resources'' needed to run the job. Typically we will see a message which tells us the ID of our job (3 in this example):

Submitted batch job 3

The options used in this example are specified inside the script using the ''#SBATCH'' directive. Any option can be specified either in the script or as a command line parameter (see ''man sbatch'' for details).

We can specify custom arguments '''before''' the name of the script:

sbatch --export=ARG1='firstArg',ARG2='secondArg' job_script.sh

These can be accessed in the job script as <code>$ARG1</code> and <code>$ARG2</code>.

Submitting GPU Jobs

2022-11-16T15:23:06Z

Admin:

Start by reading [[Submitting CPU Jobs]] page.

The GPU jobs are submitted to <code>gpu</code> partition.

To ask for one GPU card, use <code>#SBATCH --gres=gpu:1</code> directive or <code>--gres=gpu:1</code> option on the command line. The submitted job has <code>CUDA_VISIBLE_DEVICES</code> set appropriately, so all CUDA applications should use only the allocated GPUs.

== Rules ==

* Always use GPUs via ''sbatch'' (or ''srun''), never via ''ssh''. You can ssh to any machine e.g. to run ''nvidia-smi'' or ''htop'', but not to start computing on GPU.
* Don't forget to specify you RAM requirements with e.g. ''--mem=10G''.
* Always specify the number of GPU cards (e.g. ''--gres=gpu:1''). Thus e.g. <code>srun -p gpu --mem=64G --gres=gpu:2 --pty bash</code>
* For interactive jobs, you can use ''srun'', but make sure to end your job as soon as you don't need the GPU (so don't use srun for long training).
* In general: don't reserve a GPU (as described above) without actually using it for longer time, e.g., try separating steps which need GPU and steps which do not and execute those separately on our GPU resp. CPU cluster.
* If you know an approximate runtime of your job, please specify it with ''-t <time>''. Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds".

== CUDA and cuDNN ==

Default CUDA (currently 11.2 as of Nov 2021) is available in
/opt/cuda
Specific version can be found in
/lnet/aic/opt/cuda/cuda-{9.0,9.2,10.0,10.1,10.2,11.2,...}
Depending on what version you need, you should add <code>LD_LIBRARY_PATH="/lnet/aic/opt/cuda/cuda-X.Y/lib64:$LD_LIBRARY_PATH"</code> to your configuration.

Regarding cuDNN:
* for CUDA 9.0, 9.2, 10.0 and 10.1, cuDNN is available directly in ''lib64'' directory of the respective CUDA, so no need to configure it specifically;
* for CUDA 10.1 and later, cuDNN is available in ''cudnn/VERSION/lib64'' subdirectory of the respective CUDA, so you need to add <code>LD_LIBRARY_PATH="/lnet/aic/opt/cuda/cuda-X.Y/cudnn/VERSION/lib64:$LD_LIBRARY_PATH"</code> to your configuration.

Submitting CPU Jobs

2022-11-16T15:13:39Z

Admin: modified for SLURM - first draft

Quotas

2022-11-16T14:51:09Z

Admin:

This page lists resource quotas for the AIC cluster users.

== Disk Quotas ==

Everyone has a disk quota set (the default was 50G as of Jan 2020).

You can find out the quota and your current disk usage by running

lfs quota -h /lnet/aic

Main Page

2022-11-16T14:49:20Z

Admin: /* Basic HOWTO - modified for SLURM */

<div style='text-align: center;'>CZ.02.2.69/0.0/0.0/17_044/0008562</div>
<div style='text-align: center;'>Podpora rozvoje studijního prostředí na Univerzitě Karlově - VRR</div>
[[File:OP_VVV_logo.jpg|frameless|center|upright=2.5]]

== Welcome to AIC ==

AIC (Artificial Intelligence Cluster) is a computational grid with sufficient computational capacity for research in the field of [https://en.wikipedia.org/wiki/Deep_learning deep learning] using both CPU and GPU. It was built on top of [https://arc.liv.ac.uk/trac/SGE SGE] scheduling system. MFF students of Bc. and Mgr. degrees can use it to run their experiments and learn the proper ways of grid computing in the process.

=== Access ===
AIC is dedicated to UFAL students who will get an account if requested by authorized lector.

=== Connecting to the Cluster. ===
Use SSH to connect to the cluster:
ssh LOGIN@aic.ufal.mff.cuni.cz

=== Basic HOWTO ===

Following HOWTO is meant to provide only a simplified overview of the cluster usage. It is strongly recommended to read some further documentation ([[Submitting_CPU_Jobs|CPU]] or [[Submitting_GPU_Jobs|GPU]]) before running some serious experiments.
More serious experiments tend to take more resources. In order to avoid unexpected failures please make sure your [[Quotas|quota]] is not exceeded.

'''Rule 0: NEVER RUN JOBS DIRECTLY ON aic.ufal.mff.cuni.cz HEADNODE. Use *srun* to get computational node shell!'''

Suppose we want to run some computations described by a script called <code>job_script.sh</code>:

#!/bin/bash
#SBATCH -J helloWorld # name of job
#SBATCH -p cpu # name of partition or queue (default is cpu)
#SBATCH -o helloWorld.out # name of output file for this submission script
#SBATCH -e helloWorld.err # name of error file for this submission script
# run my job (some executable)
sleep 5
echo "Hello I am running on cluster!"

We need to ''submit'' the job to the cluster which is done by logging on the submit host <code>aic.ufal.mff.cuni.cz</code> and issuing the command:<br>
<code>sbatch job_script.sh</code>

This will enqueue our ''job'' to the default ''partition'' (or ''queue'') which is <code>cpu</code>. The scheduler decides which particular machine in the specified queue has ''resources'' needed to run the job. Typically we will see a message which tells us the ID of our job (3 in this example):

Submitted batch job 3

The options used in this example are specified inside the script using the ''#SBATCH'' directive. Any option can be specified either in the script or as a command line parameter (see ''man sbatch'' for details).

We can specify custom arguments '''before''' the name of the script:

sbatch --export=ARG1='firstArg',ARG2='secondArg' job_script.sh

These can be accessed in the job script as <code>$ARG1</code> and <code>$ARG2</code>.

Main Page

2022-11-16T14:10:21Z

Admin: /* Basic HOWTO */

<div style='text-align: center;'>CZ.02.2.69/0.0/0.0/17_044/0008562</div>
<div style='text-align: center;'>Podpora rozvoje studijního prostředí na Univerzitě Karlově - VRR</div>
[[File:OP_VVV_logo.jpg|frameless|center|upright=2.5]]

== Welcome to AIC ==

AIC (Artificial Intelligence Cluster) is a computational grid with sufficient computational capacity for research in the field of [https://en.wikipedia.org/wiki/Deep_learning deep learning] using both CPU and GPU. It was built on top of [https://arc.liv.ac.uk/trac/SGE SGE] scheduling system. MFF students of Bc. and Mgr. degrees can use it to run their experiments and learn the proper ways of grid computing in the process.

=== Access ===
AIC is dedicated to UFAL students who will get an account if requested by authorized lector.

=== Connecting to the Cluster. ===
Use SSH to connect to the cluster:
ssh LOGIN@aic.ufal.mff.cuni.cz

=== Basic HOWTO ===

Following HOWTO is meant to provide only a simplified overview of the cluster usage. It is strongly recommended to read some further documentation ([[Submitting_CPU_Jobs|CPU]] or [[Submitting_GPU_Jobs|GPU]]) before running some serious experiments.
More serious experiments tend to take more resources. In order to avoid unexpected failures please make sure your [[Quotas|quota]] is not exceeded.

'''Rule 0: NEVER RUN JOBS DIRECTLY ON aic.ufal.mff.cuni.cz HEADNODE. Use *srun* to get computational node shell!'''

Suppose we want to run some computations described by a script called <code>job_script.sh</code>:

#!/bin/bash
#SBATCH -J helloWorld # name of job
#SBATCH -p cpu # name of partition or queue (default is cpu)
#SBATCH -o helloWorld.out # name of output file for this submission script
#SBATCH -e helloWorld.err # name of error file for this submission script
# run my job (some executable)
sleep 5
echo "Hello I am running on cluster!"

We need to ''submit'' the job to the cluster which is done by logging on the submit host <code>aic.ufal.mff.cuni.cz</code> and issuing the command:<br>
<code>sbatch job_script.sh</code>

This will enqueue our ''job'' to the default ''partition'' (or ''queue'') which is <code>cpu</code>. The scheduler decides which particular machine in the specified queue has ''resources'' needed to run the job. Typically we will see a message which tells us the ID of our job (82 in this example):

Your job 82 ("job_script.sh") has been submitted

The basic options used in this example are:
* <code>-cwd</code> - the script is executed in the current directory (the default is your <code>$HOME</code>)
* <code>-j y</code> - ''stdout'' and ''stderr'' outputs are merged and redirected to a file (<code>job_script.sh.o82</code>)

We have specified two parameters <code>Hello</code> and <code>World</code>. The output of the script will be located in your <code>$HOME</code> directory after the script is executed. It will be merged with ''stderr'' and it should look like this:

AIC:ubuntu 18.04: SGE 8.1.9 configured...
This is just a test.
printing parameter1: Hello
prinitng parameter2: World
======= EPILOG: Tue Jun 4 12:41:07 CEST 2019
== Limits:
== Usage: cpu=00:00:00, mem=0.00000 GB s, io=0.00000 GB, vmem=N/A, maxvmem=N/A
== Duration: 00:00:00 (0 s)
== Server name: cpu-node13

Main Page

2022-03-22T08:21:52Z

Admin: /* Basic HOWTO */

<div style='text-align: center;'>CZ.02.2.69/0.0/0.0/17_044/0008562</div>
<div style='text-align: center;'>Podpora rozvoje studijního prostředí na Univerzitě Karlově - VRR</div>
[[File:OP_VVV_logo.jpg|frameless|center|upright=2.5]]

== Welcome to AIC ==

AIC (Artificial Intelligence Cluster) is a computational grid with sufficient computational capacity for research in the field of [https://en.wikipedia.org/wiki/Deep_learning deep learning] using both CPU and GPU. It was built on top of [https://arc.liv.ac.uk/trac/SGE SGE] scheduling system. MFF students of Bc. and Mgr. degrees can use it to run their experiments and learn the proper ways of grid computing in the process.

=== Access ===
AIC is dedicated to UFAL students who will get an account if requested by authorized lector.

=== Connecting to the Cluster. ===
Use SSH to connect to the cluster:
ssh LOGIN@aic.ufal.mff.cuni.cz

=== Basic HOWTO ===

Following HOWTO is meant to provide only a simplified overview of the cluster usage. It is strongly recommended to read some further documentation ([[Submitting_CPU_Jobs|CPU]] or [[Submitting_GPU_Jobs|GPU]]) before running some serious experiments.
More serious experiments tend to take more resources. In order to avoid unexpected failures please make sure your [[Quotas|quota]] is not exceeded.

'''Rule 0: NEVER RUN JOBS DIRECTLY ON aic.ufal.mff.cuni.cz HEADNODE. Use qrsh to get computational node shell!'''

Suppose we want to run some computations described by a script called <code>job_script.sh</code>:

#!/bin/bash
echo "This is just a test."
echo "printing parameter1: $1"
echo "prinitng parameter2: $2"

We need to ''submit'' the job to the grid which is done by logging on the submit host <code>aic.ufal.mff.cuni.cz</code> and issuing the command:<br>
<code>qsub -cwd -j y job_script.sh Hello World</code>

This will enqueue our ''job'' to the default ''queue'' which is <code>cpu.q@*</code>. The scheduler decides which particular machine in the specified queue has ''resources'' needed to run the job. Typically we will see a message which tells us the ID of our job (82 in this example):

Your job 82 ("job_script.sh") has been submitted

The basic options used in this example are:
* <code>-cwd</code> - the script is executed in the current directory (the default is your <code>$HOME</code>)
* <code>-j y</code> - ''stdout'' and ''stderr'' outputs are merged and redirected to a file (<code>job_script.sh.o82</code>)

We have specified two parameters <code>Hello</code> and <code>World</code>. The output of the script will be located in your <code>$HOME</code> directory after the script is executed. It will be merged with ''stderr'' and it should look like this:

AIC:ubuntu 18.04: SGE 8.1.9 configured...
This is just a test.
printing parameter1: Hello
prinitng parameter2: World
======= EPILOG: Tue Jun 4 12:41:07 CEST 2019
== Limits:
== Usage: cpu=00:00:00, mem=0.00000 GB s, io=0.00000 GB, vmem=N/A, maxvmem=N/A
== Duration: 00:00:00 (0 s)
== Server name: cpu-node13

Submitting CPU Jobs

2020-01-29T10:26:27Z

Admin: Admin moved page Submitting Jobs to Submitting CPU Jobs without leaving a redirect: Preserve history of original page

The CPU jobs should be submitted to <code>cpu.q</code> queue.

== Resource specification ==

You should specify the memory and CPU requirements (if higher than the defaults) and don't exceed them.
If your job needs more than one CPU (on a single machine) for most of the time, reserve the given number of CPU cores (and SGE slots) with

<code>qsub -pe smp <number-of-CPU-cores></code>

The maximum for AIC cluster is 4 cores. If your job needs e.g. up to 110% CPU most of the time and just occasionally 200%, it is OK to reserve just one core (so you don't waste).

If you are sure your job needs less than 1GB RAM, then you can skip this. Otherwise, if you need e.g. 8 GiB, you must always use <code>qsub</code> (or <code>qrsh</code>) with <code>-l mem_free=8G</code>. You should specify also <code>act_mem_free</code> with the same value and <code>h_vmem</code> with possibly a slightly bigger value. See [[#Memory]] for details.

<code>qsub -l mem_free=8G,act_mem_free=8G,h_vmem=12G</code>

== Monitoring and interaction ==

=== Job monitoring ===
We should be able to see what is going on when we run a job. Following examples shows usage of some typical commands:
* <code>qstat</code> - this way we inspect all our jobs (both waiting in the queue and scheduled, i.e. running).
* <code>qstat [-u user]</code> - print a list of running/waiting jobs of a given user
* <code>qstat -u '*' | less</code> - this shows the jobs of all users.
* <code>qstat -j 121144</code> - this shows detailed info about the job with this number (if it is still running).
* <code>qhost</code> - print available/total resources
* <code>qacct -j job_id</code> - print info even for ended job (for which ''qstat -j job_id'' does not work). See <code>man qacct</code> for more.

=== Output monitoring ===
If we need to see output produced by our job (suppose the ID is 121144), we can inspect the job's output (in our case stored in <code>job_script.sh.o121144</code>) with:<br>
<code>less job_script.sh.o*</code><br>
''Hint:'' if the job is still running, press '''F''' in <code>less</code> to simulate <code>tail -f</code>.

==== How to read output epilog ====
The epilog section contains some interesting pieces of information. However this it can get confusing sometimes.

======= EPILOG: Tue Jun 4 12:41:07 CEST 2019
== Limits:
== Usage: cpu=00:00:00, mem=0.00000 GB s, io=0.00000 GB, vmem=N/A, maxvmem=N/A
== Duration: 00:00:00 (0 s)
== Server name: cpu-node13

* ''Limits'' - on this line you can see job limits specified through <code>qsub</code> options
* ''Usage'' - resource usage during computation
** ''cpu=HH:MM:SS'' - the accumulated CPU time usage
** ''mem=XY GB'' - gigabytes of RAM used times the duration of the job in seconds, so don't be afraid XY is usually a very high number (unlike in this toy example)
** ''io=XY GB'' - the amount of data transferred in input/output operations in GB
** ''vmem=XY'' - actual virtual memory consumption when the job finished
** ''maxvmem=XY'' - peak virtual memory consumption
* ''Duration'' - total execution time
* ''Server name'' - name of the executing server

=== Job interaction ===

<code>qdel 121144</code>
This way you can delete (''kill'') a job with a given number, or comma-or-space separated list of job numbers.

<code>qdel \*</code>
This way you can delete all your jobs. Don't be afraid - you cannot delete others jobs.

<code>qalter</code>
You can change some properties of already submitted jobs (both waiting in the queue and running). Changeable properties are listed in <code>man qsub</code>.

== Advanced usage ==
<code>qsub '''-q''' cpu.q</code>
This way your job is submitted to the CPU queue which is the default. If you need GPU use <code>gpu.q</code> instead.

<code>qsub '''-l''' ...</code>
See <code>man complex</code> (run it on aic) for a list of possible resources you may require (in addition to <code>mem_free</code> etc. discussed above).

<code>qsub '''-p''' -200</code>
Define a priority of your job as a number between -1024 and 0. Only SGE admins may use a number higher than 0. Default is set to TODO. You should ask for lower priority (-1024..-101) if you submit many jobs at once or if the jobs are not urgent. SGE uses the priority to decide when to start which pending job in the queue (it computes a real number called <code>prior</code>, which is reported in <code>qstat</code>, which grows as the job is waiting in the queue). Note that once a job is started, you cannot ''unschedule'' it, so from that moment on, it is irrelevant what was its priority.

<code>qsub '''-o''' LOG.stdout '''-e''' LOG.stderr</code>
redirect std{out,err} to separate files with given names, instead of the defaults <code>$JOB_NAME.o$JOB_ID</code> and <code>$JOB_NAME.e$JOB_ID</code>.

<code>qsub '''-@''' optionfile</code>
Instead of specifying all the <code>qsub</code> options on the command line, you can store them in a file (you can use # comments in the file).

<code>qsub '''-a''' 12312359</code>
Execute your job no sooner than at the given time (in <code>[YY]MMDDhhmm</code> format). An alternative to <code>sleep 3600 && qsub ... &</code>.

<code>qsub '''-b''' y</code>
Treat <code>script.sh</code> (or whatever is the name of the command you execute) as a binary, i.e. don't search for in-script options within the file, don't transfer it to the ''qmaster'' and then to the execution node. This makes the execution a bit faster and it may prevent some rare but hard-to-detect errors caused SGE interpreting the script. The script must be available on the execution node via Lustre (which is our case), etc. With <code>-b y</code> (shortcut for <code>-b yes</code>), <code>script.sh</code> can be a script or a binary. With <code>-b n</code> (which is the default for <code>qsub</code>), <code>script.sh</code> must be a script (text file).

<code>qsub '''-M''' person1@email.somewhere.cz,person2@email.somewhere.cz '''-m''' beas</code>
Specify the emails where you want to be notified when the job has been '''b''' started, '''e''' ended, '''a''' aborted, rescheduled or '''s''' suspended.
The default is now <code>-m a</code> and the default email address is forwarded to you (so there is no need to use '''-M'''). You can use <code>-m n</code> to override the defaults and send no emails.

<code>qsub '''-hold_jid''' 121144,121145</code> (or <code>qsub '''-hold_jid''' get_src.sh,get_tgt.sh</code>)
The current job is not executed before all the specified jobs are completed.

<code>qsub '''-now''' y</code>
Start the job immediately or not at all, i.e. don't put it as pending to the queue. This is the default for <code>qrsh</code>, but you can change it with <code>-now n</code> (which is the default for <code>qsub</code>).

<code>qsub '''-N''' my-name</code>
By default the name of a job (which you can see e.g. in <code>qstat</code>) is the name of the <code>script.sh</code>. This way you can override it.

<code>qsub '''-S''' /bin/bash</code>
The hashbang (<code>!#/bin/bash</code>) in your <code>script.sh</code> is ignored, but you can change the interpreter with ''-S''. The default interpreter is <code>/bin/bash</code>.

<code>qsub '''-v''' PATH[=value]</code>
Export a given environment variable from the current shell to the job.

<code>qsub '''-V'''</code>
Export all environment variables. (This is not so needed now, when bash is the default interpreter and it seems your <code>~/.bashrc</code> is always sourced.)

<code>qsub '''-soft''' -l ... '''-hard''' -l ... -q ...</code>
By default, all the resource requirements (specified with <code>-l</code>) and queue requirements (specified with ''-q'') are '''hard''', i.e. your job won't be scheduled unless they can be fulfilled. You can use <code>-soft</code> to mark all following requirements as nice-to-have. And with <code>-hard</code> you can switch back to hard requirements.

<code>qsub '''-sync''' y</code>
This causes qsub to wait for the job to complete before exiting (with the same exit code as the job). Useful in scripts.

== Memory ==

* There are three commonly used options for specifying memory requirements: '''mem_free''', '''act_mem_free''' and '''h_vmem'''. Each has a different purpose.
* '''mem_free=1G''' means 1024×1024×1024 bytes, i.e. one [[https://en.wikipedia.org/wiki/Gibibyte|GiB (gibibyte)]]. '''mem_free=1g''' means 1000×1000×1000 bytes, i.e. one gigabyte. Similarly for the other options and other prefixes (k, K, m, M).
* '''mem_free''' (or '''mf''') specifies a ''consumable resource'' tracked by SGE and it affects job scheduling. Each machine has an initial value assigned (slightly lower than the real total physical RAM capacity). When you specify <code>qsub -l mem_free=4G</code>, SGE finds a machine with '''mem_free''' >= 4GB, and subtracts 4GB from it. This limit is not enforced, so if a job exceeds this limit, ''it is not automatically killed'' and thus the SGE value of '''mem_free''' may not represent the real free memory. The default value is 1G. By not using this option and eating more than 1 GiB, you are breaking the rules.
* '''act_mem_free''' (or '''amf''') is a ÚFAL-specific option, which specifies the real amount of free memory (at the time of scheduling). You can specify it when submitting a job and it will be scheduled to a machine with at least this amount of memory free. In an ideal world, where no jobs are exceeding their ''mem_free'' requirements, we would not need this option. However, in the real world, it is recommended to use this option with the same value as ''mem_free'' to protect your job from failing with out-of-memory error (because of naughty jobs of other users).
* '''h_vmem''' is equivalent to setting '''ulimit -v''', i.e. it is a hard limit on the size of virtual memory (see RLIMIT_AS in <code>man setrlimit</code>). If your job exceeds this limit, memory allocation fails (i.e., malloc or mmap will return NULL), and your job will probably crash on SIGSEGV. TODO: according to <code>man queue_conf</code>, the job is killed with SIGKILL, not with SIGSEGV. Note that '''h_vmem''' specifies the maximal size of ''allocated_memory'', not ''used_memory'', in other words it is the VIRT column in <code>top</code>, not the RES column. SGE does not use this parameter in any other way. Notably, job scheduling is not affected by this parameter and therefore there is no guarantee that there will be this amount of memory on the chosen machine. The problem is that some programs (e.g. Java with the default setting) allocate much more (virtual) memory than they actually use in the end. If we want to be ultra conservative, we should set '''h_vmem''' to the same value as '''mem_free'''. If we want to be only moderately conservative, we should specify something like '''h_vmem=1.5*mem_free''', because some jobs will not use the whole mem_free requested, but still our job will be killed if it allocated much more than declared. The default effectively means that your job has no limits.
* For GPU jobs, it is usually better to use '''h_data''' instead of '''h_vmem'''. CUDA driver allocates a lot of ''unused'' virtual memory (tens of GB per card), which is counted in '''h_vmem''', but not in '''h_data'''. All usual allocations (''malloc'', ''new'', Python allocations) seem to be included in '''h_data'''.
* It is recommended to ''profile your task first'' (see [[#Profiling]] below, so you can estimate reasonable memory requirements before submitting many jobs with the same task (varying in parameters which do not affect memory consumption). So for the first time, declare mem_free with much more memory than expected and ssh to a given machine and check <code>htop</code> (sum all processes of your job) or (if the job is done quickly) check the epilog. When running other jobs of this type, set '''mem_free''' (and '''act_mem_free''' and '''h_vmem''') so you are not wasting resources, but still have some reserve.
* '''s_vmem''' is similar to '''h_vmem''', but instead of SIGSEGV/SIGKILL, the job is sent a SIGXCPU signal which can be caught by the job and exit gracefully before it is killed. So if you need it, set '''s_vmem''' to a lower value than '''h_vmem''' and implement SIGXCPU handling and cleanup.

== Profiling ==
As stated above, you should always specify the exact memory limits when running your tasks, so that you neither waste RAM nor starve others of memory by using more than you requested. However, memory requirements can be difficult to estimate in advance. That's why you should profile your tasks first.

A simple method is to run the task and observe the memory usage reported in the epilog, but SGE may not record transient allocations. As documented in <code>man 5 accounting</code> and observed in <code>qconf -sconf</code>, SGE only collects stats every '''accounting_flush_time'''. If this is not set, it defaults to '''flush_time''', which is preset to 15 seconds. But the kernel records all info immediately without polling, and you can view these exact stats by looking into <code>/proc/$PID/status</code> while the task is running.

You can still miss allocations made shortly before the program exits – which often happens when trying to debug why your program gets killed by SGE after exhausting the reserved space. To record these, use <code>/usr/bin/time -v</code> (the actual binary, not the shell-builtin command <code>time</code>). Be aware that unlike the builtin, it cannot measure shell functions and behaves differently on pipelines.

Obtaining peak usage of multiprocess applications is trickier. Detached and backgrounded processes are ignored completely by <code>time -v</code> and you get the maximum footprint of any children, not the sum of all maximal footprints nor the largest footprint in any instant.

If you program in C and need to know the peak memory usage of your children, you can also use the '''wait4()''' syscall and calculate the stats yourself.

If your job is the only one on a given machine, you can also look how much free memory is left when running the job (e.g. with <code>htop</code> if you know when is the peak moment).

Main Page

2020-01-29T10:13:12Z

Admin: Removed link to original documentation

<div style='text-align: center;'>CZ.02.2.69/0.0/0.0/17_044/0008562</div>
<div style='text-align: center;'>Podpora rozvoje studijního prostředí na Univerzitě Karlově - VRR</div>
[[File:OP_VVV_logo.jpg|frameless|center|upright=2.5]]

== Welcome to AIC ==

AIC (Artificial Intelligence Cluster) is a computational grid with sufficient computational capacity for research in the field of [https://en.wikipedia.org/wiki/Deep_learning deep learning] using both CPU and GPU. It was built on top of [https://arc.liv.ac.uk/trac/SGE SGE] scheduling system. MFF students of Bc. and Mgr. degrees can use it to run their experiments and learn the proper ways of grid computing in the process.

=== Access ===
AIC is dedicated to UFAL students who will get an account if requested by authorized lector.

=== Basic HOWTO ===

Following HOWTO is meant to provide only a simplified overview of the cluster usage. It is strongly recommended to read some further documentation ([[Submitting_CPU_Jobs|CPU]] or [[Submitting_GPU_Jobs|GPU]]) before running some serious experiments.
More serious experiments tend to take more resources. In order to avoid unexpected failures please make sure your [[Quotas|quota]] is not exceeded.

Suppose we want to run some computations described by a script called <code>job_script.sh</code>:

#!/bin/bash
echo "This is just a test."
echo "printing parameter1: $1"
echo "prinitng parameter2: $2"

We need to ''submit'' the job to the grid which is done by logging on the submit host <code>aic.ufal.mff.cuni.cz</code> and issuing the command:<br>
<code>qsub -cwd -j y job_script.sh Hello World</code>

This will enqueue our ''job'' to the default ''queue'' which is <code>cpu.q@*</code>. The scheduler decides which particular machine in the specified queue has ''resources'' needed to run the job. Typically we will see a message which tells us the ID of our job (82 in this example):

Your job 82 ("job_script.sh") has been submitted

The basic options used in this example are:
* <code>-cwd</code> - the script is executed in the current directory (the default is your <code>$HOME</code>)
* <code>-j y</code> - ''stdout'' and ''stderr'' outputs are merged and redirected to a file (<code>job_script.sh.o82</code>)

We have specified two parameters <code>Hello</code> and <code>World</code>. The output of the script will be located in your <code>$HOME</code> directory after the script is executed. It will be merged with ''stderr'' and it should look like this:

AIC:ubuntu 18.04: SGE 8.1.9 configured...
This is just a test.
printing parameter1: Hello
prinitng parameter2: World
======= EPILOG: Tue Jun 4 12:41:07 CEST 2019
== Limits:
== Usage: cpu=00:00:00, mem=0.00000 GB s, io=0.00000 GB, vmem=N/A, maxvmem=N/A
== Duration: 00:00:00 (0 s)
== Server name: cpu-node13

Main Page

2019-06-04T14:02:58Z

Admin: /* Access */

== Welcome to AIC ==

AIC (Artificial Intelligence Cluster) is a computational grid with '''96 CPU cores''', a total of '''1536 GB RAM''' and '''16 Nvidia GTX 1080 GPUs'''. It was built on top of [https://arc.liv.ac.uk/trac/SGE SGE] scheduling system. MFF students of Bc. and Mgr. degrees can use it to run their experiments and learn the proper ways of grid computing in the process.

=== Access ===
AIC is dedicated to UFAL students who will get an account if requested by authorized lector.

=== Basic HOWTO ===

Following HOWTO is meant to provide only a simplified overview of the cluster usage. It is strongly recommended to read some further documentation before running some serious experiments.

Suppose we want to run some computations described by a script called <code>job_script.sh</code>:

#!/bin/bash
echo "This is just a test."
echo "printing parameter1: $1"
echo "prinitng parameter2: $2"

We need to ''submit'' the job to the grid which is done by logging on the submit host <code>aic.ufal.mff.cuni.cz</code> and issuing the command <code>qsub -cwd -j y job_script.sh Hello World</code>.

This will enqueue our ''job'' to the default ''queue'' which is <code>cpu.q@*</code>. The scheduler decides which particular machine in the specified queue has ''resources'' needed to run the job. Typically we will see a message which tells us the ID of our job (82 in this example):

Your job 82 ("job_script.sh") has been submitted

The basic options used in this example are:
* <code>-cwd</code> - the script is executed in the current directory (the default is your <code>$HOME</code>)
* <code>-j y</code> - ''stdout'' and ''stderr'' outputs are merged and redirected to a file (<code>job_script.sh.o82</code>)

We have specified two parameters <code>Hello</code> and <code>World</code>. The output of the script will be located in your <code>$HOME</code> directory after the script is executed. It will be merged with ''stderr'' and it should look like this:

AIC:ubuntu 18.04: SGE 8.1.9 configured...
This is just a test.
printing parameter1: Hello
prinitng parameter2: World
======= EPILOG: Tue Jun 4 12:41:07 CEST 2019
== Limits:
== Usage: cpu=00:00:00, mem=0.00000 GB s, io=0.00000 GB, vmem=N/A, maxvmem=N/A
== Duration: 00:00:00 (0 s)
== Server name: cpu-node13

MediaWiki:Sidebar

2019-06-04T13:52:35Z

Admin:

Submitting CPU Jobs

2019-06-04T13:51:09Z

Admin: Created page with "== Documentation =="

== Documentation ==

MediaWiki:Sidebar

2019-06-04T13:50:26Z

Admin: test adding sidebar item

File:Logo ufal 110.png

2019-05-30T12:24:33Z

Admin: