Containers¶

Singularity¶

Singularity is free and open source container solution that has been created out of necessity, for scientific and application driven workloads. Singularity defines the "Mobility of Compute" concept as the ability to organize, create and maintain a workflow and be confident that the workflow can be executed on different hosts, operating systems (as long as it is Linux) and service providers. Being able to contain the entire software stack, from data files to library stack, and portably move it from system to system is true mobility.

Below are clarifications on various aspects of using Singularity.

Bound directories¶

The following directories are automatically bound and available from within each container:

• /home
• /scratch
• /data
• /shares
• /sctmp

This is achieved by setting SINGULARITY_BINDPATH environmental variable for all users. The current setting is sufficient for the majority of cases.

Default User¶

The default user within each container is the same the one you have on the cluster (i.e., your UZH Active Directory Account).

Building singularity containers¶

Building a container may entail actions that are not permitted on the login or compute nodes. However, this typically applies to containers built from custom definition files. If you are using an image from docker hub, you should be able to build it on ScienceCluster. We recommend to use an interactive session as building often requires more memory than available to users on the login nodes.

For example, you can create a rocker container with R and tidyverse packages as follows.

srun --pty -n 1 -c 2 --time=00:30:00 --mem=7G bash -l
module load singularityce
singularity build --sandbox /data/$USER/tidyverse docker://rocker/tidyverse

The first command opens an interactive session. The second loads the singularity module. The final command builds the sandboxed container in /data/$USER/tidyverse. Please note that the container directory tidyverse will be in your data directory. If you save containers in your home directory, you will exceed your quota very quickly.

The --sandbox flag means that the container will be built in the form of a directory with files rather than a single file. The sandboxed containers should be preferred over the "single file", or more precisely unexpanded, containers such as sif files. Due to security reasons, singularity has to expand (sandbox) all unexpanded containers and this process may be quite slow. So, if you run multiple jobs using an unexpanded container, singularity will have to create a sandbox version for each job. This would not only make the jobs to run slower but also put a considerable strain on the cluster file systems.

If you need a specific version (tag) rather than the latest one, you can add it after the column.

singularity build --sandbox /data/$USER/tidyverse docker://rocker/tidyverse:4.3.1

If the building process fails due to insufficient permissions or if you need to use a custom definition file, you would have to build the container elsewhere, e.g. on a ScienceCloud node. In such cases, it is reasonable to build an unexpanded container, transfer it to the cluster, and sandbox it on the cluster. Suppose you built a container named matrix.sif and uploaded it to /data/$USER. Then you can sandbox it as follows.

srun --pty -n 1 -c 2 --time=00:30:00 --mem=7G bash -l
module load singularityce
singularity build --sandbox /data/$USER/matrix /data/$USER/matrix.sif

You can remove matrix.sif after you expand it and ensure that the sandboxed container works. One way to test it might be with the shell command.

singularity shell -u /data/$USER/matrix

This command starts a shell inside the container.

Running jobs with singularity containers¶

Submitting jobs using containers is done in the very same way as for the normal jobs. An example of a sbatch script line calling singularity:

#!/bin/bash
#SBATCH --time=00:30:00
#SBATCH --mem=7G
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
singularity exec -u <CONTAINER> <COMMAND>

Alternatively if you've already edited the /singularity script in your container you can directly run the container, which by default will execute that script.

#!/bin/bash
#SBATCH --time=01:00:00
#SBATCH --mem=7G
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
singularity run -u <CONTAINER>

Directory binding¶

You can make additional directories available in the container using the --bind option. For example, in case you want to have your appX_input directory available as /input within the container, you should follow this example:

singularity exec -u --bind ./appX_input:/input <CONTAINER> <COMMAND>

Running MPI Jobs¶

Out of the box compatibility is reported for OpenMPI (v2.1) and Singularity. An example of calling an MPI enabled applications is as follows:

#SBATCH -n 32
#SBATCH -N 2
#SBATCH --time=00:10:00
#SBATCH --mem=15G
singularity exec -u mpi-hello-world ./mpi_hello_world

Running GPU-accelerated jobs¶

Singularity supports GPUs. An example of calling a GPU enabled application is as follows:

#!/bin/bash
#SBATCH --time=00:10:00
#SBATCH --gpus=2
#SBATCH --mem=3000
singularity exec -u --nv tensorflow-gpu python ./tensorflow_app.py

Running Tensorflow jobs in a Singularity container¶

You can build a container from an officially supported Tensorflow image as follows.

srun --pty -n 1 -c 2 --time=00:30:00 --gpus=1 --mem=7G bash -l
module load singularityce
singularity build --sandbox /data/$USER/tf_gpu docker://tensorflow/tensorflow:latest-gpu

You can test it by starting a shell session in the container and running nvidia-smi as well as the Tensorflow GPU validation commands.

singularity shell -u --nv /data/$USER/tf_gpu
nvidia-smi
python -c 'import tensorflow as tf; \
   print("Built with CUDA:", tf.test.is_built_with_cuda()); \
   print("Num GPUs Available:", len(tf.config.list_physical_devices("GPU"))); \
   print("TF version:", tf.__version__)'

Additional resources¶

If you are new to containers, and particularly to Singularity, you find more information here.