Run Snakemake on ScienceCluster¶

This guide describes how to run Snakemake version 8.0 or later on ScienceCluster.

The guide assumes that you are already familiar with Snakemake. If not, you may want to start with the Snakemake tutorial in the Snakemake official documentation.

Installation¶

The easiest way to create a conda or mamba environment for running Snakemake is to load the miniforge3 module. Mamba is a package manager that is fully compatible with conda but performs better on certain tasks such as resolving requirements. In addition to the snakemake package, you would also need to install snakemake-executor-plugin-slurm. We will use an interactive session as mamba may require more resources than available on the login nodes.

srun --pty -c 2 --mem=8G --time=2:00:00 bash -l
module load miniforge3
mamba create -n snakemake -c conda-forge -c bioconda --strict-channel-priority \
  'snakemake>=8' snakemake-executor-plugin-slurm

You can exit the interactive session at this point.

Snakefile¶

First of all, we create a directory for the pipeline. For the rest of the guide, we assume that we are in that directory.

mkdir $HOME/scratch/snakemake_test
cd $HOME/scratch/snakemake_test

For illustrative purposes, the Snakefile contains two rules. The first rule runs a couple of bash commands then sleeps for a random number of seconds between 1 and 100. The second rule counts the number of characters in the output file that the first rule generates. In addition, both rules would print the information that Slurm has about the submitted jobs. This will help us determine if the resources were allocated correctly.

rule all:
   input:
      expand("data/small_job_{iteration}.txt", iteration=[1,2,3])

rule big_job:
   output:
      "data/big_job_{iteration}.txt"
   threads: 3
   resources:
      cpus_per_task = 3,
      mem_mb = 2000,
      runtime = "1h"
   shell:
      """
      echo "$(date '+%Y-%m-%d %H:%M:%S') Starting big job {wildcards.iteration}"
      date '+%Y-%m-%d %H:%M:%S' > {output}
      hostname >> {output}
      echo "Host has $(ps -e | wc -l) processes running" >> {output}
      echo "Rule has a limit of {threads} threads"
      echo "Snakemake requested {resources.cpus_per_task} cpus_per_task"
      echo "Snakemake requested {resources.mem_mb} MB of memory"
      scontrol show jobid=$SLURM_JOBID >> {output}
      delay=$((1 + $RANDOM % 100))
      echo "Will sleep for $delay seconds" >> {output}
      sleep $delay
      date '+%Y-%m-%d %H:%M:%S' >> {output}
      echo "$(date '+%Y-%m-%d %H:%M:%S') Finished big job {wildcards.iteration}"
      """

rule small_job:
   input:
      "data/big_job_{iteration}.txt"
   output:
      "data/small_job_{iteration}.txt"
   threads: 1
   resources:
      cpus_per_task = 1,
      mem_mb = 1000,
      runtime = 10
   shell:
      """
      echo "$(date '+%Y-%m-%d %H:%M:%S') Starting small job {wildcards.iteration}"
      date '+%Y-%m-%d %H:%M:%S' > {output}
      hostname >> {output}
      wc -c {input} >> {output}
      echo "Rule has a limit of {threads} threads"
      echo "Snakemake requested {resources.cpus_per_task} cpus_per_task"
      echo "Snakemake requested {resources.mem_mb} MB of memory"
      scontrol show jobid=$SLURM_JOBID >> {output}
      date '+%Y-%m-%d %H:%M:%S' >> {output}
      echo "$(date '+%Y-%m-%d %H:%M:%S') Finished small job {wildcards.iteration}"
      """

Profiles¶

It is a best practice to make workflows portable. To achieve this, it is recommended to keep the runtime environment configuration, including cluster settings, outside of Snakefile. This allows users to easily apply their own configurations when running the workflow. For that purpose, Snakemake developers implemented configuration profiles.

There are two types of profiles. Global profiles can be used for storing general configurations that are applicable across multiple workflows. These are stored in $HOME/.config/snakemake and the desired profile has to be specified explicitly. Workflow specific profile can be stored in the local profiles directory. If you name it default it will be loaded automatically.

Parameters have the following precedence.

Global profile
Snakefile
Workflow profile
Command line

For example, a parameter value specified in a workflow profile overrides the parameter values specified in Snakefile and the global profile.

Global profile¶

In most cases, you will be using the Slurm executor plugin to run workflows on ScienceCluster. Thus, it would make sense to create a global profile for Slurm. First, you need to create a directory for the profile.

mkdir -p $HOME/.config/snakemake/slurm

Then you can create the configuration file $HOME/.config/snakemake/slurm/config.yaml with the following contents. Please make sure to set slurm_account accordingly.

executor: slurm
jobs: 400
default-resources:
  slurm_account: mylab.mydep.uzh
  cpus_per_task: 2
  mem_mb: 3700
  runtime: 1h

The default-resources section specifies the default job parameters to be used when the corresponding parameter is not specified for a job elsewhere explicitly.

slurm_account: the Slurm account (project) under which you would like to run the jobs. You can list your accounts with sacctmgr show assoc -nP format=account user=$USER. If you have access only to a single project, the parameter is optional. However, if you do not specify it, Snakemake will print a warning saying that it had to guess the account. An example for multiple projects is provided in Advanced Topics.
mem_mb: the amount of memory in megabytes.
mem_mb_per_cpu: the amount of memory in megabytes per CPU; mutually exclusive with mem_mb.
runtime: without any suffixes, the runtime is assumed to be in minutes. Alternatively, you can use h for hours or d for days.

The jobs parameter sets the maximum number of jobs that Snakemake can maintain in the running and pending state at any given time. Once the limit is reached, Snakemake will not submit any additional jobs until some of the running and pending jobs finish.

Resources¶

While the runtime environment should be kept outside of Snakefile, it might still be helpful to include resource requirements with your workflow. If somebody runs your workflow in a different environment, they could start with the provided resource requirements and adjust them for their environment.

Depending on your preferences, there are two main approaches. If you prefer a more transparent approach with less typing, you could specify the resources in Snakefile. This is the approach we take in this tutorial. If you prefer to keep your Snakefile cleaner and slimmer, you could define the resources in the workflow profile and provide it with your workflow.

In most cases, you would only need to include cpus_per_task, mem_mb, and runtime. These were described in the Global Profile section. Please note that in Snakefile you define them as python variables, so you have to use equal signs for assignment and quote the strings, e.g. runtime = "1h".

GPU resources are specific to the environment, so it is better to avoid specifying them in Snakefile. Handling of GPU resources is described later in the guide.

It is possible to reference resources in the rule's shell and run sections the same way your reference input and wildcards. This is useful when the application that you run accepts memory or thread parameters. For example, you can set the memory limit for java as follows.

shell:
   """
   java -Xmx{resources.mem_mb}m AppLauncher
   """

With threads, it is more complicated. You might have noticed that Snakefile above specifies both threads and cpus_per_task. The former is independent from the environment and it is always available even when not explicitly set for a rule. It could be determined based on how well the task can be parallelised. Snakemake uses threads to determine how many jobs to execute when you run a pipeline locally without submitting the tasks to a scheduler. (You should never run workflow steps on a login node, by the way.)

On the other hand, cpus_per_task is specific to Slurm. It may be used by some other executors but there is no guaranty that all other executors have it. Hence, you should avoid referencing cpus_per_task in the run and shell sections and use threads instead. Please note that cpus_per_task is what Snakemake requests from Slurm. So, if cpus_per_task is lower than threads and you referenced threads in the run or shell section, your application may experience resource contention trying to run more threads than available. This typically leads to slower execution times.

Submission¶

The pipeline output will go into the data directory. In addition, we will send the Slurm output from the main job into the log directory. So, we should create those directories before launching the pipeline.

mkdir data log

Command line¶

To submit the pipeline from the command line, you would need to load miniforge3, activate your environment, and run snakemake with the profile parameter.

module load miniforge3
source activate snakemake
sbatch --cpus-per-task=1 --mem=2000 --time=1:00:00 --output=log/main_%j snakemake --profile slurm

Important

Please make sure that the profile name matches the directory name of the profile you have created.

Here, we request 1 CPU, 3800 MB of memory and 1 hour for the main Snakemake job. This job will submit all the other jobs, monitor their progress, and submit the dependent jobs as needed. The output from this job will be sent to log/main_<job_id>.

You may find that Snakemake complained about running in "a Slurm job context". This is because it expects to run on fat login nodes with unrestricted resources and potentially the environment where job submission from compute nodes is blocked. In our environment, the login nodes are thin and the resource usage on login nodes is restricted. This may cause problems if Snakemake runs out of the available resources. In particular, this is likely when it needs to build containers, create conda environments, or generate large DAGs (job execution plans). For that reason, we recommend to run Snakemake as a job if your pipeline contains a large number of jobs or if your rules require separate containers or conda environments.

You can track the progress by watching log/main_<job_id> file or with the squeue -u $USER command. After your job is complete, you can check the output files in the data directory to see if the resources were requested correctly. In particular, TimeLimit should match runtime and AllocatedTRES has the matching mem and cpus values.

Log files for individual rules can be found in .snakemake/slurm_logs.

Batch script¶

You can simplify the submission by placing the submission commands into a batch script. For example, we can create a file named run.slurm with the following content.

#!/bin/bash -l
#SBATCH --cpus-per-task=1
#SBATCH --mem=2000
#SBATCH --time=1:00:00
#SBATCH --output=log/main_%j

module load miniforge3
source activate snakemake

snakemake --profile slurm "$@"

All sbatch parameters are now specified at the top of the batch script. The script loads the miniforge3 module and activates the conda environment that contains Snakemake. Finally, there is a snakemake call with "$@", which expands to all arguments passed to the script. This is useful if you need to specify additional parameters but do not want to include them in the script. For example, executing sbatch run.slurm --keepgoing --rerun-incomplete will pass --keepgoing --rerun-incomplete to Snakemake.

Now, we can submit the whole pipeline with a simple command: sbatch run.slurm. As before, you can see the status by running squeue -u $USER. The pipeline should finish fairly quickly, in a couple of minutes. However, a typical pipeline make take days to complete, so you would need to adjust the --time parameter according to your estimates.

Advanced topics¶

GPU resources¶

For requesting a GPU, you can use the slurm_extra resource parameter. As with other Slurm-specific parameters, it is better to avoid specifying this parameter in Snakefile to keep the pipeline portable. It could be specified either on the command line or in a workflow profile.

On the command line, this is achieved with the --set-resources parameter. For example, the following command requests 1 GPU for the rule named gpu_job.

sbatch --cpus-per-task=1 --mem=2000 --time=1:00:00 --output=log/main_%j snakemake \
   --set-resources "gpu_job:slurm_extra='--gpus=1'"

You can also add a specific GPU type, e.g. --set-resources "gpu_job:slurm_extra='--gpus=V100:1'".

In a workflow profile, this request would look as follows.

# ...
set-resources:
  gpu_job:
    slurm_extra: "'--gpus=V100:1'"
# ...

To try it out, we can add a simple GPU job to Snakefile that we used above. In addition to the basic job information, the GPU job will also print the information about the allocated GPU.

rule all:
   input:
      expand("data/small_job_{iteration}.txt", iteration=[1,2,3]), "data/gpu_job.txt"

[...]

rule gpu_job:
  output:
    "data/gpu_job.txt"
   threads: 1
   resources:
      cpus_per_task = 2,
      mem_mb = 1500,
      runtime = 15
  shell:
      """
      echo "$(date '+%Y-%m-%d %H:%M:%S') Starting GPU job"
      date '+%Y-%m-%d %H:%M:%S' > {output}
      hostname >> {output}
      echo "Rule has a limit of {threads} threads"
      echo "Snakemake requested {resources.cpus_per_task} cpus_per_task"
      echo "Snakemake requested {resources.mem_mb} MB of memory"
      nvidia-smi >> {output}
      scontrol show jobid=$SLURM_JOBID >> {output}
      date '+%Y-%m-%d %H:%M:%S' >> {output}
      echo "$(date '+%Y-%m-%d %H:%M:%S') Finished GPU job"
      """

We also need to create a default workflow profile and specify GPU resources for the gpu_job rule.

set-resources:
  gpu_job:
    slurm_extra: "'--gpus=1'"

If you ran the pipeline previously and would like to rerun it completely, you could delete the old output files with rm data/*.txt.

sbatch --cpus-per-task=1 --mem=2000 --time=1:00:00 --output=log/main_%j snakemake

If successful, the output of the GPU job in data/gpu_job.txt should contain information about the GPU allocated for the job.

Multiple projects¶

If you have access to ScienceCluster through multiple projects, you can create a separate global profile for each one. For example, if you are associated with projects "proj1.mylab.mydep.uzh" and "proj2.mylab.mydep.uzh", you could create directories $HOME/.config/snakemake/proj1 and $HOME/.config/snakemake/proj2. The first directory will contain config.yaml with

executor: slurm
jobs: 400
default-resources:
  slurm_account: proj1.mylab.mydep.uzh
  mem_mb: 3700
  runtime: 1h
  cpus_per_task: 2

The second directory will have respectively

executor: slurm
jobs: 400
default-resources:
  slurm_account: proj2.mylab.mydep.uzh
  mem_mb: 3700
  runtime: 1h
  cpus_per_task: 2

If you run your workflow from the command line, you just specify proj1 or proj2 as the value for the --profile parameter. If you are using a batch script, you could set the profile name in the script correspondingly. Alternatively, you could remove the --profile parameter from the batch script and specify it when submitting the batch, e.g. sbatch run.slurm --profile proj2.

Workflow profiles¶

To keep Snakefile clean and slim, you can choose to specify resources in a workflow profile rather than in Snakefile. It is also useful if you need to request GPU resources for your jobs.

Workflow profiles are typically stored in the profiles directory of your Snakemake workflow. As with the global profiles, each workflow profile has its own directory containing config.yaml file. We will use profiles/default, so that Snakemake loaded it automatically. If you want to specify the workflow profile manually, you should use a different name, e.g. profiles/wfprofile1, and provide it as the value for the --workflow-profile parameter, e.g. snakemake --workflow-profile=wfprofile1.

mkdir -p profiles/default

Then, you could create profiles/default/config.yaml with the following content. All values match what was used in Snakefile above.

set-resources:
  big_job:
    cpus_per_task: 3
    mem_mb: 2000
    runtime: "1h"
  small_job:
    cpus_per_task: 1
    mem_mb: 1000
    runtime: 10

You could also specify the default resources here. However, the workflow profile overrides the global profile parameters at the top level. For example, if you want to specify a different runtime under default-resources in your workflow profile, you have to specify all other parameters from the default-resources section as well because they will not be inherited from the global profile.

If you need to request GPU resources for some of your rules, you do not need to repeat all other parameters, e.g. the following will work for the GPU workflow example.

set-resources:
  gpu_job:
    slurm_extra: "'--gpus=1'"

You could also move all your resource definitions into the workflow profile as well.

set-resources:
  big_job:
    cpus_per_task: 3
    mem_mb: 2000
    runtime: "1h"
  small_job:
    cpus_per_task: 1
    mem_mb: 1000
    runtime: 10
  gpu_job:
    cpus_per_task: 2
    mem_mb: 1500
    runtime: 15
    slurm_extra: "'--gpus=1'"