How to run Snakemake on ScienceCluster¶

This guide describes how to run Snakemake version 8.0 or later on ScienceCluster. The old guide for versions prior to 8.0 is available at the end of this document.

The guide assumes that you are already familiar with Snakemake. If not, you may want to start with the Snakamake tutorial in the Snakemake official documentation.

Installation¶

The easiest way to create a conda environment for running Snakemake is by using the mamba module. Mamba is a package manager that is fully compatible with conda but performs better on certain tasks such as resolving requirements. In addition to the snakemake package, you would also need to install snakemake-executor-plugin-slurm. We will use an interactive session as mamba may require more resources than available on the login nodes.

srun --pty -c 2 --mem=8G --time=2:00:00 bash -l
module load mamba
mamba create -n snakemake -c conda-forge -c bioconda --strict-channel-priority \
  'snakemake>=8' snakemake-executor-plugin-slurm

You can exit the interactive session at this point.

Snakefile¶

First of all, we create a directory for the pipeline. For the rest of the guide, we assume that we are in that directory.

mkdir $HOME/scratch/snakemake_test
cd $HOME/scratch/snakemake_test

For illustrative purposes, the Snakefile contains two rules. The first rule runs a couple of bash commands then sleeps for a random number of seconds between 1 and 100. The second rule counts the number of characters in the output file that the first rule generates. In addition, both rules would print the information that SLURM has about the submitted jobs. This will help us determine if the resources were allocated correctly.

rule all:
   input:
      expand("data/small_job_{iteration}.txt", iteration=[1,2,3])

rule big_job:
   output:
      "data/big_job_{iteration}.txt"
   threads: 3
   resources:
      cpus_per_task = 3,
      mem_mb = 2000,
      runtime = "1h"
   shell:
      """
      echo "$(date '+%Y-%m-%d %H:%M:%S') Starting big job {wildcards.iteration}"
      date '+%Y-%m-%d %H:%M:%S' > {output}
      hostname >> {output}
      echo "Host has $(ps -e | wc -l) processes running" >> {output}
      echo "Rule has a limit of {threads} threads"
      echo "Snakemake requested {resources.cpus_per_task} cpus_per_task"
      echo "Snakemake requested {resources.mem_mb} MB of memory"
      scontrol show jobid=$SLURM_JOBID >> {output}
      delay=$((1 + $RANDOM % 100))
      echo "Will sleep for $delay seconds" >> {output}
      sleep $delay
      date '+%Y-%m-%d %H:%M:%S' >> {output}
      echo "$(date '+%Y-%m-%d %H:%M:%S') Finished big job {wildcards.iteration}"
      """

rule small_job:
   input:
      "data/big_job_{iteration}.txt"
   output:
      "data/small_job_{iteration}.txt"
   threads: 1
   resources:
      cpus_per_task = 1,
      mem_mb = 1000,
      runtime = 10
   shell:
      """
      echo "$(date '+%Y-%m-%d %H:%M:%S') Starting small job {wildcards.iteration}"
      date '+%Y-%m-%d %H:%M:%S' > {output}
      hostname >> {output}
      wc -c {input} >> {output}
      echo "Rule has a limit of {threads} threads"
      echo "Snakemake requested {resources.cpus_per_task} cpus_per_task"
      echo "Snakemake requested {resources.mem_mb} MB of memory"
      scontrol show jobid=$SLURM_JOBID >> {output}
      date '+%Y-%m-%d %H:%M:%S' >> {output}
      echo "$(date '+%Y-%m-%d %H:%M:%S') Finished small job {wildcards.iteration}"
      """

Profiles¶

It is a best practice to make workflows portable. To achieve this, it is recommended to keep the runtime environment configuration, including cluster settings, outside of Snakefile. This allows users to easily apply their own configurations when running the workflow. For that purpose, Snakemake developers implemented configuration profiles.

There are two types of profiles. Global profiles can be used for storing general configurations that are applicable across multiple workflows. These are stored in $HOME/.config/snakemake and the desired profile has to be specified explicitly. Workflow specific profile can be stored in the local profiles directory. If you name it default it will be loaded automatically.

Parameters have the following precedence.

• Global profile
• Snakefile
• Workflow profile
• Command line

For example, a parameter value specified in a workflow profile overrides the parameter values specified in Snakefile and the global profile.

Global profile¶

In most cases, you will be using the SLURM executor plugin to run workflows on ScienceCluster. Thus, it would make sense to create a global profile for SLURM. First, you need to create a directory for the profile.

mkdir -p $HOME/.config/snakemake/slurm

Then you can create the configuration file $HOME/.config/snakemake/slurm/config.yaml with the following contents. Please make sure to set slurm_account accordingly.

executor: slurm
jobs: 400
default-resources:
  slurm_account: mylab.mydep.uzh
  cpus_per_task: 2
  mem_mb: 3700
  runtime: 1h

The default-resources section specifies the default job parameters to be used when the corresponding parameter is not specified for a job elsewhere explicitly.

• slurm_account: the SLURM account (project) under which you would like to run the jobs. You can list your accounts with sacctmgr show assoc -nP format=account user=$USER. If you have access only to a single project, the parameter is optional. However, if you do not specify it, Snakemake will print a warning saying that it had to guess the account. An example for multiple projects is provided in Advanced Topics.
• mem_mb: the amount of memory in megabytes.
• mem_mb_per_cpu: the amount of memory in megabytes per CPU; mutually exclusive with mem_mb.
• runtime: without any suffixes, the runtime is assumed to be in minutes. Alternatively, you can use h for hours or d for days.

The jobs parameter sets the maximum number of jobs that Snakemake can maintain in the running and pending state at any given time. Once the limit is reached, Snakemake will not submit any additional jobs until some of the running and pending jobs finish.

Resources¶

While the runtime environment should be kept outside of Snakefile, it might still be helpful to include resource requirements with your workflow. If somebody runs your workflow in a different environment, they could start with the provided resource requirements and adjust them for their environment.

Depending on your preferences, there are two main approaches. If you prefer a more transparent approach with less typing, you could specify the resources in Snakefile. This is the approach we take in this tutorial. If you prefer to keep your Snakefile cleaner and slimmer, you could define the resources in the workflow profile and provide it with your workflow.

In most cases, you would only need to include cpus_per_task, mem_mb, and runtime. These were described in the Global Profile section. Please note that in Snakefile you define them as python variables, so you have to use equal signs for assignment and quote the strings, e.g. runtime = "1h".

GPU resources are specific to the environment, so it is better to avoid specifying them in Snakefile. Handling of GPU resources is described later in the guide.

It is possible to reference resources in the rule's shell and run sections the same way your reference input and wildcards. This is useful when the application that you run accepts memory or thread parameters. For example, you can set the memory limit for java as follows.

shell:
   """
   java -Xmx{resources.mem_mb}m AppLauncher
   """

With threads, it is more complicated. You might have noticed that Snakefile above specifies both threads and cpus_per_task. The former is independent from the environment and it is always available even when not explicitly set for a rule. It could be determined based on how well the task can be parallelised. Snakemake uses threads to determine how many jobs to execute when you run a pipeline locally without submitting the tasks to a scheduler. (You should never run workflow steps on a login node, by the way.)

On the other hand, cpus_per_task is specific to SLURM. It may be used by some other executors but there is no guaranty that all other executors have it. Hence, you should avoid referencing cpus_per_task in the run and shell sections and use threads instead. Please note that cpus_per_task is what Snakemake requests from SLURM. So, if cpus_per_task is lower than threads and you referenced threads in the run or shell section, your application may experience resource contention trying to run more threads than available. This typically leads to slower execution times.

Submission¶

The pipeline output will go into the data directory. In addition, we will send the SLURM output from the main job into the log directory. So, we should create those directories before launching the pipeline.

mkdir data log

Command line¶

To submit the pipeline from the command line, you would need to load mamba, activate your environment, and run snakemake with the profile parameter.

module load mamba
source activate snakemake
sbatch --cpus-per-task=1 --mem=2000 --time=1:00:00 --output=log/main_%j snakemake --profile slurm

Here, we request 1 CPU, 3800 MB of memory and 1 hour for the main Snakemake job. This job will submit all the other jobs, monitor their progress, and submit the dependent jobs as needed. The output from this job will be sent to log/main_<job_id>.

You may find that Snakemake complained about running in "a SLURM job context". This is because it expects to run on fat login nodes with unrestricted resources and potentially the environment where job submission from compute nodes is blocked. In our environment, the login nodes are thin and the resource usage on login nodes is restricted. This may cause problems if Snakemake runs out of the available resources. In particular, this is likely when it needs to build containers, create conda environments, or generate large DAGs (job execution plans). For that reason, we recommend to run Snakemake as a job if your pipeline contains a large number of jobs or if your rules require separate containers or conda environments.

You can track the progress by watching log/main_<job_id> file or with the squeue -u $USER command. After your job is complete, you can check the output files in the data directory to see if the resources were requested correctly. In particular, TimeLimit should match runtime and AllocatedTRES has the matching mem and cpus values.

Log files for individual rules can be found in .snakemake/slurm_logs.

Batch script¶

You can simplify the submission by placing the submission commands into a batch script. For example, we can create a file named run.slurm with the following content.

#!/bin/bash -l
#SBATCH --cpus-per-task=1
#SBATCH --mem=2000
#SBATCH --time=1:00:00
#SBATCH --output=log/main_%j

module load mamba
source activate snakemake

snakemake --profile slurm "$@"

All sbatch parameters are now specified at the top of the batch script. The script loads the mamba module and activates the conda environment that contains Snakemake. Finally, there is a snakemake call with "$@", which expands to all arguments passed to the script. This is useful if you need to specify additional parameters but do not want to include them in the script. For example, executing sbatch run.slurm --keepgoing --rerun-incomplete will pass --keepgoing --rerun-incomplete to Snakemake.

Now, we can submit the whole pipeline with a simple command: sbatch run.slurm. As before, you can see the status by running squeue -u $USER. The pipeline should finish fairly quickly, in a couple of minutes. However, a typical pipeline make take days to complete, so you would need to adjust the --time parameter according to your estimates.

Advanced topics¶

GPU resources¶

For requesting a GPU, you can use the slurm_extra resource parameter. As with other SLURM-specific parameters, it is better to avoid specifying this parameter in Snakefile to keep the pipeline portable. It could be specified either on the command line or in a workflow profile.

On the command line, this is achieved with the --set-resources parameter. For example, the following command requests 1 GPU for the rule named gpu_job.

sbatch --cpus-per-task=1 --mem=2000 --time=1:00:00 --output=log/main_%j snakemake \
   --set-resources "gpu_job:slurm_extra='--gpus=1'"

You can also add a specific GPU type, e.g. --set-resources "gpu_job:slurm_extra='--gpus=V100:1'".

In a workflow profile, this request would look as follows.

# ...
set-resources:
  gpu_job:
    slurm_extra: "'--gpus=V100:1'"
# ...

To try it out, we can add a simple GPU job to Snakefile that we used above. In addition to the basic job information, the GPU job will also print the information about the allocated GPU.

rule all:
   input:
      expand("data/small_job_{iteration}.txt", iteration=[1,2,3]), "data/gpu_job.txt"

[...]

rule gpu_job:
  output:
    "data/gpu_job.txt"
   threads: 1
   resources:
      cpus_per_task = 2,
      mem_mb = 1500,
      runtime = 15
  shell:
      """
      echo "$(date '+%Y-%m-%d %H:%M:%S') Starting GPU job"
      date '+%Y-%m-%d %H:%M:%S' > {output}
      hostname >> {output}
      echo "Rule has a limit of {threads} threads"
      echo "Snakemake requested {resources.cpus_per_task} cpus_per_task"
      echo "Snakemake requested {resources.mem_mb} MB of memory"
      nvidia-smi >> {output}
      scontrol show jobid=$SLURM_JOBID >> {output}
      date '+%Y-%m-%d %H:%M:%S' >> {output}
      echo "$(date '+%Y-%m-%d %H:%M:%S') Finished GPU job"
      """

We also need to create a default workflow profile and specify GPU resources for the gpu_job rule.

set-resources:
  gpu_job:
    slurm_extra: "'--gpus=1'"

If you ran the pipeline previously and would like to rerun it completely, you could delete the old output files with rm data/*.txt.

sbatch --cpus-per-task=1 --mem=2000 --time=1:00:00 --output=log/main_%j snakemake

If successful, the output of the GPU job in data/gpu_job.txt should contain information about the GPU allocated for the job.

Multiple projects¶

If you have access to ScienceCluster through multiple projects, you can create a separate global profile for each one. For example, if you are associated with projects "proj1.mylab.mydep.uzh" and "proj2.mylab.mydep.uzh", you could create directories $HOME/.config/snakemake/proj1 and $HOME/.config/snakemake/proj2. The first directory will contain config.yaml with

executor: slurm
jobs: 400
default-resources:
  slurm_account: proj1.mylab.mydep.uzh
  mem_mb: 3700
  runtime: 1h
  cpus_per_task: 2

The second directory will have respectively

executor: slurm
jobs: 400
default-resources:
  slurm_account: proj2.mylab.mydep.uzh
  mem_mb: 3700
  runtime: 1h
  cpus_per_task: 2

If you run your workflow from the command line, you just specify proj1 or proj2 as the value for the --profile parameter. If you are using a batch script, you could set the profile name in the script correspondingly. Alternatively, you could remove the --profile parameter from the batch script and specify it when submitting the batch, e.g. sbatch run.slurm --profile proj2.

Workflow profiles¶

To keep Snakefile clean and slim, you can choose to specify resources in a workflow profile rather than in Snakefile. It is also useful if you need to request GPU resources for your jobs.

Workflow profiles are typically stored in the profiles directory of your Snakemake workflow. As with the global profiles, each workflow profile has its own directory containing config.yaml file. We will use profiles/default, so that Snakemake loaded it automatically. If you want to specify the workflow profile manually, you should use a different name, e.g. profiles/wfprofile1, and provide it as the value for the --workflow-profile parameter, e.g. snakemake --workflow-profile=wfprofile1.

mkdir -p profiles/default

Then, you could create profiles/default/config.yaml with the following content. All values match what was used in Snakefile above.

set-resources:
  big_job:
    cpus_per_task: 3
    mem_mb: 2000
    runtime: "1h"
  small_job:
    cpus_per_task: 1
    mem_mb: 1000
    runtime: 10

You could also specify the default resources here. However, the workflow profile overrides the global profile parameters at the top level. For example, if you want to specify a different runtime under default-resources in your workflow profile, you have to specify all other parameters from the default-resources section as well because they will not be inherited from the global profile.

If you need to request GPU resources for some of your rules, you do not need to repeat all other parameters, e.g. the following will work for the GPU workflow example.

set-resources:
  gpu_job:
    slurm_extra: "'--gpus=1'"

You could also move all your resource definitions into the workflow profile as well.

set-resources:
  big_job:
    cpus_per_task: 3
    mem_mb: 2000
    runtime: "1h"
  small_job:
    cpus_per_task: 1
    mem_mb: 1000
    runtime: 10
  gpu_job:
    cpus_per_task: 2
    mem_mb: 1500
    runtime: 15
    slurm_extra: "'--gpus=1'"

Prior Snakemake versions¶

Below is the old guide for Snakemake versions from 5.0 up to but not including 8.0.

This guide describes how to configure Snakemake so that it submits each job automatically to the cluster queue. Snakemake documentation provides basic guidelines for cluster execution. The approach based on cluster config has been deprecated in favour of resources and profiles. Although a slurm profile is available, it has certain limitations, in particular the low job submission rate, which can make it impractical for large pipelines. There is also a simpler implementation that does not have the job submission rate problem. Below we provide an even simpler implementation. Even though it does not provide the same functionality as the official profile, it does not have any known limitations. It is loosely (but not entirely) based on the (archived) blog post by Sichong Peng.

You may also find useful the fully implemented sample pipeline that illustrates how to run GATK on ScienceCluster.

The guide assumes that you are familiar with Snakemake and have already installed it into your user space. If you have not installed it yet, the easiest way would be to use the mamba or anaconda3 module and to install snakemake-minimal or snakemake packages into a new environment. In most cases, snakemake-minimal is sufficient especially if you are relatively new to Snakemake.

Mamba is a package manager that is fully compatible with conda but performs better on certain tasks such as resolving requirements. For those reasons, we will use mamba in this example. To load the module, run module load mamba.

Conda environment¶

We will use the following minimal conda environment.

name: snakemake_cluster
channels:
   - conda-forge
   - bioconda
dependencies:
   - python
   - snakemake-minimal>=5,<8

You can easily recreate it with mamba by placing the definition to a yml file, e.g. snakemake_cluster.yml, and running mamba env create -f snakemake_cluster.yml.

Cluster profile config¶

The profile configuration file defines the command used to submit jobs as well as default resource specifications for jobs. We will store it in env/slurm/config.yaml. You can create the directory structure with mkdir -p env/slurm where slurm is the profile's name. Now, you can create the file env/slurm/config.yaml and place the cluster parameters there. The advantage of this approach as opposed to storing the profile in the default location (see the Snakemake manual for details) is that the slurm profile will be added to your git repository.

jobs: 100
cluster: >-
    sbatch
    --ntasks 1
    --cpus-per-task {resources.threads}
    --mem {resources.mem_mb}
    --time {resources.time}
    -o log/jobs/{rule}_{wildcards}_%j.out
    -e log/jobs/{rule}_{wildcards}_%j.err
default-resources:
    - threads=1
    - mem_mb=3850
    - time="1:00:00"

The first parameter, jobs, defines the maximum number of jobs snakemake would keep running and pending simultaneously. The number of jobs that a user can submit depends on the qos (Quality of Service), which in turn depends on the job's requested runtime. You can see the current limits by running sacctmgr show qos format=name,priority,maxwall,maxsubmit,maxtrespu%30,maxjobspu, At the time of writing, the following limits are enforced (subject to change).

Priority is the additional priority a job gets. MaxWall shows the maximum runtime; MaxSubmit is the number of jobs you can submit at a time. MaxTRESPU is the highest amount of special resources a user can reserve. MaxJobsPU is the maximum number of jobs that can run in parallel. Empty value in the MaxJobsPU column means that the value is the same as for MaxSubmit.

The second parameter, cluster, specifies the command to be used for job submission. The only change you may possibly need here is the location of the output and error files. In the definition above, the file names consist of the rule name, wildcards and slurm's job id. You have to create the log directories in advance or the jobs will fail. In particular, you will need to run mkdir -p log/jobs in the same directory where your Snakefile resides.

The last parameter, default-resources, specifies the default resources a job will be allocated if the respective values are not defined in the rules within Snakefile. The values are passed directly to slurm, so they should have the same format as the corresponding slurm parameters. For example, time can be specified as 24:00:00 (24 hours) or as 1-00:00:00 (1 day).

Snakefile¶

For illustrative purposes, the Snakefile contains two rules. The first runs a couple of bash commands then sleeps for a random number of seconds between 1 and 100. The second counts the number of characters in the output file that the first rule generates. In addition, both rules would print the information that slurm has about the submitted jobs. This will help us determine if the resources were allocated correctly.

rule all:
   input:
      expand("data/small_job_{iteration}.txt", iteration=[1,2,3])

rule big_job:
   output:
      "data/big_job_{iteration}.txt"
   resources:
      mem_mb=7700,
      time="2:00:00"
   shell:
      """
      date '+%Y-%m-%d %H:%M:%S' > {output}
      hostname >> {output}
      echo "Host has $(ps -e | wc -l) processes running" >> {output}
      scontrol show jobid=$SLURM_JOBID >> {output}
      delay=$((1 + $RANDOM % 100))
      echo "Will sleep for $delay seconds" >> {output}
      sleep $delay
      date '+%Y-%m-%d %H:%M:%S' >> {output}
      """

rule small_job:
   input:
      "data/big_job_{iteration}.txt"
   output:
      "data/small_job_{iteration}.txt"
   shell:
      """
      date '+%Y-%m-%d %H:%M:%S' > {output}
      hostname >> {output}
      wc -c {input} >> {output}
      scontrol show jobid=$SLURM_JOBID >> {output}
      date '+%Y-%m-%d %H:%M:%S' >> {output}
      """

Since the big_job rule is expected to run longer and require more memory, we override the default mem_mb and time parameters in the resources section of that rule. All other resource-related parameters will be taken from the profile that we have defined previously. Respectively, since the small_job rule lacks the resources section, Snakemake will obtain all parameters from the profile.

Batch script¶

For running the pipeline on the cluster, we are going to create a script called run.slurm with the following content.

#!/usr/bin/env bash
#SBATCH --cpus-per-task=1
#SBATCH --mem=3850
#SBATCH --time=1-00:00:00
#SBATCH --output=log/main_%j

module load mamba/4.14.0
eval "$(conda shell.bash hook)"
conda activate snakemake_cluster

snakemake --profile ./env/slurm "$@"

The first line indicates that the script should be processed with bash when run. After that, there are several slurm parameters for the batch script. It will be using 1 thread and 3850 MB RAM for at most one day. The output will be saved to the file named log/main_%j where %j is the slurm job id that slurm will fill in automatically.

Next, we have three lines that initialise conda and activate the environment that we created for this project. At the moment, it is more convenient to use conda for activation. The command is included with mamba.

Finally, there is a snakemake call. The first parameter specifies the environment relative to the location of Snakefile. The last argument to the snakemake command is "$@", which expands to all arguments passed to the script. This is useful if you need to specify additional parameters but do not want to include them in the script. For example, sbatch run.slurm --keepgoing --rerun-incomplete.

Before running the pipeline, we need to create log/jobs directory where slurm will be writing the output as well as the data directory for the output files.

mkdir -p log/jobs data

Now, we can submit the whole pipeline with sbatch run.slurm. You can see the status by running squeue -u $USER. The pipeline should finish fairly quickly, in a couple of minutes. However, a typical pipeline make take days to complete.