6.6 Profiles

For deploying a pipeline in a cluster environment or a cloud, we need to add some information in the nextflow.config file.

In particular we need to indicate the kind of executor should be used.

In the Nextflow framework architecture, the executor indicates which is the batch-queuing system to use to submit jobs to the HPC or to the cloud.
The executor is completely abstracted, so you can switch from SGE to SLURM just by changing this parameter in the configuration file.

You can group different classes of configuration or profiles within a single nextflow.config file. In this way that you can indicate at run time which executor and resources to use for a pipeline execution.

Let’s inspect the nextflow.config file in test3 folder. We can look at three different profiles:

• standard
• cluster
• cloud

The first one indicates the resources needed for running the pipeline locally. They are quite small since we have little power and CPUs on the test node.

profiles {
  standard {
     process {
        containerOptions = { workflow.containerEngine == "docker" ? '-u $(id -u):$(id -g)': null}
        executor="local"
        memory='0.6G'
        cpus='1'
        time='6h'

        withLabel: 'twocpus' {
            memory='0.6G'
            cpus='1'
        }
      }
   }

As you can see we indicate explicitly the local executor. So this will be the default when running the pipeline indicating without specifying a profiles.

The second one is cluster:

   cluster {
     process {
        containerOptions = { workflow.containerEngine == "docker" ? '-u $(id -u):$(id -g)': null}
        executor="SGE"
        queue="smallcpus"

        memory='1G'
        cpus='1'
        time='6h'

        withLabel: 'twocpus' {
            queue="bigcpus"
            memory='4G'
            cpus='2'
        }
      }
   }

This indicates that the system uses Sun Grid Engine as job scheduler and that we have different queues for small jobs and more intensive ones.