4.2 Build process

4.2.1 Examples

There are different ways to generate Singularity image files. The most common way is so far thanks to existing Docker images already present in public registries.

4.2.1.1 Through registries

4.2.1.1.1 Docker Hub

https://hub.docker.com/r/biocontainers/fastqc

    singularity build fastqc-0.11.9_cv7.sif docker://biocontainers/fastqc:v0.11.9_cv7
4.2.1.1.2 Biocontainers
4.2.1.1.2.1 Via quay.io

https://quay.io/repository/biocontainers/fastqc

    singularity build fastqc-0.11.9.sif docker://quay.io/biocontainers/fastqc:0.11.9--0
4.2.1.1.2.2 Via Galaxy project prebuilt images
    singularity pull --name fastqc-0.11.9.sif https://depot.galaxyproject.org/singularity/fastqc:0.11.9--0
Galaxy project provides all Bioinformatics software from Biocontainers initiative as Singularity prebuilt images. If download and conversion time of images is an issue for you, this is likely the best option if working in the biomedical field.
4.2.1.1.3 Sylabs registry

Not as popular so far, but it provides fast download of images.

4.2.1.2 Docker Daemon

If you create your own images (as we did during the course) and you don’t want to share them publicly (and you have not access to any private image registry, either), you can convert your locally built Docker images into Singularity image files.

        singularity build fastqc-web-0.11.9.sif docker-daemon://fastqcwww

4.2.2 Sandboxing

Instead of generating an image file, it is actually possible to use a whole directory with its contents. This is handy when specific changes may be needed.

     singularity build --sandbox ./sandbox docker://ubuntu:18.04
     touch sandbox/etc/myetc.conf
     singularity build sandbox.sif ./sandbox
Apart from testing or debugging, as we commented with Docker, we don’t recommend this approach because it makes reproducibility more difficult.

4.2.3 Singularity recipes

Singularity provides its own build system and recipe syntax. Despite it is actually possible to generate images from scratch (known as bootstraping) thanks to these recipes, this is at time of writing far slower than converting from Docker ones. Docker has the advantage of saving every action line as a cached image. That is not happening with Singularity.

When using recipes, it’s mandatory to have administrator permissions (e.g., as beeing root or via sudo).

In any case, it can still be useful to boostrap an image derived from a previously existing one. Below we provide two common approaches:

4.2.3.1 Docker bootstrap

Instead of converting a Docker image into a Singularity one, it’s possible to use one as a base one and modify it by using Singularity recipe syntax.

BootStrap: docker
From: biocontainers/fastqc:v0.11.9_cv7

%runscript
    echo "Welcome to FastQC Image"
    fastqc --version

%post
    echo "Image built"
    sudo singularity build fastqc.sif docker.singularity
The command %runscript would be equivalent to ENTRYPOINT/CMD in Docker. It is only triggered when using singularity run. This is useful if you want to hide from the user the complexity of a command-line path or an included custom script.

4.2.3.2 Debian bootstrap

Alternatively, we can build the whole image from Debian/Ubuntu distribution. This will normally take a while.

BootStrap: debootstrap
OSVersion: bionic
MirrorURL:  http://fr.archive.ubuntu.com/ubuntu/
Include: build-essential curl python python-dev openjdk-11-jdk bzip2 zip unzip

%runscript
    echo "Welcome to my Singularity Image"
    fastqc --version
    multiqc --version
    bowtie --version

%post

    FASTQC_VERSION=0.11.9
    MULTIQC_VERSION=1.9
    BOWTIE_VERSION=1.3.0

    cd /usr/local; curl -k -L https://www.bioinformatics.babraham.ac.uk/projects/fastqc/fastqc_v${FASTQC_VERSION}.zip > fastqc.zip
    cd /usr/local; unzip fastqc.zip; rm fastqc.zip; chmod 775 FastQC/fastqc; ln -s /usr/local/FastQC/fastqc /usr/local/bin/fastqc

    cd /usr/local; curl --fail --silent --show-error --location --remote-name https://github.com/BenLangmead/bowtie/releases/download/v$BOWTIE_VERSION/bowtie-${BOWTIE_VERSION}-linux-x86_64.zip
    cd /usr/local; unzip -d /usr/local bowtie-${BOWTIE_VERSION}-linux-x86_64.zip
    cd /usr/local; rm bowtie-${BOWTIE_VERSION}-linux-x86_64.zip
    cd /usr/local/bin; ln -s ../bowtie-${BOWTIE_VERSION}-linux-x86_64/bowtie* .

    curl --fail --silent --show-error --location --remote-name  https://bootstrap.pypa.io/pip/2.7/get-pip.py
    python get-pip.py

    pip install numpy matplotlib
    pip install -I multiqc==${MULTIQC_VERSION}

    echo "Biocore image built"

%labels
    Maintainer Biocorecrg
Version 0.1.0
    sudo singularity build fastqc-multi-bowtie.sif debootstrap.singularity
The available deboostrap recipes depend on the version you have installed in the system. Check at: /usr/share/debootstrap/scripts/ for more details. For Debian, instead of Ubuntu, you can replace mirrorURL with https://ftp.debian.org/debian/ More details at: https://sylabs.io/guides/3.7/user-guide/appendix.html#build-debootstrap
It’s possible to sign cryptographically your images, so third parties can verify they are coming from their actual authors. This implies some matters beyond the scope of this course, but you have some details if interested at: https://sylabs.io/guides/3.7/user-guide/signNverify.html

4.2.4 Remote building

This allows us to build a Singularity image without using our own computer. This is convenient if your machine has not many resources or if you don’t have administrator privileges.

    singularity remote login
  • In order to use it our username and passwords we generate a token at https://cloud.sylabs.io/auth/tokens and copy its contents to a file (e.g., singularity.token). Then we use in the command-line below:
    singularity remote login --tokenfile singularity.token
Output
(base) [ ~]$ singularity remote login –tokenfile singularity.token INFO: Access Token Verified! INFO: Token stored in /home/ec2-user/.singularity/remote.yaml
For safety reasons, remove the token from Cloud Sylabs interface after the course. Follow the same philosophy for similar circumstances. Keep an only token for a specific environment or action.
  • Once we are authenticated with our access token, we can build an image:
    singularity build --remote focal.sif docker://ubuntu:focal
  • Since we are already authenticated, we can take advantage to push a Singularity image in our account, similarly to what we did with Docker. We use -U so we do not sign the image.
    singularity push -U focal.sif library://toniher/test/focal.sif:latest
IMPORTANT: Unless it is not defined in advance as private from Cloud Sylabs interface, pushed image will be public and there does not seem an easy way to turn it private. As an interesting feature, you can actually sign your images after you pushed them from Cloud Sylabs web interface by using a key associated to your account (which can also be created there).
More details at https://sylabs.io/guides/3.7/user-guide/cloud_library.html