Courses@CRG 2021 - Containers and Nextflow
1
Welcome and about the course
1.1
Meet the organizers and instructors
1.2
About the course
1.2.1
Outline
1.2.2
Learning objectives
1.2.3
Prerequisite / technical requirements
1.2.4
Dates, time, location
1.3
Program
1.3.1
May 3-4: Containers
1.3.2
May 10-11: Nextflow
2
Introduction to containers
2.1
What are containers ?
2.2
Virtual machines or containers ?
2.2.1
Virtualisation
2.2.2
Containerisation (aka lightweight virtualisation)
2.2.3
Virtual machines vs containers
2.3
History of containers
2.3.1
chroot
2.3.2
Additions in Linux kernel
3
Docker
3.1
Introduction to Docker
3.1.1
What is Docker?
3.1.2
Docker components
3.1.3
Images versus containers
3.1.4
Docker vocabulary
3.2
Using existing images
3.2.1
Getting started
3.2.2
Explore Docker Hub
3.2.3
docker pull: import an image
3.2.4
docker run: run image, i.e. start a container
3.2.5
docker ps: check containers status
3.2.6
docker exec: execute process in a running container
3.2.7
docker stop, start, restart: actions on container
3.2.8
docker rm, docker rmi: clean up!
3.3
Exercise 1 - Docker as a user
3.4
Linux packages
3.4.1
Update and upgrade packages
3.4.2
Search and install packages:
3.5
Docker recipes
3.5.1
Writing recipes and building images
3.5.2
docker tag
3.5.3
Build cache
3.6
Exercise 2 - Docker recipes and build images
3.7
Additional commands
3.8
Volumes
3.9
Ports
3.10
Integrative examples
3.10.1
FASTQC Web Application
3.10.2
Shiny Application
3.11
Publish images
3.11.1
Push an image
3.11.2
Automatic builds
3.12
Exercise 3 - Docker volumes and ports
4
Singularity
4.1
Introduction to Singularity
4.1.1
Singularity architecture
4.1.2
Strengths
4.1.3
Weaknesses
4.2
Build process
4.2.1
Examples
4.2.2
Sandboxing
4.2.3
Singularity recipes
4.2.4
Remote building
4.3
Run and execution process
4.3.1
Singularity shell
4.3.2
Singularity exec
4.3.3
Singularity run
4.3.4
Environment control
4.3.5
Execute from sandboxed images / directories
4.3.6
Execute straight from a registry
4.4
Bind paths (aka volumes)
4.5
Example execution:
4.6
Instances
4.7
Troubleshooting
4.7.1
Fakeroot
4.7.2
Singularity cache directory
4.7.3
Global singularity configuration
4.8
Exercise 4 - Singularity running and building
4.8.1
Blast command-line (1)
4.8.2
Blast command-line (2)
5
Nextflow day 1
5.1
Introduction
5.1.1
What is Nextflow?
5.1.2
What is Nextflow for?
5.1.3
Main advantages
5.2
Workflow structure
5.3
Installation
5.4
Channels and Operators
5.5
EXERCISE 1
5.6
Processes
5.7
Workflow and log
5.7.1
Nextflow log
5.8
EXERCISE 2
5.9
More complex scripts
5.10
Directives
5.11
Resuming your pipeline
5.12
EXERCISE 3
6
Nextflow day 2
6.1
Decoupling resources, parameters and nextflow script
6.1.1
Publishing final results
6.2
Adding a help section for the whole pipeline
6.3
EXERCISE 4
6.4
Using Singularity
6.5
Adding more steps
6.6
Profiles
6.7
Deployment in the AWS cloud
6.8
EXERCISE 5
6.9
Modules and re-usage of the code
6.10
EXERCISE 6
6.11
Reporting and graphical interface
6.12
Share Nextflow pipelines and good practices
BIOINFORMATICS CORE @ CRG
Courses @ CRG: Reproducible research and data analysis with Linux containers and Nextflow pipelines
Part 2
Introduction to containers