What a pipeline actually is, for people who have never written one

If you work at the bench, you already know what a pipeline is. You call it a protocol. A protocol says what to do, in what order, with what reagents and at what settings, so that anyone following it gets a comparable result. A pipeline says the same about data.

What is a bioinformatics pipeline?

A bioinformatics pipeline is a series of analysis steps applied to sequencing or other experimental data. Each step takes something in, does one job to it, and hands the result to the next step. An RNA-seq pipeline, for example, trims the raw reads, counts how many reads came from each gene, tests which genes changed between conditions, and draws the figures.

Steps, inputs, outputs and parameters

A step is usually one established tool: a read trimmer, an aligner, a statistical test. It has three kinds of thing attached to it:

• Inputs: the files it needs, such as FASTQ reads or a reference genome.
• Outputs: the files it writes, such as a table of counts or a report.
• Parameters: its settings, such as a minimum read length or a significance threshold. These are the values you would report in a methods section.

Connecting the output of one step to the input of the next is what makes a list of tools into a pipeline. The connections matter: a step that expects a table of counts cannot use a file of aligned reads, in the same way a PCR cannot use a sample that was never extracted.

Why pipelines are usually written as code

Because, for a long time, there was no other practical way to make the steps run in order, on the right files, with the right versions of each tool, on a machine with enough memory. Workflow languages and workflow managers exist to solve exactly that, and they solve it well, for people who write code. For everyone else, they are the reason analysis waits for someone who does.

Why a canvas works

A pipeline is already a diagram. Every methods figure that shows boxes joined by arrows is a pipeline drawn after the fact. Drawing it first, and making the drawing the thing that runs, means the diagram cannot drift from what was actually done: the figure in the paper and the analysis are the same object.

If you can draw your analysis on a whiteboard, you can build it.

That is the idea behind OpenBionode. Each step is an established tool, the connections only allow files that fit, and each parameter sits in a panel with a line saying what it does. To see one built end to end, read your first differential expression analysis or follow the protocols.