10  Shell Scripting

NoteLearning Objectives

After completing this chapter, you will be able to:

  • Create and execute shell scripts
  • Use variables, arrays, and command substitution
  • Write conditional statements and loops
  • Define and call functions
  • Handle command-line arguments
  • Implement proper error handling
  • Build reproducible analysis pipelines

10.1 Why Shell Scripting?

Throughout the previous chapters, you’ve been typing commands interactively. While this is great for exploration, real-world analyses require something more (Sandve et al., 2013). You need reproducibility—a complete record of exactly what you did so you (or others) can repeat the analysis. You need automation so you can run the same analysis on many files without typing each command manually. Scripts provide efficiency by eliminating repetitive typing, and they serve as documentation that explains why you did each step. Finally, scripts enable sharing—letting collaborators reproduce your work with a single command.

Shell scripts transform a series of commands into a reusable, documented program.

10.2 Your First Shell Script

A shell script is simply a text file containing commands. Let’s create one:

#!/bin/bash
# first_script.sh - My first shell script
# Author: Your Name
# Date: 2025-01-15

# Display a greeting
echo "Hello, Bioengineering!"

# Show the current date
echo "Today is: $(date +%Y-%m-%d)"

# List files in current directory
echo "Files in this directory:"
ls -la

The Shebang Line

The first line #!/bin/bash is called a shebang (or hashbang). It tells the operating system which interpreter should execute this script. Always include it; Table 10.1 lists the common ones.

Table 10.1: Common shebang lines
Shebang Interpreter Notes
#!/bin/bash Bash shell Most common for shell scripts
#!/usr/bin/env bash Bash shell Finds bash wherever it is installed; used in the course example scripts
#!/bin/sh POSIX shell More portable across systems
#!/usr/bin/env python3 Python 3 Uses env to find Python
#!/usr/bin/env Rscript R For R scripts

Running Scripts

There are several ways to run a script:

# Method 1: Invoke the interpreter explicitly
$ bash first_script.sh

# Method 2: Make executable and run directly
$ chmod +x first_script.sh
$ ./first_script.sh

The chmod +x command adds execute permission to the file. After that, you can run it directly with ./. The ./ is necessary because your current directory typically isn’t in your system’s PATH (see Section 6.4; permissions are explained in Section 6.2.2).

Tip

Always use chmod +x on your scripts. It signals that the file is meant to be executed and allows direct invocation.

NoteWorked examples

The course’s Example Files page has four heavily commented scripts from Week 2 that use the techniques in this chapter—arguments with defaults and ${1:?usage} checks, command substitution, here-documents, and pipelines inside and between scripts. Reading them alongside this chapter, and then modifying them, is the fastest way to make the material stick.

10.3 Variables

Variables store values for later use. In Bash, variable assignment has strict syntax rules.

Assigning Variables

#!/bin/bash

# Assign variables (NO spaces around the =)
NAME="DNA Analysis"
COUNT=100
INPUT_FILE="sequences.fa"

# Use variables (with $ prefix)
echo "Running $NAME"
echo "Processing $COUNT sequences from $INPUT_FILE"

# Curly braces for clarity
echo "File: ${INPUT_FILE}"
echo "Creating ${INPUT_FILE}.backup"
WarningCommon Mistake

Spaces around = will cause errors!

# WRONG - causes errors
NAME = "value"
COUNT= 100

# RIGHT - no spaces
NAME="value"
COUNT=100

Command Substitution

Capture command output in a variable:

#!/bin/bash

# Old syntax (backticks)
TODAY=`date +%Y-%m-%d`

# Modern syntax (preferred)
TODAY=$(date +%Y-%m-%d)
FILECOUNT=$(ls *.fa | wc -l)
GENOME_SIZE=$(grep -v "^>" genome.fa | tr -d '\n' | wc -c)

echo "Analysis date: $TODAY"
echo "Processing $FILECOUNT FASTA files"
echo "Genome size: $GENOME_SIZE bp"

Arithmetic Operations

Bash arithmetic uses $(( )):

#!/bin/bash

COUNT=10
DOUBLE=$((COUNT * 2))
INCREMENT=$((COUNT + 1))

# Calculate percentage
TOTAL=1000
MATCHED=350
PERCENT=$((MATCHED * 100 / TOTAL))

echo "Doubled: $DOUBLE"
echo "Match rate: $PERCENT%"

For floating-point math, use external tools like bc:

GC_CONTENT=$(echo "scale=2; 450 / 1000 * 100" | bc)
echo "GC content: $GC_CONTENT%"

String Operations

#!/bin/bash

FILE="sample_001.fastq.gz"

# Remove suffix
NAME="${FILE%.fastq.gz}"    # sample_001

# Remove prefix  
NUMBER="${FILE#sample_}"    # 001.fastq.gz

# Remove longest match from end
BASENAME="${FILE%%.*}"      # sample_001

# String length
LENGTH=${#FILE}             # 19

echo "Original: $FILE"
echo "Name without extension: $NAME"
echo "Length: $LENGTH characters"

The #, ##, %, and %% operators above belong to a larger family of parameter expansions; Table 10.2 collects the forms this course uses most, each applied to the familiar FILE="sample_001.fastq.gz".

Table 10.2: Bash parameter expansions at a glance
Expansion Meaning Example Result
${VAR#pat} Remove shortest matching prefix ${FILE#sample_} 001.fastq.gz
${VAR##pat} Remove longest matching prefix ${FILE##*.} gz
${VAR%pat} Remove shortest matching suffix ${FILE%.gz} sample_001.fastq
${VAR%%pat} Remove longest matching suffix ${FILE%%.*} sample_001
${#VAR} Length in characters ${#FILE} 19
${VAR:-default} Default if unset or empty ${THREADS:-4} 4 when THREADS is unset
${VAR:?msg} Error and exit if unset or empty ${INPUT:?no input file given} script stops with the message
${VAR/old/new} Replace first match ${FILE/fastq/fq} sample_001.fq.gz

10.4 Arrays

Arrays store multiple values:

#!/bin/bash

# Declare an array
BASES=(A C G T)
SAMPLES=("control" "treatment_1" "treatment_2")

# Access elements (0-indexed)
echo "First base: ${BASES[0]}"        # A
echo "Second sample: ${SAMPLES[1]}"   # treatment_1

# All elements
echo "All bases: ${BASES[@]}"

# Array length
echo "Number of samples: ${#SAMPLES[@]}"

# Add element
BASES+=(N)

# Loop through array
for sample in "${SAMPLES[@]}"; do
    echo "Processing: $sample"
done

10.5 Conditional Statements

Conditionals control program flow based on tests.

Basic if/then/else

#!/bin/bash

COUNT=100

if [ $COUNT -gt 50 ]; then
    echo "High count"
elif [ $COUNT -gt 10 ]; then
    echo "Medium count"
else
    echo "Low count"
fi

Test Operators

The operators available inside [ ] are grouped by type in Table 10.3, Table 10.4, and Table 10.5.

Numeric comparisons:

Table 10.3: Numeric comparison operators for [ ] and [[ ]] tests
Operator Meaning
-eq Equal
-ne Not equal
-gt Greater than
-ge Greater than or equal
-lt Less than
-le Less than or equal

String comparisons:

Table 10.4: String comparison operators
Operator Meaning
= Equal
!= Not equal
-z Zero length (empty)
-n Non-zero length

File tests:

Table 10.5: File test operators
Operator Meaning
-f File exists and is regular file
-d Directory exists
-e File exists (any type)
-r File is readable
-w File is writable
-x File is executable
-s File exists and is not empty

Examples

#!/bin/bash

# File existence check
if [ -f "data.txt" ]; then
    echo "File exists"
else
    echo "File not found"
fi

# Directory check with creation
if [ ! -d "results" ]; then
    mkdir results
    echo "Created results directory"
fi

# String comparison
FILETYPE="fasta"
if [ "$FILETYPE" = "fasta" ]; then
    echo "Processing FASTA file"
fi

# Combining conditions
if [ -f "$FILE" ] && [ -r "$FILE" ]; then
    echo "File exists and is readable"
fi
Important

Always quote variables in tests: [ "$VAR" = "value" ]. Without quotes, empty variables cause syntax errors.

10.6 Loops

Loops repeat actions multiple times.

For Loops

#!/bin/bash

# Loop over a list
for base in A C G T; do
    echo "Base: $base"
done

# Loop over files
for file in *.fastq; do
    echo "Processing: $file"
    gzip "$file"
done

# Loop with counter
for i in {1..10}; do
    echo "Iteration $i"
done

# C-style loop
for ((i=0; i<10; i++)); do
    echo "Index: $i"
done

# Loop over array
FILES=("sample1.fa" "sample2.fa" "sample3.fa")
for file in "${FILES[@]}"; do
    echo "Analyzing $file"
done

While Loops

#!/bin/bash

# Count down
COUNT=5
while [ $COUNT -gt 0 ]; do
    echo "Count: $COUNT"
    COUNT=$((COUNT - 1))
done

# Read file line by line
while read line; do
    echo "Processing: $line"
done < input.txt

# Process command output
ls *.fa | while read file; do
    echo "Found: $file"
done

Loop Control

#!/bin/bash

for file in *.fa; do
    # Skip files starting with "test"
    if [[ "$file" == test* ]]; then
        continue
    fi
    
    # Stop if we find "final.fa"
    if [ "$file" = "final.fa" ]; then
        echo "Found final file, stopping"
        break
    fi
    
    echo "Processing: $file"
done

10.7 Command-Line Arguments

Scripts become more flexible when they accept arguments:

#!/bin/bash
# process_fasta.sh - Process a FASTA file

# $0 is the script name
echo "Script: $0"

# $# is the number of arguments
echo "Arguments received: $#"

# $1, $2, etc. are positional arguments
INPUT=$1
OUTPUT=$2

# $@ is all arguments
echo "All arguments: $@"

# Check for required arguments
if [ $# -lt 2 ]; then
    echo "Usage: $0 input.fasta output.txt"
    exit 1
fi

# Now process the files
echo "Input: $INPUT"
echo "Output: $OUTPUT"

Usage:

$ ./process_fasta.sh sequences.fa results.txt

Argument Validation

#!/bin/bash

# Check argument count
if [ $# -ne 2 ]; then
    echo "Error: Expected 2 arguments, got $#" >&2
    echo "Usage: $0 input.fa output.txt" >&2
    exit 1
fi

INPUT=$1
OUTPUT=$2

# Validate input file
if [ ! -f "$INPUT" ]; then
    echo "Error: Input file '$INPUT' not found" >&2
    exit 1
fi

# Check if output would overwrite
if [ -f "$OUTPUT" ]; then
    echo "Warning: Output file exists. Overwrite? (y/n)"
    read answer
    if [ "$answer" != "y" ]; then
        exit 0
    fi
fi

# Continue with processing...

10.8 Functions

Functions make scripts modular and reusable:

#!/bin/bash

# Define a function
count_sequences() {
    local FILE=$1  # Local variable
    local COUNT=$(grep -c "^>" "$FILE")
    echo $COUNT
}

# Function with multiple parameters
analyze_fasta() {
    local INPUT=$1
    local OUTPUT=$2
    
    echo "Analyzing $INPUT..."
    
    local SEQ_COUNT=$(count_sequences "$INPUT")
    local TOTAL_LENGTH=$(grep -v "^>" "$INPUT" | tr -d '\n' | wc -c)
    
    echo "Sequences: $SEQ_COUNT" > "$OUTPUT"
    echo "Total bases: $TOTAL_LENGTH" >> "$OUTPUT"
}

# Call functions
SEQ_NUM=$(count_sequences "proteins.fa")
echo "Found $SEQ_NUM sequences"

analyze_fasta "input.fa" "statistics.txt"

Return Values

Functions return exit status (0-255), not strings. To return data:

#!/bin/bash

# Return via stdout
get_gc_content() {
    local FILE=$1
    local GC=$(grep -v "^>" "$FILE" | tr 'acgt' 'ACGT' | tr -d '\n' | grep -o "[GC]" | wc -l)
    local TOTAL=$(grep -v "^>" "$FILE" | tr -d '\n' | wc -c)
    echo "$((GC * 100 / TOTAL))"
}

# Capture output
GC_PERCENT=$(get_gc_content "genome.fa")
echo "GC content: $GC_PERCENT%"

# Return status for success/failure
check_file() {
    if [ -f "$1" ] && [ -r "$1" ]; then
        return 0  # Success
    else
        return 1  # Failure
    fi
}

if check_file "data.txt"; then
    echo "File is ready"
else
    echo "File not accessible"
fi

10.9 Error Handling

Robust scripts handle errors gracefully.

Exit Status

Every command returns an exit status: 0 for success, non-zero for failure.

#!/bin/bash

# Check last command's status
grep "pattern" file.txt
if [ $? -ne 0 ]; then
    echo "Pattern not found"
fi

# Conditional execution
grep "pattern" file.txt && echo "Found!"
grep "pattern" file.txt || echo "Not found"

# Test a command directly in an if (no need for $?); ! negates the
# result, and the redirections silence grep's own output
if ! grep "pattern" file.txt > /dev/null 2>&1; then
    echo "Pattern not found"
fi

The last form is the shell’s equivalent of a try/catch: any command can serve as the condition of an if, because if simply checks whether the command’s exit status was 0.

Strict Mode

Start scripts with these settings for better error detection:

#!/bin/bash
set -e          # Exit on any error
set -u          # Error on undefined variables
set -o pipefail # Catch errors in pipes

# Or combine: set -euo pipefail

Each flag closes a different silent-failure loophole. Without -e, a script whose third command fails cheerfully runs the fourth on garbage input; with it, the script stops at the first non-zero exit status. Without -u, a misspelled variable ($OUTDIRR for $OUTDIR) silently expands to the empty string—so a command like rm -r "$OUTDIRR/tmp" operates on a path you never intended; with it, the misspelling is an immediate, named error. And without pipefail, a pipeline’s exit status is that of its last command, so zcat missing.gz | head reports success because head succeeded even though zcat found nothing; pipefail makes the pipeline fail if any stage fails.

Make the shebang line and set -euo pipefail the first two lines of every script you write—the course scripts, including gene_means.sh in Section 10.14, all do. One caveat to know about: -e is suspended for commands whose exit status is being tested, such as the condition of an if or the left side of &&/||, which is exactly what makes the try/catch pattern of Section 10.9.1 work inside a strict-mode script.

Custom Error Handling

#!/bin/bash
set -euo pipefail

# Define error handler
error_exit() {
    echo "Error on line $1" >&2
    exit 1
}

# Set trap to call handler on error
trap 'error_exit $LINENO' ERR

# Now errors are caught
check_file() {
    if [ ! -f "$1" ]; then
        echo "Error: File $1 not found!" >&2
        exit 1
    fi
}

# Validate inputs
check_file "$1"

# Continue processing...

Cleanup on Exit

Ensure temporary files are removed even if script fails:

#!/bin/bash

# Create temp file
TMPFILE=$(mktemp)

# Remove temp file on exit (normal or error)
trap "rm -f $TMPFILE" EXIT

# Use temp file...
echo "working data" > "$TMPFILE"
# ... do processing ...

# Cleanup happens automatically

10.10 Running R Scripts from the Shell

While shell scripts are powerful, sometimes you need R’s statistical capabilities. The Rscript command lets you run R code directly from the shell.

Basic Usage

# Run an R script
$ Rscript my_analysis.R

# Execute a single R expression
$ Rscript -e "print('Hello from R!')"

# Quick calculation
$ Rscript -e "mean(c(1, 2, 3, 4, 5))"

Passing Arguments to R Scripts

You can pass command-line arguments to R scripts, making them flexible and reusable:

#!/usr/bin/env Rscript
# process_data.R - Process a data file with custom parameters

# Capture command-line arguments
args <- commandArgs(trailingOnly = TRUE)

# Check for required arguments
if (length(args) < 2) {
  stop("Usage: Rscript process_data.R <input_file> <output_file>")
}

input_file <- args[1]
output_file <- args[2]

# Now use these in your analysis
data <- read.csv(input_file)
# ... process data ...
write.csv(data, output_file)

cat("Processed", input_file, "-> ", output_file, "\n")

Run it from the shell:

$ Rscript process_data.R raw_data.csv cleaned_data.csv

Combining Shell and R

A common pattern is using shell scripts to orchestrate R analyses:

#!/bin/bash
# run_analysis.sh - Process all CSV files in a directory

for file in data/*.csv; do
    output="results/$(basename "$file" .csv)_processed.csv"
    echo "Processing $file..."
    Rscript process_data.R "$file" "$output"
done

echo "All files processed!"
Tip

Using Rscript is particularly valuable for batch processing, scheduled tasks (cron jobs), and integration with high-performance computing clusters where you submit shell scripts as jobs (Chapter 20).

10.11 Scripts in Other Languages

Everything in this chapter about shebangs, chmod +x, and command-line arguments applies to scripts in any interpreted language. The shell reads the shebang line, hands the file to that interpreter, and passes the arguments along. Here is the same tiny program, counting the records in a FASTA file whose name is given on the command line, written in bash, R, and Python. Notice that the structure is identical: read the first argument, complain if it is missing, count lines beginning with >, and print the result.

#!/bin/bash
# count_fasta.sh - count records in a FASTA file
if [ $# -lt 1 ]; then
    echo "Usage: $0 <file.fasta>" >&2
    exit 1
fi

n=$(grep -c '^>' "$1")
echo "$1: $n records"
chmod +x count_fasta.sh
./count_fasta.sh sequences.fasta
# sequences.fasta: 3 records
#!/usr/bin/env Rscript
# count_fasta.R - count records in a FASTA file
args <- commandArgs(trailingOnly = TRUE)
if (length(args) < 1) {
  stop("Usage: count_fasta.R <file.fasta>")
}

lines <- readLines(args[1])
n <- sum(startsWith(lines, ">"))
cat(args[1], ": ", n, " records\n", sep = "")
chmod +x count_fasta.R
./count_fasta.R sequences.fasta        # or: Rscript count_fasta.R sequences.fasta
# sequences.fasta: 3 records
#!/usr/bin/env python3
# count_fasta.py - count records in a FASTA file
import sys

if len(sys.argv) < 2:
    sys.exit("Usage: count_fasta.py <file.fasta>")

with open(sys.argv[1]) as fh:
    n = sum(1 for line in fh if line.startswith(">"))
print(f"{sys.argv[1]}: {n} records")
chmod +x count_fasta.py
./count_fasta.py sequences.fasta       # or: python3 count_fasta.py sequences.fasta
# sequences.fasta: 3 records

The differences are small and systematic: bash sees arguments as $1, $2, …; R collects them with commandArgs(trailingOnly = TRUE) into a character vector indexed from 1; Python puts them in the list sys.argv, where element 0 is the script name and the first real argument is sys.argv[1]. Whichever language you choose, the shell treats the script the same way, so a bash driver script can call any of them in a loop, as in the previous section. R scripting is covered in Chapter 11 and Python scripting, including the more capable argparse module, in Chapter 12.

10.12 A Complete Bioinformatics Script

Let’s put it all together with a realistic example:

#!/bin/bash
#===============================================================================
# fasta_stats.sh - Calculate statistics for FASTA files
# 
# Usage: ./fasta_stats.sh input.fasta [output.txt]
#
# Author: Your Name
# Date: 2025-01-15
#===============================================================================

set -euo pipefail

#-------------------------------------------------------------------------------
# Configuration
#-------------------------------------------------------------------------------
VERSION="1.0.0"

#-------------------------------------------------------------------------------
# Functions
#-------------------------------------------------------------------------------

usage() {
    cat << EOF
Usage: $(basename "$0") [OPTIONS] input.fasta [output.txt]

Calculate basic statistics for FASTA files.

Options:
    -h, --help      Show this help message
    -v, --version   Show version information
    -q, --quiet     Suppress progress messages

Arguments:
    input.fasta     Input FASTA file (required)
    output.txt      Output file (optional, defaults to stdout)

Examples:
    $(basename "$0") sequences.fa
    $(basename "$0") genome.fa stats.txt
    $(basename "$0") -q input.fa > results.txt
EOF
}

log() {
    if [ "$QUIET" = false ]; then
        echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" >&2
    fi
}

error() {
    echo "Error: $1" >&2
    exit 1
}

count_sequences() {
    grep -c "^>" "$1"
}

total_length() {
    grep -v "^>" "$1" | tr -d '\n' | wc -c | tr -d ' '
}

gc_content() {
    local gc=$(grep -v "^>" "$1" | tr 'acgt' 'ACGT' | tr -d '\n' | grep -o "[GC]" | wc -l | tr -d ' ')
    local total=$(total_length "$1")
    if [ "$total" -gt 0 ]; then
        echo "$((gc * 100 / total))"
    else
        echo "0"
    fi
}

n_count() {
    grep -v "^>" "$1" | tr -cd 'Nn' | wc -c | tr -d ' '
}

orf_like_count() {
    # uppercase-only pattern: soft-masked (lowercase) repeats cannot match
    grep -v "^>" "$1" | tr -d '\n' | \
        grep -oE "ATG([ACGT]{3}){20,}(TAA|TAG|TGA)" | wc -l | tr -d ' '
}

#-------------------------------------------------------------------------------
# Parse Arguments
#-------------------------------------------------------------------------------

QUIET=false

while [[ $# -gt 0 ]]; do
    case $1 in
        -h|--help)
            usage
            exit 0
            ;;
        -v|--version)
            echo "$(basename "$0") version $VERSION"
            exit 0
            ;;
        -q|--quiet)
            QUIET=true
            shift
            ;;
        -*)
            error "Unknown option: $1"
            ;;
        *)
            break
            ;;
    esac
done

# Check required arguments
if [ $# -lt 1 ]; then
    usage >&2
    exit 1
fi

INPUT="$1"
OUTPUT="${2:-/dev/stdout}"

#-------------------------------------------------------------------------------
# Validate Input
#-------------------------------------------------------------------------------

if [ ! -f "$INPUT" ]; then
    error "Input file not found: $INPUT"
fi

if [ ! -r "$INPUT" ]; then
    error "Cannot read input file: $INPUT"
fi

#-------------------------------------------------------------------------------
# Main Analysis
#-------------------------------------------------------------------------------

log "Starting analysis of $INPUT"

log "Counting sequences..."
SEQ_COUNT=$(count_sequences "$INPUT")

log "Calculating total length..."
TOTAL_LEN=$(total_length "$INPUT")

log "Computing GC content..."
GC_PCT=$(gc_content "$INPUT")

log "Counting N bases..."
N_BASES=$(n_count "$INPUT")

log "Scanning for ORF-like stretches..."
ORF_COUNT=$(orf_like_count "$INPUT")

# Calculate average length
if [ "$SEQ_COUNT" -gt 0 ]; then
    AVG_LEN=$((TOTAL_LEN / SEQ_COUNT))
else
    AVG_LEN=0
fi

#-------------------------------------------------------------------------------
# Output Results
#-------------------------------------------------------------------------------

{
    echo "=== FASTA Statistics ==="
    echo "File: $INPUT"
    echo "Date: $(date '+%Y-%m-%d %H:%M:%S')"
    echo ""
    echo "Number of sequences: $SEQ_COUNT"
    echo "Total length: $TOTAL_LEN bp"
    echo "Average length: $AVG_LEN bp"
    echo "GC content: $GC_PCT%"
    echo "N bases: $N_BASES"
    echo "ORF-like stretches: $ORF_COUNT"
} > "$OUTPUT"

log "Analysis complete. Results written to $OUTPUT"

10.13 Best Practices Summary

TipShell Scripting Best Practices
  1. Always include a shebang: #!/bin/bash
  2. Use strict mode: set -euo pipefail
  3. Comment your code: Explain what and why
  4. Use meaningful names: SEQ_COUNT not SC
  5. Validate inputs: Check files exist, arguments are provided
  6. Handle errors: Provide useful error messages
  7. Make it portable: Don’t hardcode paths
  8. Test incrementally: Build and test piece by piece
  9. Version control: Track changes with git (Chapter 19)
  10. Document usage: Include help messages

10.14 Running Example: The Stickleback Data

NoteRunning example

This section applies the chapter to the course dataset described in Section 1.5 (80 stickleback, 4 treatment groups, 600 genes). The same steps appear on the Week 4 slides, so the book and the lecture can be read side by side.

The awk one-liners of Chapter 9 answer one question about one gene. A script answers the same question for any gene, checks its input, and can be run in a loop. gene_means.sh takes a column number, looks up the gene’s name from the header, and prints the mean count in each of the four treatment groups:

#!/bin/bash
# gene_means.sh - mean count of one gene per treatment group
# Usage: ./gene_means.sh <gene column number>
set -euo pipefail
col=${1:?Give a gene column number, e.g. 5 for Gene1}
infile=results/stickle_clean.tsv

gene=$(head -n 1 "$infile" | cut -f "$col")
echo "Mean of $gene by group:"
awk -F'\t' -v c="$col" 'NR > 1 { sum[$4] += $c; n[$4]++ } END { for (g in sum) printf "  %-6s %8.1f\n", g, sum[g]/n[g] }' "$infile"

Three habits from this chapter are on display. set -euo pipefail makes the script stop at the first error rather than plough on with bad data. ${1:?...} is a parameter expansion that aborts with a helpful message if no argument was given. And -v c="$col" is how a shell variable is handed to awk, which has its own variables and cannot see the shell’s. Make it executable and run it, then loop over several genes:

chmod +x gene_means.sh
./gene_means.sh 5                              # Gene1
for c in 5 6 7; do ./gene_means.sh "$c"; done  # the first three genes

For genotype A fish, Gene1 averages about 16 counts in the mono-associated group and 12 in the conventional group; for genotype B, about 2 and 0.6. In Chapter 20 this same script runs unchanged as a batch job on Talapas.

10.15 Summary

This chapter covered shell scripting essentials:

  • Scripts are text files containing commands with a shebang line
  • Variables store values; arrays store multiple values
  • Conditionals (if/elif/else) control flow based on tests
  • Loops (for, while) repeat actions
  • Functions make code modular and reusable
  • Command-line arguments make scripts flexible
  • Error handling ensures robustness
  • Combining these elements creates powerful analysis pipelines

Shell scripting is a foundational skill for computational biology. Even as you learn other languages like R and Python, shell scripts remain invaluable for automation, file management, and orchestrating complex workflows.

Tip

For quick reference on Unix commands used in shell scripts, see Appendix B.

Additional Reading

To advance your shell scripting skills:

For bioinformatics workflow development:

  • Snakemake — A workflow management system that builds on shell scripting concepts
  • Nextflow — A domain-specific language for data-driven pipelines
  • GNU Make Tutorial — Build automation that complements shell scripts

Exercises

TipPractice Exercises

Exercise 1: Basic Script

Create a script that: 1. Accepts a directory as an argument 2. Counts files of each type (.txt, .csv, .fa, etc.) 3. Reports the results

Exercise 2: File Processing Loop

Write a script that: 1. Loops through all .fastq files in a directory 2. Counts the number of reads in each (hint: lines/4 in FASTQ) 3. Saves results to a summary file

Exercise 3: FASTA Validator

Create a script that validates FASTA files: 1. Check that every header line starts with > 2. Check that sequence lines only contain valid characters 3. Report any errors found

Exercise 4: Analysis Pipeline

Build a complete analysis script that: 1. Accepts input/output arguments with validation 2. Downloads a bacterial genome if not present 3. Calculates genome statistics 4. Finds all occurrences of a user-specified motif 5. Generates a summary report 6. Includes proper error handling and logging

Exercise 5: Extending the Course Scripts

Starting from the Week 2 example scripts on the Example Files page:

  1. Modify filter_rows.sh to accept an optional filename argument, reading from stdin when none is given
  2. Add a --mean flag to total_counts.sh that reports each fish’s mean count across genes instead of the total
  3. Extend make_project.sh to take an optional second argument naming the project’s author, written into every template it creates
  4. Add a safety check to backup_project.sh: warn and exit if a backup of the same project was already made today