10 Shell Scripting
After completing this chapter, you will be able to:
- Create and execute shell scripts
- Use variables, arrays, and command substitution
- Write conditional statements and loops
- Define and call functions
- Handle command-line arguments
- Implement proper error handling
- Build reproducible analysis pipelines
10.1 Why Shell Scripting?
Throughout the previous chapters, you’ve been typing commands interactively. While this is great for exploration, real-world analyses require something more (Sandve et al., 2013). You need reproducibility—a complete record of exactly what you did so you (or others) can repeat the analysis. You need automation so you can run the same analysis on many files without typing each command manually. Scripts provide efficiency by eliminating repetitive typing, and they serve as documentation that explains why you did each step. Finally, scripts enable sharing—letting collaborators reproduce your work with a single command.
Shell scripts transform a series of commands into a reusable, documented program.
10.2 Your First Shell Script
A shell script is simply a text file containing commands. Let’s create one:
The Shebang Line
The first line #!/bin/bash is called a shebang (or hashbang). It tells the operating system which interpreter should execute this script. Always include it; Table 10.1 lists the common ones.
| Shebang | Interpreter | Notes |
|---|---|---|
#!/bin/bash |
Bash shell | Most common for shell scripts |
#!/usr/bin/env bash |
Bash shell | Finds bash wherever it is installed; used in the course example scripts |
#!/bin/sh |
POSIX shell | More portable across systems |
#!/usr/bin/env python3 |
Python 3 | Uses env to find Python |
#!/usr/bin/env Rscript |
R | For R scripts |
Running Scripts
There are several ways to run a script:
The chmod +x command adds execute permission to the file. After that, you can run it directly with ./. The ./ is necessary because your current directory typically isn’t in your system’s PATH (see Section 6.4; permissions are explained in Section 6.2.2).
Always use chmod +x on your scripts. It signals that the file is meant to be executed and allows direct invocation.
The course’s Example Files page has four heavily commented scripts from Week 2 that use the techniques in this chapter—arguments with defaults and ${1:?usage} checks, command substitution, here-documents, and pipelines inside and between scripts. Reading them alongside this chapter, and then modifying them, is the fastest way to make the material stick.
10.3 Variables
Variables store values for later use. In Bash, variable assignment has strict syntax rules.
Assigning Variables
#!/bin/bash
# Assign variables (NO spaces around the =)
NAME="DNA Analysis"
COUNT=100
INPUT_FILE="sequences.fa"
# Use variables (with $ prefix)
echo "Running $NAME"
echo "Processing $COUNT sequences from $INPUT_FILE"
# Curly braces for clarity
echo "File: ${INPUT_FILE}"
echo "Creating ${INPUT_FILE}.backup"Command Substitution
Capture command output in a variable:
#!/bin/bash
# Old syntax (backticks)
TODAY=`date +%Y-%m-%d`
# Modern syntax (preferred)
TODAY=$(date +%Y-%m-%d)
FILECOUNT=$(ls *.fa | wc -l)
GENOME_SIZE=$(grep -v "^>" genome.fa | tr -d '\n' | wc -c)
echo "Analysis date: $TODAY"
echo "Processing $FILECOUNT FASTA files"
echo "Genome size: $GENOME_SIZE bp"Arithmetic Operations
Bash arithmetic uses $(( )):
For floating-point math, use external tools like bc:
String Operations
#!/bin/bash
FILE="sample_001.fastq.gz"
# Remove suffix
NAME="${FILE%.fastq.gz}" # sample_001
# Remove prefix
NUMBER="${FILE#sample_}" # 001.fastq.gz
# Remove longest match from end
BASENAME="${FILE%%.*}" # sample_001
# String length
LENGTH=${#FILE} # 19
echo "Original: $FILE"
echo "Name without extension: $NAME"
echo "Length: $LENGTH characters"The #, ##, %, and %% operators above belong to a larger family of parameter expansions; Table 10.2 collects the forms this course uses most, each applied to the familiar FILE="sample_001.fastq.gz".
| Expansion | Meaning | Example | Result |
|---|---|---|---|
${VAR#pat} |
Remove shortest matching prefix | ${FILE#sample_} |
001.fastq.gz |
${VAR##pat} |
Remove longest matching prefix | ${FILE##*.} |
gz |
${VAR%pat} |
Remove shortest matching suffix | ${FILE%.gz} |
sample_001.fastq |
${VAR%%pat} |
Remove longest matching suffix | ${FILE%%.*} |
sample_001 |
${#VAR} |
Length in characters | ${#FILE} |
19 |
${VAR:-default} |
Default if unset or empty | ${THREADS:-4} |
4 when THREADS is unset |
${VAR:?msg} |
Error and exit if unset or empty | ${INPUT:?no input file given} |
script stops with the message |
${VAR/old/new} |
Replace first match | ${FILE/fastq/fq} |
sample_001.fq.gz |
10.4 Arrays
Arrays store multiple values:
#!/bin/bash
# Declare an array
BASES=(A C G T)
SAMPLES=("control" "treatment_1" "treatment_2")
# Access elements (0-indexed)
echo "First base: ${BASES[0]}" # A
echo "Second sample: ${SAMPLES[1]}" # treatment_1
# All elements
echo "All bases: ${BASES[@]}"
# Array length
echo "Number of samples: ${#SAMPLES[@]}"
# Add element
BASES+=(N)
# Loop through array
for sample in "${SAMPLES[@]}"; do
echo "Processing: $sample"
done10.5 Conditional Statements
Conditionals control program flow based on tests.
Basic if/then/else
Test Operators
The operators available inside [ ] are grouped by type in Table 10.3, Table 10.4, and Table 10.5.
Numeric comparisons:
[ ] and [[ ]] tests
| Operator | Meaning |
|---|---|
-eq |
Equal |
-ne |
Not equal |
-gt |
Greater than |
-ge |
Greater than or equal |
-lt |
Less than |
-le |
Less than or equal |
String comparisons:
| Operator | Meaning |
|---|---|
= |
Equal |
!= |
Not equal |
-z |
Zero length (empty) |
-n |
Non-zero length |
File tests:
| Operator | Meaning |
|---|---|
-f |
File exists and is regular file |
-d |
Directory exists |
-e |
File exists (any type) |
-r |
File is readable |
-w |
File is writable |
-x |
File is executable |
-s |
File exists and is not empty |
Examples
#!/bin/bash
# File existence check
if [ -f "data.txt" ]; then
echo "File exists"
else
echo "File not found"
fi
# Directory check with creation
if [ ! -d "results" ]; then
mkdir results
echo "Created results directory"
fi
# String comparison
FILETYPE="fasta"
if [ "$FILETYPE" = "fasta" ]; then
echo "Processing FASTA file"
fi
# Combining conditions
if [ -f "$FILE" ] && [ -r "$FILE" ]; then
echo "File exists and is readable"
fiAlways quote variables in tests: [ "$VAR" = "value" ]. Without quotes, empty variables cause syntax errors.
10.6 Loops
Loops repeat actions multiple times.
For Loops
#!/bin/bash
# Loop over a list
for base in A C G T; do
echo "Base: $base"
done
# Loop over files
for file in *.fastq; do
echo "Processing: $file"
gzip "$file"
done
# Loop with counter
for i in {1..10}; do
echo "Iteration $i"
done
# C-style loop
for ((i=0; i<10; i++)); do
echo "Index: $i"
done
# Loop over array
FILES=("sample1.fa" "sample2.fa" "sample3.fa")
for file in "${FILES[@]}"; do
echo "Analyzing $file"
doneWhile Loops
Loop Control
10.7 Command-Line Arguments
Scripts become more flexible when they accept arguments:
#!/bin/bash
# process_fasta.sh - Process a FASTA file
# $0 is the script name
echo "Script: $0"
# $# is the number of arguments
echo "Arguments received: $#"
# $1, $2, etc. are positional arguments
INPUT=$1
OUTPUT=$2
# $@ is all arguments
echo "All arguments: $@"
# Check for required arguments
if [ $# -lt 2 ]; then
echo "Usage: $0 input.fasta output.txt"
exit 1
fi
# Now process the files
echo "Input: $INPUT"
echo "Output: $OUTPUT"Usage:
Argument Validation
#!/bin/bash
# Check argument count
if [ $# -ne 2 ]; then
echo "Error: Expected 2 arguments, got $#" >&2
echo "Usage: $0 input.fa output.txt" >&2
exit 1
fi
INPUT=$1
OUTPUT=$2
# Validate input file
if [ ! -f "$INPUT" ]; then
echo "Error: Input file '$INPUT' not found" >&2
exit 1
fi
# Check if output would overwrite
if [ -f "$OUTPUT" ]; then
echo "Warning: Output file exists. Overwrite? (y/n)"
read answer
if [ "$answer" != "y" ]; then
exit 0
fi
fi
# Continue with processing...10.8 Functions
Functions make scripts modular and reusable:
#!/bin/bash
# Define a function
count_sequences() {
local FILE=$1 # Local variable
local COUNT=$(grep -c "^>" "$FILE")
echo $COUNT
}
# Function with multiple parameters
analyze_fasta() {
local INPUT=$1
local OUTPUT=$2
echo "Analyzing $INPUT..."
local SEQ_COUNT=$(count_sequences "$INPUT")
local TOTAL_LENGTH=$(grep -v "^>" "$INPUT" | tr -d '\n' | wc -c)
echo "Sequences: $SEQ_COUNT" > "$OUTPUT"
echo "Total bases: $TOTAL_LENGTH" >> "$OUTPUT"
}
# Call functions
SEQ_NUM=$(count_sequences "proteins.fa")
echo "Found $SEQ_NUM sequences"
analyze_fasta "input.fa" "statistics.txt"Return Values
Functions return exit status (0-255), not strings. To return data:
#!/bin/bash
# Return via stdout
get_gc_content() {
local FILE=$1
local GC=$(grep -v "^>" "$FILE" | tr 'acgt' 'ACGT' | tr -d '\n' | grep -o "[GC]" | wc -l)
local TOTAL=$(grep -v "^>" "$FILE" | tr -d '\n' | wc -c)
echo "$((GC * 100 / TOTAL))"
}
# Capture output
GC_PERCENT=$(get_gc_content "genome.fa")
echo "GC content: $GC_PERCENT%"
# Return status for success/failure
check_file() {
if [ -f "$1" ] && [ -r "$1" ]; then
return 0 # Success
else
return 1 # Failure
fi
}
if check_file "data.txt"; then
echo "File is ready"
else
echo "File not accessible"
fi10.9 Error Handling
Robust scripts handle errors gracefully.
Exit Status
Every command returns an exit status: 0 for success, non-zero for failure.
#!/bin/bash
# Check last command's status
grep "pattern" file.txt
if [ $? -ne 0 ]; then
echo "Pattern not found"
fi
# Conditional execution
grep "pattern" file.txt && echo "Found!"
grep "pattern" file.txt || echo "Not found"
# Test a command directly in an if (no need for $?); ! negates the
# result, and the redirections silence grep's own output
if ! grep "pattern" file.txt > /dev/null 2>&1; then
echo "Pattern not found"
fiThe last form is the shell’s equivalent of a try/catch: any command can serve as the condition of an if, because if simply checks whether the command’s exit status was 0.
Strict Mode
Start scripts with these settings for better error detection:
Each flag closes a different silent-failure loophole. Without -e, a script whose third command fails cheerfully runs the fourth on garbage input; with it, the script stops at the first non-zero exit status. Without -u, a misspelled variable ($OUTDIRR for $OUTDIR) silently expands to the empty string—so a command like rm -r "$OUTDIRR/tmp" operates on a path you never intended; with it, the misspelling is an immediate, named error. And without pipefail, a pipeline’s exit status is that of its last command, so zcat missing.gz | head reports success because head succeeded even though zcat found nothing; pipefail makes the pipeline fail if any stage fails.
Make the shebang line and set -euo pipefail the first two lines of every script you write—the course scripts, including gene_means.sh in Section 10.14, all do. One caveat to know about: -e is suspended for commands whose exit status is being tested, such as the condition of an if or the left side of &&/||, which is exactly what makes the try/catch pattern of Section 10.9.1 work inside a strict-mode script.
Custom Error Handling
#!/bin/bash
set -euo pipefail
# Define error handler
error_exit() {
echo "Error on line $1" >&2
exit 1
}
# Set trap to call handler on error
trap 'error_exit $LINENO' ERR
# Now errors are caught
check_file() {
if [ ! -f "$1" ]; then
echo "Error: File $1 not found!" >&2
exit 1
fi
}
# Validate inputs
check_file "$1"
# Continue processing...Cleanup on Exit
Ensure temporary files are removed even if script fails:
10.10 Running R Scripts from the Shell
While shell scripts are powerful, sometimes you need R’s statistical capabilities. The Rscript command lets you run R code directly from the shell.
Basic Usage
Passing Arguments to R Scripts
You can pass command-line arguments to R scripts, making them flexible and reusable:
#!/usr/bin/env Rscript
# process_data.R - Process a data file with custom parameters
# Capture command-line arguments
args <- commandArgs(trailingOnly = TRUE)
# Check for required arguments
if (length(args) < 2) {
stop("Usage: Rscript process_data.R <input_file> <output_file>")
}
input_file <- args[1]
output_file <- args[2]
# Now use these in your analysis
data <- read.csv(input_file)
# ... process data ...
write.csv(data, output_file)
cat("Processed", input_file, "-> ", output_file, "\n")Run it from the shell:
Combining Shell and R
A common pattern is using shell scripts to orchestrate R analyses:
Using Rscript is particularly valuable for batch processing, scheduled tasks (cron jobs), and integration with high-performance computing clusters where you submit shell scripts as jobs (Chapter 20).
10.11 Scripts in Other Languages
Everything in this chapter about shebangs, chmod +x, and command-line arguments applies to scripts in any interpreted language. The shell reads the shebang line, hands the file to that interpreter, and passes the arguments along. Here is the same tiny program, counting the records in a FASTA file whose name is given on the command line, written in bash, R, and Python. Notice that the structure is identical: read the first argument, complain if it is missing, count lines beginning with >, and print the result.
The differences are small and systematic: bash sees arguments as $1, $2, …; R collects them with commandArgs(trailingOnly = TRUE) into a character vector indexed from 1; Python puts them in the list sys.argv, where element 0 is the script name and the first real argument is sys.argv[1]. Whichever language you choose, the shell treats the script the same way, so a bash driver script can call any of them in a loop, as in the previous section. R scripting is covered in Chapter 11 and Python scripting, including the more capable argparse module, in Chapter 12.
10.12 A Complete Bioinformatics Script
Let’s put it all together with a realistic example:
#!/bin/bash
#===============================================================================
# fasta_stats.sh - Calculate statistics for FASTA files
#
# Usage: ./fasta_stats.sh input.fasta [output.txt]
#
# Author: Your Name
# Date: 2025-01-15
#===============================================================================
set -euo pipefail
#-------------------------------------------------------------------------------
# Configuration
#-------------------------------------------------------------------------------
VERSION="1.0.0"
#-------------------------------------------------------------------------------
# Functions
#-------------------------------------------------------------------------------
usage() {
cat << EOF
Usage: $(basename "$0") [OPTIONS] input.fasta [output.txt]
Calculate basic statistics for FASTA files.
Options:
-h, --help Show this help message
-v, --version Show version information
-q, --quiet Suppress progress messages
Arguments:
input.fasta Input FASTA file (required)
output.txt Output file (optional, defaults to stdout)
Examples:
$(basename "$0") sequences.fa
$(basename "$0") genome.fa stats.txt
$(basename "$0") -q input.fa > results.txt
EOF
}
log() {
if [ "$QUIET" = false ]; then
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" >&2
fi
}
error() {
echo "Error: $1" >&2
exit 1
}
count_sequences() {
grep -c "^>" "$1"
}
total_length() {
grep -v "^>" "$1" | tr -d '\n' | wc -c | tr -d ' '
}
gc_content() {
local gc=$(grep -v "^>" "$1" | tr 'acgt' 'ACGT' | tr -d '\n' | grep -o "[GC]" | wc -l | tr -d ' ')
local total=$(total_length "$1")
if [ "$total" -gt 0 ]; then
echo "$((gc * 100 / total))"
else
echo "0"
fi
}
n_count() {
grep -v "^>" "$1" | tr -cd 'Nn' | wc -c | tr -d ' '
}
orf_like_count() {
# uppercase-only pattern: soft-masked (lowercase) repeats cannot match
grep -v "^>" "$1" | tr -d '\n' | \
grep -oE "ATG([ACGT]{3}){20,}(TAA|TAG|TGA)" | wc -l | tr -d ' '
}
#-------------------------------------------------------------------------------
# Parse Arguments
#-------------------------------------------------------------------------------
QUIET=false
while [[ $# -gt 0 ]]; do
case $1 in
-h|--help)
usage
exit 0
;;
-v|--version)
echo "$(basename "$0") version $VERSION"
exit 0
;;
-q|--quiet)
QUIET=true
shift
;;
-*)
error "Unknown option: $1"
;;
*)
break
;;
esac
done
# Check required arguments
if [ $# -lt 1 ]; then
usage >&2
exit 1
fi
INPUT="$1"
OUTPUT="${2:-/dev/stdout}"
#-------------------------------------------------------------------------------
# Validate Input
#-------------------------------------------------------------------------------
if [ ! -f "$INPUT" ]; then
error "Input file not found: $INPUT"
fi
if [ ! -r "$INPUT" ]; then
error "Cannot read input file: $INPUT"
fi
#-------------------------------------------------------------------------------
# Main Analysis
#-------------------------------------------------------------------------------
log "Starting analysis of $INPUT"
log "Counting sequences..."
SEQ_COUNT=$(count_sequences "$INPUT")
log "Calculating total length..."
TOTAL_LEN=$(total_length "$INPUT")
log "Computing GC content..."
GC_PCT=$(gc_content "$INPUT")
log "Counting N bases..."
N_BASES=$(n_count "$INPUT")
log "Scanning for ORF-like stretches..."
ORF_COUNT=$(orf_like_count "$INPUT")
# Calculate average length
if [ "$SEQ_COUNT" -gt 0 ]; then
AVG_LEN=$((TOTAL_LEN / SEQ_COUNT))
else
AVG_LEN=0
fi
#-------------------------------------------------------------------------------
# Output Results
#-------------------------------------------------------------------------------
{
echo "=== FASTA Statistics ==="
echo "File: $INPUT"
echo "Date: $(date '+%Y-%m-%d %H:%M:%S')"
echo ""
echo "Number of sequences: $SEQ_COUNT"
echo "Total length: $TOTAL_LEN bp"
echo "Average length: $AVG_LEN bp"
echo "GC content: $GC_PCT%"
echo "N bases: $N_BASES"
echo "ORF-like stretches: $ORF_COUNT"
} > "$OUTPUT"
log "Analysis complete. Results written to $OUTPUT"10.13 Best Practices Summary
-
Always include a shebang:
#!/bin/bash -
Use strict mode:
set -euo pipefail - Comment your code: Explain what and why
-
Use meaningful names:
SEQ_COUNTnotSC - Validate inputs: Check files exist, arguments are provided
- Handle errors: Provide useful error messages
- Make it portable: Don’t hardcode paths
- Test incrementally: Build and test piece by piece
- Version control: Track changes with git (Chapter 19)
- Document usage: Include help messages
10.14 Running Example: The Stickleback Data
This section applies the chapter to the course dataset described in Section 1.5 (80 stickleback, 4 treatment groups, 600 genes). The same steps appear on the Week 4 slides, so the book and the lecture can be read side by side.
The awk one-liners of Chapter 9 answer one question about one gene. A script answers the same question for any gene, checks its input, and can be run in a loop. gene_means.sh takes a column number, looks up the gene’s name from the header, and prints the mean count in each of the four treatment groups:
#!/bin/bash
# gene_means.sh - mean count of one gene per treatment group
# Usage: ./gene_means.sh <gene column number>
set -euo pipefail
col=${1:?Give a gene column number, e.g. 5 for Gene1}
infile=results/stickle_clean.tsv
gene=$(head -n 1 "$infile" | cut -f "$col")
echo "Mean of $gene by group:"
awk -F'\t' -v c="$col" 'NR > 1 { sum[$4] += $c; n[$4]++ } END { for (g in sum) printf " %-6s %8.1f\n", g, sum[g]/n[g] }' "$infile"Three habits from this chapter are on display. set -euo pipefail makes the script stop at the first error rather than plough on with bad data. ${1:?...} is a parameter expansion that aborts with a helpful message if no argument was given. And -v c="$col" is how a shell variable is handed to awk, which has its own variables and cannot see the shell’s. Make it executable and run it, then loop over several genes:
For genotype A fish, Gene1 averages about 16 counts in the mono-associated group and 12 in the conventional group; for genotype B, about 2 and 0.6. In Chapter 20 this same script runs unchanged as a batch job on Talapas.
10.15 Summary
This chapter covered shell scripting essentials:
- Scripts are text files containing commands with a shebang line
- Variables store values; arrays store multiple values
- Conditionals (
if/elif/else) control flow based on tests - Loops (
for,while) repeat actions - Functions make code modular and reusable
- Command-line arguments make scripts flexible
- Error handling ensures robustness
- Combining these elements creates powerful analysis pipelines
Shell scripting is a foundational skill for computational biology. Even as you learn other languages like R and Python, shell scripts remain invaluable for automation, file management, and orchestrating complex workflows.
For quick reference on Unix commands used in shell scripts, see Appendix B.
Additional Reading
To advance your shell scripting skills:
- Bash Guide for Beginners — A comprehensive introduction to Bash scripting
- Advanced Bash-Scripting Guide — An in-depth exploration of shell scripting techniques
- ShellCheck — An online tool for finding bugs in your shell scripts
- “Learning the bash Shell” by Cameron Newham — A thorough guide to Bash programming
- Google Shell Style Guide — Best practices for writing maintainable scripts
For bioinformatics workflow development:
- Snakemake — A workflow management system that builds on shell scripting concepts
- Nextflow — A domain-specific language for data-driven pipelines
- GNU Make Tutorial — Build automation that complements shell scripts
Exercises
Exercise 1: Basic Script
Create a script that: 1. Accepts a directory as an argument 2. Counts files of each type (.txt, .csv, .fa, etc.) 3. Reports the results
Exercise 2: File Processing Loop
Write a script that: 1. Loops through all .fastq files in a directory 2. Counts the number of reads in each (hint: lines/4 in FASTQ) 3. Saves results to a summary file
Exercise 3: FASTA Validator
Create a script that validates FASTA files: 1. Check that every header line starts with > 2. Check that sequence lines only contain valid characters 3. Report any errors found
Exercise 4: Analysis Pipeline
Build a complete analysis script that: 1. Accepts input/output arguments with validation 2. Downloads a bacterial genome if not present 3. Calculates genome statistics 4. Finds all occurrences of a user-specified motif 5. Generates a summary report 6. Includes proper error handling and logging
Exercise 5: Extending the Course Scripts
Starting from the Week 2 example scripts on the Example Files page:
- Modify
filter_rows.shto accept an optional filename argument, reading from stdin when none is given - Add a
--meanflag tototal_counts.shthat reports each fish’s mean count across genes instead of the total - Extend
make_project.shto take an optional second argument naming the project’s author, written into every template it creates - Add a safety check to
backup_project.sh: warn and exit if a backup of the same project was already made today