9  GREP, Regular Expressions, sed and awk

NoteLearning Objectives

After completing this chapter, you will be able to:

  • Use grep to search for patterns in files and data streams
  • Construct regular expressions to match complex patterns
  • Apply extended regular expressions for advanced pattern matching
  • Combine grep with other Unix tools for sophisticated text processing
  • Search genomic data for biological patterns like restriction sites and motifs
  • Edit text streams with sed and filter, reformat, and summarize columns with awk

9.1 Introduction to Pattern Matching

While the tools in Chapter 7 can filter and transform data, they work on whole lines or fixed positions. Pattern matching lets you search based on the content of text—finding lines that contain specific sequences of characters.

GREP (Global Regular Expression Print) is the Unix standard for pattern matching. Originally developed for the ed text editor in the 1970s, grep remains essential for bioinformatics, log analysis, and text processing.

9.2 Basic GREP Usage

At its simplest, grep finds lines containing a specific string:

# Find lines containing "error"
$ grep "error" logfile.txt

# Search multiple files
$ grep "mutation" *.txt

# Search recursively in directories
$ grep -r "BRCA1" data/

Essential GREP Options

Table 9.1: Common grep options
Option Description
-i Case-insensitive search
-v Invert match (lines NOT containing pattern)
-c Count matching lines
-n Show line numbers
-l List only filenames with matches
-o Print only the matched part
-w Match whole words only
-A n Show n lines after match
-B n Show n lines before match
-C n Show n lines before and after match

Examples

# Case-insensitive search
$ grep -i "gene" annotations.txt

# Count matches (not lines, just the count)
$ grep -c ">" sequences.fasta

# Show line numbers
$ grep -n "ERROR" application.log

# Find files containing pattern
$ grep -l "experimental" *.md

# Remove comment lines (inverted match)
$ grep -v "^#" config.txt

# Show context around matches
$ grep -C 2 "significant" results.txt

9.3 Introduction to Regular Expressions

So far we’ve searched for literal text. Regular expressions (regex) let you describe patterns that match multiple strings (Friedl, 2006). A condensed syntax reference is in Appendix D, and the same patterns work in R (stringr) and Python (re).

For example, the pattern [ACGT]+ matches any sequence of DNA bases, while colou?r matches both “color” and “colour”.

Metacharacters: Characters with Special Meaning

Regular expressions use certain characters as pattern-building blocks (Table 9.2):

Table 9.2: Basic regular expression metacharacters
Character Meaning Example
. Any single character a.c matches “abc”, “a1c”, “a-c”
* Zero or more of preceding ab*c matches “ac”, “abc”, “abbc”
^ Start of line ^Hello matches lines starting with “Hello”
$ End of line end$ matches lines ending with “end”
[] Character class [aeiou] matches any vowel
[^] Negated class [^0-9] matches non-digits
\ Escape special characters \. matches literal period
Note

The * in regular expressions is different from the * wildcard in the shell. In regex, * means “zero or more of the preceding character.” In shell globbing, * means “any characters.”

Anchors: Position Matching

Anchors don’t match characters—they match positions:

# Lines starting with "Chr"
$ grep "^Chr" genome.gff

# Lines ending with "protein"
$ grep "protein$" annotations.txt

# Empty lines
$ grep "^$" file.txt

# Lines with exactly "gene"
$ grep "^gene$" list.txt

Character Classes

Character classes match any single character from a set:

# Match any vowel
$ grep "[aeiou]" words.txt

# Match any digit
$ grep "[0-9]" data.txt

# Match any uppercase letter
$ grep "[A-Z]" text.txt

# Match DNA bases
$ grep "[ACGT]" sequence.txt

# Match non-DNA characters (errors in sequence)
$ grep "[^ACGTN]" sequence.txt

Predefined character classes:

  • [[:alpha:]] — letters
  • [[:digit:]] — digits (same as [0-9])
  • [[:alnum:]] — letters and digits
  • [[:space:]] — whitespace
  • [[:upper:]] — uppercase letters
  • [[:lower:]] — lowercase letters

Quantifiers

Quantifiers specify how many times a pattern should repeat:

# Match "color" or "colour" (zero or one 'u')
$ grep "colou\?r" text.txt

# Three consecutive A's
$ grep "A\{3\}" sequence.txt

# Two to four T's
$ grep "T\{2,4\}" sequence.txt

# At least three G's
$ grep "G\{3,\}" sequence.txt

# G followed by zero or more C's: matches "G", "GC", "GCC", ...
$ grep "GC*" sequence.txt

Read the last example carefully: a quantifier applies only to the single character (or group) immediately before it. GC* does not mean “GC repeated”; it means “one G, then any number of C’s,” so it matches every line containing a G. To repeat the pair, group it: \(GC\)* in basic regex, or (GC)* with -E.

Warning

In basic grep, quantifiers like ?, +, and {} must be escaped with \. Use grep -E or egrep to avoid escaping.

9.4 Extended Regular Expressions

Extended regular expressions (ERE) add more metacharacters (Table 9.3) and don’t require escaping common quantifiers. Use grep -E or egrep:

Table 9.3: Extended regular expression metacharacters
Character Meaning
+ One or more of preceding
? Zero or one of preceding
{n} Exactly n of preceding
{n,m} Between n and m of preceding
\| Alternation (OR)
() Grouping

Alternation: Matching Multiple Patterns

# Match "start" OR "stop" OR "pause"
$ grep -E "start|stop|pause" commands.txt

# Match any start codon
$ grep -E "ATG|GTG|TTG" sequences.fa

# Match file extensions
$ grep -E "\.(txt|csv|tsv)$" filelist.txt

Grouping

Parentheses group patterns together:

# Match "gray" or "grey"
$ grep -E "gr(a|e)y" text.txt

# Match "ab", "abab", "ababab", etc.
$ grep -E "(ab)+" data.txt

# Match repeated dinucleotide
$ grep -E "(AT){3,}" sequence.txt

The Plus Quantifier

Unlike * (zero or more), + requires at least one match:

# One or more A's followed by one or more T's
$ grep -E "A+T+" sequence.txt

# One or more digits
$ grep -E "[0-9]+" data.txt

# Runs of at least 20 DNA bases (e.g. primer sequences)
$ grep -E "[ACGT]{20,}" primers.txt

9.5 Practical Bioinformatics Examples

Working with FASTA Files

# Count sequences (lines starting with >)
$ grep -c "^>" sequences.fa

# Extract all headers
$ grep "^>" sequences.fa

# Find specific gene headers
$ grep -i "kinase" sequences.fa

# Find sequences by organism
$ grep -E "^>.*Homo sapiens" proteins.fa

# UniProt headers look like ">sp|P69905|HBA_HUMAN ..."; match the
# accession field (the | is a literal pipe, so it must be escaped)
$ grep -E "^>sp\|[A-Z0-9]+\|" uniprot.fasta

# Sequences with no stop codon anywhere on the line
$ grep -v -E "TAA|TAG|TGA" orfs.txt

Searching for Biological Motifs

# Find restriction enzyme sites
$ grep -E "GAATTC" genome.fa         # EcoRI
$ grep -E "GGATCC" genome.fa         # BamHI
$ grep -E "G[ACGT]A[ACGT]TC" genome.fa    # Degenerate site (N = any base)

# Find potential ORFs (start followed by codons, then the first stop)
$ grep -oP "ATG([ACGT]{3})*?(TAA|TAG|TGA)" cds.fa

# Find poly-A tails
$ grep -E "A{10,}" transcripts.fa

# Find microsatellites (dinucleotide repeats)
$ grep -E "(AT){5,}" genome.fa
$ grep -E "(CA){5,}" genome.fa
NoteLazy quantifiers need Perl regex

Notice the *? in the ORF pattern. A plain * is greedy—it grabs as many codons as it can, so ATG([ACGT]{3})*(TAA|TAG|TGA) would run all the way to the last stop codon on the line. Adding ? makes the quantifier lazy, stopping at the first in-frame stop codon. Lazy quantifiers are not part of extended regular expressions (-E); they require Perl-compatible regex, which grep -P provides on GNU grep (Linux, WSL, Talapas). The BSD grep that ships with macOS lacks -P; there, install GNU grep with Homebrew (brew install grep, then use ggrep) or drop the ? and accept greedy matching. Also note that N is not a regex wildcard—write [ACGT] for “any base.”

Analyzing GFF/GTF Files

# Find all genes
$ grep -P "\tgene\t" annotations.gff

# Find genes on chromosome 1
$ grep -P "^chr1\t" genes.gff | grep -P "\tgene\t"

# Extract gene IDs
$ grep -oE "gene_id \"[^\"]+\"" annotations.gtf

# Find protein-coding genes
$ grep "protein_coding" annotations.gtf

Log File Analysis

# Find errors in logs
$ grep -i "error" application.log

# Find specific error codes
$ grep -E "error\s*[0-9]{3}" application.log

# Extract timestamps of errors
$ grep -E "^[0-9]{4}-[0-9]{2}-[0-9]{2}.*error" log.txt

# Count errors per type
$ grep -oE "ERROR: [A-Za-z]+" log.txt | sort | uniq -c

9.6 Combining GREP with Pipes

The real power of grep emerges when combined with other tools through pipes (Chapter 7):

# Count unique patterns
$ grep "^>" sequences.fa | sort | uniq | wc -l

# Extract sequence names without > symbol
$ grep "^>" sequences.fa | cut -c2-

# Find headers, then get organism names
$ grep "^>" proteins.fa | grep -oE "\[.*\]" | sort | uniq -c | sort -rn

# Process FASTA: count bases on each sequence line
$ grep -v "^>" sequence.fa | while read line; do
    echo -n "$line" | wc -c
  done

Extracting Matching Regions

The -o flag prints only the matched portion:

# Extract all email addresses
$ grep -oE "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}" contacts.txt

# Extract all URLs
$ grep -oE "https?://[a-zA-Z0-9./?=_-]+" webpage.html

# Extract gene names from headers
$ grep "^>" genes.fa | grep -oE "gene=[^;]+"

Context Searches

When you need to see what surrounds a match:

# Show 3 lines after each match (including the sequence)
$ grep -A 3 "^>interesting_gene" sequences.fa

# Show lines before and after
$ grep -C 5 "error" logfile.txt

# Extract sequence following a header
$ grep -A 1 "^>target_gene" sequences.fa | grep -v "^>"

9.7 Processing Genomic Data: A Complete Example

Let’s work through a realistic analysis pipeline for human chromosome data.

Downloading the Data

# Download human chromosome 21 from UCSC (~13 MB compressed; the
# smallest human autosome, so it is a good size for practice)
$ wget https://hgdownload.soe.ucsc.edu/goldenPath/hg38/chromosomes/chr21.fa.gz

# Decompress it to a plain FASTA file
$ gunzip -k chr21.fa.gz
$ mv chr21.fa genome.fa

# Or work with a subset without decompressing everything
$ zcat chr21.fa.gz | head -1000 > sample.fa
Note

The UCSC file is soft-masked: repetitive regions are written in lowercase (acgt) rather than uppercase. Patterns like GAATTC will therefore miss sites that fall in masked regions. Add -i to grep, or convert everything to uppercase first with tr 'acgt' 'ACGT', if you want to count every occurrence.

Analyzing Sequence Content

# Count sequences in the file
$ grep -c "^>" genome.fa

# View headers
$ grep "^>" genome.fa | head

# Total bases (excluding headers and newlines)
$ grep -v "^>" genome.fa | tr -d '\n' | wc -c

# Nucleotide frequencies
$ grep -v "^>" genome.fa | tr -d '\n' | fold -w1 | sort | uniq -c

Finding Restriction Enzyme Sites

# Count EcoRI sites (GAATTC)
$ grep -v "^>" genome.fa | tr -d '\n' | grep -o "GAATTC" | wc -l

# Count multiple restriction sites
$ for site in GAATTC GGATCC AAGCTT CTCGAG; do
    count=$(grep -v "^>" genome.fa | tr -d '\n' | grep -o "$site" | wc -l)
    echo "$site: $count"
  done

# Find context around restriction sites (50 bases each side)
$ grep -v "^>" genome.fa | tr -d '\n' | grep -oE ".{50}GAATTC.{50}" | head

Accounting for Soft-Masking

The counts above are undercounts. As the callout in Section 9.7.1 warned, the UCSC file is soft-masked—repeats are written in lowercase—and grep is case-sensitive, so a GAATTC that falls inside a masked repeat is invisible to the pattern. The fix, used throughout the Week 3 and 4 slides’ version of this walkthrough, is to uppercase the sequence with tr before matching:

Figure 9.1: Soft-masking hides restriction sites from a case-sensitive search: grep "GAATTC" finds the uppercase site but misses the lowercase copy inside a masked repeat, until tr 'acgt' 'ACGT' uppercases the sequence first.
# Count EcoRI sites in masked and unmasked sequence alike
$ grep -v "^>" genome.fa | tr 'acgt' 'ACGT' | tr -d '\n' | grep -o "GAATTC" | wc -l

# Nucleotide frequencies with masking folded in (expect a lot of N:
# the first ~5 Mb of chr21 is unassembled)
$ grep -v "^>" genome.fa | tr 'acgtn' 'ACGTN' | tr -d '\n' | fold -w1 | sort | uniq -c
Figure 9.2: Anatomy of the nucleotide-tally pipeline: each stage transforms the stream, from raw FASTA sequence lines to a count of every base.

The same tool answers the classic first question about any genome, its GC content. tr -cd (complement, delete) keeps only the listed characters, so two counts and a division are all it takes; bc does the floating-point arithmetic the shell cannot (Section 10.3.3):

$ total=$(grep -v "^>" genome.fa | tr -d '\n' | tr -cd 'ACGTacgt' | wc -c)
$ gc=$(grep -v "^>" genome.fa | tr -d '\n' | tr -cd 'GCgc' | wc -c)
$ echo "scale=2; 100 * $gc / $total" | bc

Note that counting only ACGT characters in the denominator deliberately excludes the Ns; whether that is what you want depends on the question. A single-pipeline variant of the same calculation appears in Section 7.8.7.

Letting the Masking Work for You: ORF-like Stretches

So far the soft-masking has been an obstacle to defeat with tr. It can also be information: lowercase marks repeat-derived sequence, and the uppercase remainder is the non-repeat fraction where genes mostly live. An uppercase-only pattern therefore searches only the unmasked sequence—no tr needed, because a single lowercase base breaks any match:

# ATG, then 20 or more in-frame codons, then a stop - masked repeats cannot match
$ grep -v "^>" genome.fa | tr -d '\n' | \
    grep -oE "ATG([ACGT]{3}){20,}(TAA|TAG|TGA)" > orf_like.txt

# grep -o writes one match per line, so counting the hits is just counting lines
$ wc -l orf_like.txt

# and awk ranks the three longest
$ awk '{ print length($0) }' orf_like.txt | sort -n | tail -3

Call these “ORF-like stretches,” not ORFs. The greedy * runs each match to the last in-frame stop before the pattern breaks, only one strand is searched, and real gene finding must handle splicing, codon bias, and much more. But twenty-plus codons of open reading frame in unmasked sequence is a respectable first filter—and the deeper lesson is worth keeping: sometimes annotation already embedded in the data (here, the case of the letters) can stand in for computation you would otherwise have to do yourself. The Week 4 slides run exactly this pipeline on chromosome 21.

Finding Simple Sequence Repeats

# Dinucleotide repeats
$ grep -v "^>" genome.fa | tr -d '\n' | grep -oE "(AT){5,}" | wc -l
$ grep -v "^>" genome.fa | tr -d '\n' | grep -oE "(CA){5,}" | wc -l

# Homopolymer runs (consecutive same base)
$ grep -v "^>" genome.fa | tr -d '\n' | grep -oE "A{10,}" | wc -l
$ grep -v "^>" genome.fa | tr -d '\n' | grep -oE "T{10,}" | wc -l

Stream Processing Large Files

Process data without loading it entirely into memory:

# Download, decompress, and analyze in one pipeline
$ curl -sL https://hgdownload.soe.ucsc.edu/goldenPath/hg38/chromosomes/chr21.fa.gz | \
  gunzip -c | \
  grep -v "^>" | \
  tr -d '\n' | \
  grep -o "GAATTC" | \
  wc -l

# Process compressed file directly
$ zgrep -c "^>" chr21.fa.gz

9.8 Common Patterns Reference

Table 9.4 is a quick reference for frequently needed patterns:

Table 9.4: Common regular expression patterns for bioinformatics
Pattern What It Matches
^> FASTA headers
^# Comment lines
^$ Empty lines
[ACGT]+ DNA sequence
[ACGU]+ RNA sequence
[A-Z]{3} Uppercase triplet (codon)
ATG([ACGT]{3})* Potential ORF
(TAA\|TAG\|TGA) Stop codons
A{n,} Poly-A of at least n
\t Tab character
\s+ One or more whitespace
[0-9]+\.[0-9]+ Decimal number

9.9 Editing Text Streams with sed

grep tells you which lines match a pattern, but it never changes them. When you want to edit text as it flows through a pipe—replace one string with another, delete unwanted lines, or pull out just part of each line—the tool is sed, the stream editor. Like grep, sed reads input one line at a time, applies your instructions, and prints the result. It never opens an interactive window; you describe the edit on the command line and sed performs it on every line.

Substitution: s/old/new/

The workhorse sed command is substitution. The syntax is s/pattern/replacement/, where pattern is a regular expression using the same metacharacters you learned for grep:

# Replace the first "chr" on each line with "chromosome"
$ sed 's/chr/chromosome/' regions.bed

# Replace tabs with commas (first tab on each line only!)
$ sed 's/\t/,/' data.tsv

By default s replaces only the first match on each line. Add the g (global) flag to replace every occurrence:

# Replace every tab, converting TSV to CSV
$ sed 's/\t/,/g' data.tsv > data.csv

# Convert a DNA sequence to RNA: every T becomes U
$ grep -v "^>" gene.fa | sed 's/T/U/g'

# Anchors work too: add a prefix to the start of every line
$ sed 's/^/sample_/' ids.txt

The / character is only a convention. If your pattern or replacement itself contains slashes (file paths, for example), pick a different delimiter such as | or #, and sed will use whatever character follows the s:

# Rewrite a directory prefix without escaping every slash
$ sed 's|/home/old_user/|/home/new_user/|g' paths.txt

Printing and Deleting Lines

sed normally prints every line, edited or not. The -n option suppresses that automatic output, and the p command prints explicitly. Together they let you select lines by number—something head and tail can do only awkwardly:

# Print lines 10 through 20 (inclusive)
$ sed -n '10,20p' results.txt

# Print only the lines where a substitution actually happened
$ sed -n 's/ERROR/**ERROR**/p' pipeline.log

An address can be a line number, a range, or a regular expression between slashes. Combined with the d (delete) command, this is a concise way to strip unwanted lines:

# Delete comment lines
$ sed '/^#/d' annotations.gff

# Delete blank lines
$ sed '/^$/d' notes.txt

# Delete the header line (line 1) of a table
$ sed '1d' measurements.tsv

Note that sed '/^#/d' and grep -v '^#' produce the same output; use whichever reads more naturally in your pipeline.

Capture Groups

With -E (extended regex, exactly as in grep -E), parentheses capture part of the match, and \1, \2, … refer back to those captured pieces in the replacement. This is how you rearrange text rather than just replace it:

# Turn "sample_12_rep3" into "12-3"
$ echo "sample_12_rep3" | sed -E 's/sample_([0-9]+)_rep([0-9]+)/\1-\2/'
12-3

# Shorten FASTA headers to just the first word
$ sed -E 's/^>([^ ]+).*/>\1/' proteins.fa

# Swap the first two tab-separated columns
$ sed -E 's/^([^\t]+)\t([^\t]+)/\2\t\1/' data.tsv

Read the second example carefully: ^> anchors to the header, ([^ ]+) captures everything up to the first space, .* swallows the rest of the line, and the replacement >\1 writes back only the captured identifier.

Editing Files in Place

Everything above writes to standard output and leaves the input file untouched. The -i option instead edits the file in place—and here two warnings are in order.

Warningsed -i overwrites your file

There is no undo. A typo in the pattern can silently corrupt a file you spent hours creating. Write to a new file until you are confident the edit does what you expect:

# Safe: check the result first
$ sed 's/T/U/g' gene.fa > gene_rna.fa
$ head gene_rna.fa

# Only once you are sure
$ sed -i 's/T/U/g' gene.fa
ImportantLinux and macOS disagree about -i

GNU sed (Linux, WSL, Talapas) and BSD sed (macOS) interpret -i differently. On macOS, -i requires an argument giving a backup suffix, and you must pass an empty string to mean “no backup”:

# Linux / WSL / Talapas
$ sed -i 's/a/b/' file.txt

# macOS
$ sed -i '' 's/a/b/' file.txt

If you use the Linux form on a Mac, sed treats s/a/b/ as the backup suffix and then complains that file.txt is not a valid command. A portable alternative that works everywhere is to give a real suffix, e.g. sed -i.bak 's/a/b/' file.txt, which also leaves you a backup. On macOS you can also install GNU sed with Homebrew (brew install gnu-sed, invoked as gsed).

9.10 Working with Columns: awk

Much biological data is tabular: GFF and BED annotations, VCF variant calls, and the tab- or comma-separated spreadsheets that come off plate readers and flow cytometers. cut can pull out columns, but it cannot do arithmetic on them or decide which rows to keep based on a column’s value. awk (named for its authors Aho, Weinberger, and Kernighan) is a small programming language built exactly for this: it splits each line into fields and lets you filter, compute, and reformat by column.

The pattern { action } Model

An awk program is a list of rules of the form pattern { action }. For every input line, awk tests each pattern; when a pattern matches, it runs the action. Either half is optional: a pattern alone prints matching lines, and an action alone runs on every line.

# No pattern: print the first column of every line
$ awk '{ print $1 }' annotations.gff

# Pattern only: print whole lines where column 3 is "gene"
$ awk '$3 == "gene"' annotations.gff

# Both: print columns 1, 4, and 5 for gene lines
$ awk '$3 == "gene" { print $1, $4, $5 }' annotations.gff

Inside an action, $1, $2, … refer to the fields of the current line, $0 is the whole line, and $NF is the last field (NF is the number of fields). The variable NR is the current line (record) number. By default fields are separated by any run of spaces or tabs, so awk works on a GFF or BED file without any extra options.

Choosing the Delimiter

For CSV files, tell awk to split on commas with -F','. If the field separator is a single tab and some columns may be empty, use -F'\t' so that consecutive tabs are not collapsed into one:

# Third column of a CSV
$ awk -F',' '{ print $3 }' od600_readings.csv

# Tab-delimited with possibly empty fields
$ awk -F'\t' '{ print $1, $9 }' annotations.gff

Skipping Headers and Filtering Rows

Because NR counts lines, NR > 1 is the idiom for “skip the header.” Patterns can be any comparison or regular expression, and several can be combined with && (and) or || (or):

# Skip the header, print sample and OD600 columns
$ awk -F',' 'NR > 1 { print $1, $4 }' od600_readings.csv

# Rows where the OD600 (column 4) exceeds 0.8
$ awk -F',' 'NR > 1 && $4 > 0.8' od600_readings.csv

# BED intervals on chr21 longer than 1 kb
$ awk '$1 == "chr21" && ($3 - $2) > 1000' peaks.bed

# Genes on the minus strand (regex match on column 7)
$ awk '$3 == "gene" && $7 ~ /-/' annotations.gff

# Rows where column 2 is exactly "control"
$ awk '$2 == "control"' samples.tsv

# Number every line (like cat -n), then the whole line
$ awk '{ print NR, $0 }' data.txt

Actions can also transform values on the way out—awk does arithmetic on any field that looks like a number:

# Print column 1 and column 2 converted from kb to bp
$ awk '{ print $1, $2 * 1000 }' lengths.tsv

Range Patterns

A pattern can also be a pair of patterns separated by a comma. awk '/start/,/stop/' switches on at the first line matching the first pattern and off at the next line matching the second, printing everything in between—the same idea as sed’s two-address ranges in Section 9.9.2, but with the full power of awk’s patterns. The classic use is pulling one record out of a multi-sequence FASTA file by name:

# All lines from the chr21 header up to and including the chr22 header
$ awk '/^>chr21$/,/^>chr22$/' hg38.fa > chr21_plus_one_line.fa

# Trim the chr22 header that closes the range
$ awk '/^>chr21$/,/^>chr22$/' hg38.fa | head -n -1 > chr21_from_genome.fa

The range is inclusive at both ends, which is why the second command trims the final line (head -n -1 prints all but the last line; it is a GNU extension, so on macOS use sed '$d' instead). One caution: a range pattern re-opens if the start pattern matches again later in the file, so this idiom is reliable for extracting a single uniquely named record, not for every selection task.

Computing Sums and Means with END

Two special patterns run outside the line-by-line loop: BEGIN before any input is read and END after the last line. END is where you report a total that you accumulated along the way. Variables spring into existence at first use and start at zero:

# Total length of all BED intervals
$ awk '{ total += $3 - $2 } END { print total }' peaks.bed

# Mean of column 4, skipping the header
$ awk -F',' 'NR > 1 { sum += $4; n++ } END { print sum / n }' od600_readings.csv

# Count gene features (equivalent to grep -c but by column)
$ awk '$3 == "gene" { n++ } END { print n }' annotations.gff

Controlling Output: OFS

When you print several fields separated by commas, awk joins them with the output field separator, OFS, which defaults to a single space. Set it in a BEGIN block (or with -v OFS=...) to produce tab- or comma-separated output:

# Convert a GFF to a simple three-column TSV
$ awk 'BEGIN { OFS = "\t" } $3 == "gene" { print $1, $4, $5 }' annotations.gff

# Convert TSV to CSV (rebuilding $0 forces awk to apply the new OFS)
$ awk -F'\t' -v OFS=',' '{ $1 = $1; print }' measurements.tsv

For fully formatted output, awk also has printf, with the C-style format strings you may know from other languages: %s for a string, %d for an integer, %f for a fixed number of decimals. A number between the % and the letter sets the field width, a - left-justifies, and .1 limits the decimals, so %-6s %8.1f means “the string in a left-justified 6-character field, then the number right-justified in 8 characters with one decimal”—which is how the gene_means.sh script of Chapter 10 lines its group means up in neat columns. Unlike print, printf adds no newline of its own; end the format string with \n.

# Group means in aligned columns, one decimal place
$ awk -F',' 'NR > 1 { sum[$2] += $3; n[$2]++ }
             END { for (t in sum) printf "%-8s %6.2f\n", t, sum[t] / n[t] }' od600_readings.csv
control    0.44
IPTG       0.82

A Bioengineering Example: Mean per Treatment

Suppose a plate reader exports a CSV with sample name, treatment, and OD600 columns:

sample,treatment,od600
S01,control,0.42
S02,control,0.45
S03,IPTG,0.81
S04,IPTG,0.79
S05,IPTG,0.85

To compute the mean OD600 for each treatment we need one running sum and one count per treatment. awk’s associative arrays—arrays indexed by a string instead of a number—make this a one-liner:

$ awk -F',' 'NR > 1 { sum[$2] += $3; n[$2]++ }
             END { for (t in sum) print t, sum[t] / n[t] }' od600_readings.csv
control 0.435
IPTG 0.816667

For each data line, sum[$2] accumulates the OD600 under the treatment name in column 2 and n[$2] counts the samples. In END, for (t in sum) loops over every treatment seen (in no particular order—pipe the output through sort if you want it alphabetical). This same pattern—group by one column, aggregate another—answers a large share of everyday questions about tabular data, and it runs at the speed of the disk on files far too big to open in a spreadsheet.

Choosing the Right Tool

A useful rule of thumb: grep finds lines, sed edits text, awk works with columns. They overlap—each can delete comment lines, for instance—but each is clearest at its own job, and they chain together naturally in a pipeline (Table 9.5).

Table 9.5: Comparing grep, sed, and awk
Tool Best at Typical call Example
grep Selecting lines that match a pattern grep -E 'pattern' file grep -c "^>" seqs.fa
sed Editing text within lines sed 's/old/new/g' file sed 's/T/U/g' gene.fa
awk Filtering and computing on columns awk 'pattern { action }' file awk '$3 == "gene"' genes.gff
Figure 9.3: Choosing the right tool: grep finds lines, sed edits text within lines, awk filters and computes on columns, and sort | uniq -c counts groups — and the tools chain together naturally in a pipeline.

Putting It Together

The real power of these tools comes from combining them with the pipeline tools of Chapter 7. Each of the following answers a question you are likely to ask of a real data file in a single line:

# How many distinct gene names are in column 1 (skipping the header)?
$ tail -n +2 data.tsv | cut -f1 | sort | uniq | wc -l

# The 20 rows with p < 0.05 (column 5) with the largest effect size (column 3)
$ awk '$5 < 0.05' results.tsv | sort -k3,3 -nr | head -20

# Convert a CSV to tab-delimited and keep three columns
$ sed 's/,/\t/g' data.csv | cut -f1,3,5 > subset.tsv

# How many times does each value in column 2 occur, most common first?
$ cut -f2 data.tsv | sort | uniq -c | sort -nr

# How many sequences per chromosome in a FASTA whose headers look like ">chr1:1000-2000"?
$ grep "^>" sequences.fasta | cut -d":" -f1 | sort | uniq -c

# Which feature types does a GFF file contain, and how many of each?
$ grep -v "^#" annotations.gff | cut -f3 | sort | uniq -c

Notice the recurring shape: select lines (grep, awk, tail -n +2), extract a column (cut), sort so that identical values are adjacent, then count (uniq -c, wc -l). Once this idiom is in your fingers, most “how many of each?” questions take seconds.

9.11 The Same Patterns in R and Python

From Lecture 05 onward, the regular-expression language you have been using with grep, sed, and awk moves inside R and Python. The patterns themselves — anchors, character classes, quantifiers, alternation — are unchanged; only the functions wrapped around them are new. The map:

Table 9.6: Pattern-matching functions across the three languages
Job Shell R Python (import re)
Which elements match? grep grepl() (TRUE/FALSE), grep() (positions) re.search()
Pull out the match itself grep -o regmatches() + gregexpr() re.findall()
Substitute sed 's/a/b/' sub() (first), gsub() (all) re.sub()
Split on a pattern awk -F strsplit() re.split()

One practical difference between the languages is escaping. In an R string, a backslash must itself be escaped, so the digit class \d is written "\\d". Python avoids the problem with raw strings: write every pattern with an r prefix, as in r"\d", and backslashes pass through untouched. Make the raw-string prefix a habit — a Python regex without it will eventually bite you.

R’s pattern functions

sequences <- c("ATGAAATAA", "CCCGGG", "ATGTTTTGA")

grepl("^ATG", sequences)           # which elements start with ATG?
# [1]  TRUE FALSE  TRUE
grep("^ATG", sequences)            # their positions ...
# [1] 1 3
grep("^ATG", sequences, value = TRUE)   # ... or the elements themselves
# [1] "ATGAAATAA" "ATGTTTTGA"

gsub("T", "U", sequences)          # substitute ALL matches (sub() does the first)
# [1] "AUGAAAUAA" "CCCGGG"    "AUGUUUUGA"

# Extract every match, like grep -o: regmatches() + gregexpr()
seq <- "ATGAAATAACCCATGTTTTGA"
regmatches(seq, gregexpr("ATG([ACGT]{3})*?(TAA|TAG|TGA)", seq))[[1]]
# [1] "ATGAAATAA" "ATGTTTTGA"

strsplit("convA_rep1_day3", "_")[[1]]   # split on a pattern, like awk -F
# [1] "convA" "rep1"  "day3"

The tidyverse’s stringr package (Section 14.1.1) wraps these in functions with consistent names and argument order — str_detect(), str_subset(), str_replace_all(), str_extract_all(), str_split() — and is what most modern R code uses; the base functions above are what you will meet in older code and in other people’s scripts.

Python’s re module

import re

sequences = ["ATGAAATAA", "CCCGGG", "ATGTTTTGA"]

[s for s in sequences if re.search(r"^ATG", s)]     # keep the matching elements
# ['ATGAAATAA', 'ATGTTTTGA']

re.sub(r"T", "U", "ATGAAATAA")                      # substitute all matches
# 'AUGAAAUAA'

seq = "ATGAAATAACCCATGTTTTGA"
re.findall(r"ATG(?:[ACGT]{3})*?(?:TAA|TAG|TGA)", seq)   # extract every match
# ['ATGAAATAA', 'ATGTTTTGA']

re.split(r"_", "convA_rep1_day3")                   # split on a pattern
# ['convA', 'rep1', 'day3']

Two Python details to note. First, (?: ... ) is a non-capturing group: it groups for structure, like ( ... ), but keeps re.findall() returning the whole match rather than the group’s contents. Second, for a pandas column the .str accessor runs a regex down every row at once — df["Individual"].str.contains(r"1$"), .str.replace(), .str.extract() — which is the data-frame version of grepl() on an R vector.

One pattern, three languages

The ORF-like pattern from Section 9.5.2 works everywhere; only the function changes:

grep -oP "ATG([ACGT]{3})*?(TAA|TAG|TGA)" cds.fa                        # shell
regmatches(seq, gregexpr("ATG([ACGT]{3})*?(TAA|TAG|TGA)", seq))[[1]]   # R
re.findall(r"ATG(?:[ACGT]{3})*?(?:TAA|TAG|TGA)", seq)                  # Python

This is the usual division of labor: prototype and explore a pattern with grep on the command line, where feedback is immediate, then move it into an R or Python script once it needs to feed counts, tables, or figures (the fasta_stats scripts of Lecture 05 do exactly that). The pattern syntax reference in Appendix D applies to all three.

9.12 Running Example: The Stickleback Data

NoteRunning example

This section applies the chapter to the course dataset described in Section 1.5 (80 stickleback, 4 treatment groups, 600 genes). The same steps appear on the Week 3 and 4 slides, so the book and the lecture can be read side by side.

The stickleback table is small enough to read, but the same commands scale to files with millions of lines. grep -w matches whole words, so the treatment label in the third column selects rows without also matching the combined label in the fourth; a regular expression anchored at the start of the line picks out the individuals whose number is exactly 1:

cd ~/projects/comptools
grep -w "convB" data/Stickle_RNAseq.tsv | cut -f 1      # individuals: genotype B, conventional
grep -E "^[A-Z]+1\b" data/Stickle_RNAseq.tsv | cut -f 1-3   # AC1, AM1, BC1, BM1
grep -c mono data/Stickle_RNAseq.tsv; grep -cw mono data/Stickle_RNAseq.tsv   # both 40 - why?

The last line is worth pausing on. Without -w, mono also matches monoA and monoB in the fourth column, but every line that contains those also contains the bare word mono in the third, so the counts agree. Small experiments like this, comparing two flags on data you understand, are how you learn what a flag really does.

sed and awk complete the picture. The fourth column is named Geno&Micro, and the ampersand will annoy every tool downstream, so the first sed command below renames it in a new file, leaving the raw data untouched (Section 8.5.4). awk then computes the mean of the first gene column over all fish, and, using an associative array keyed on the genotype, per genotype:

sed '1s/Geno&Micro/Geno_Micro/' data/Stickle_RNAseq.tsv > results/stickle_clean.tsv
awk -F'\t' 'NR > 1 { sum += $5; n++ } END { print "Gene1 mean:", sum/n }' results/stickle_clean.tsv
awk -F'\t' 'NR > 1 { sum[$2] += $5; n[$2]++ } END { for (g in sum) print g, sum[g]/n[g] }' results/stickle_clean.tsv

The mean count of the first gene is about 14 in genotype A fish and about 1.4 in genotype B, a tenfold difference visible from three lines of awk. Turning this into a reusable script is the job of Chapter 10.

9.13 Summary

This chapter covered pattern matching with grep and regular expressions, and the two companion tools that edit and compute on the text grep finds:

  • grep finds lines matching patterns in files or streams
  • Basic regex uses metacharacters: ., *, ^, $, []
  • Extended regex (grep -E) adds: +, ?, |, {}
  • Character classes match sets of characters
  • Anchors match positions (start/end of line)
  • Combining grep with pipes enables powerful analysis
  • Genomic data analysis uses these tools for motif searching
  • sed edits text streams: s/old/new/g substitutes, -n 'a,bp' prints ranges, /pattern/d deletes, and -E with \1 rearranges captured text
  • sed -i edits files in place—write to a new file until you are confident, and remember that macOS needs sed -i ''
  • awk works with columns: $1 ... $NF, NR, -F',', NR > 1 to skip headers, END blocks for sums and means, and associative arrays for per-group summaries
  • Rule of thumb: grep finds lines, sed edits text, awk works with columns

Regular expressions are a foundational skill. While the syntax can seem cryptic initially, investment in learning regex pays off enormously across all aspects of computational biology. Keep Appendix D at hand while you practice.

Tip

For a comprehensive reference of regular expression patterns and grep command syntax, see Appendix D. For other Unix commands, see Appendix B.

Additional Reading

To deepen your understanding of regular expressions:

  • Regular-Expressions.info — A comprehensive tutorial and reference site for regex
  • RegexOne — An interactive tutorial for learning regular expressions step by step
  • regex101.com — An online regex tester with explanations of each component
  • “Mastering Regular Expressions” by Jeffrey Friedl (Friedl, 2006) — The definitive book on regex theory and practice

For bioinformatics-specific pattern matching:

Exercises

TipPractice Exercises

Exercise 1: Basic Pattern Matching

Create a file with various text and practice: 1. Find all lines containing “the” (case-insensitive) 2. Find lines starting with a capital letter 3. Find lines ending with a period 4. Count empty lines

Exercise 2: Biological Patterns

Using a FASTA file: 1. Count the number of sequences 2. Extract headers for sequences longer than 1000 bp (hint: you’ll need multiple commands) 3. Find all sequences containing the pattern “TATAAA” (TATA box) 4. Count how many sequences have “chromosome” in their header

Exercise 3: Regular Expression Practice

Write grep commands to match: 1. Phone numbers in format: (555) 555-5555 2. Email addresses 3. Dates in YYYY-MM-DD format 4. DNA sequences that are exactly 20 bases long

Exercise 4: Genomic Analysis Pipeline

Download a bacterial genome and: 1. Calculate the genome size 2. Count all restriction sites for at least 3 different enzymes 3. Find the longest homopolymer run 4. Calculate GC content using only Unix tools 5. Document your pipeline in a shell script

Exercise 5: chr21 Self-Check (from the Week 4 slides)

Using the chr21.fa file from Section 9.7 and only grep -v, tr, fold, sort, uniq, and wc:

  1. Count the total number of bases in chr21.fa
  2. Calculate the GC content (percent of bases that are G or C)
  3. Find runs of the dinucleotide repeat AT five or more units long
  4. What fraction of the chromosome is N?

Build each pipeline one command at a time and check the output at every stage. Remember the soft-masking (Section 9.7.4): uppercase first with tr 'acgt' 'ACGT'. Hints: for (1) start from the total-bases pipeline in Section 9.7.2; for (2) and (4), tr -cd 'GCgc' and tr -cd 'Nn' keep only the listed characters, and dividing by the answer to (1) gives the fraction; for (3), grep -oE "(AT){5,}" prints each run on its own line.