Lecture 06

R and Python Fundamentals 2: Data Frames, the Tidyverse and Figures

Bill Cresko

2026-11-02

Book reading for this lecture

Goals for today

  • Data frames:

    • creating

    • inspecting

    • summarizing

  • Creating data sets using random draws

  • The Tidyverse

  • ggplot2: the grammar of graphics

  • Example figures for bioengineering data

Example 1 - a panel of figures

Figure 1: Read Counts of Four Genes. RNA-seq read counts for Gene1-Gene4 in n = 80 threespine stickleback (20 fish per host genotype x microbiota combination), split by host genotype and colored by microbiota treatment. Violins show each group’s full count distribution; each point is one fish, jitter-dodged to reduce overplotting. Panel y-axes are scaled independently.

Example 2 - A range of 600 genes at once

Figure 2: The mean-variance relationship for all 600 genes. Each point is one gene, placed by the mean and the variance of its read counts across the 80 fish, on log-log axes. The dashed line marks variance = mean, the expectation under Poisson sampling; nearly all genes fall above it, the overdispersion typical of RNA-seq counts.

Data frames

Reading and writing files

Job R Python
Read a table read.delim("data/f.tsv", check.names = FALSE) pd.read_csv("f.tsv", sep="\t")
Write results write.csv(x, "results/f.csv", row.names = FALSE) x.to_csv("f.csv", index=False)
Save a figure ggsave("results/fig.svg") · svg() … dev.off() plt.savefig("results/fig.svg")
One column / rows by condition df$col · df[df$g == "A", ] df["col"] · df[df["g"] == "A"]

Reading and writing files

  • Plain-text tables (CSV/TSV) in, derived results out:
    • write into results/
    • never over anything in data/ (raw data is read-only)
  • Vector figure formats (.svg, .pdf) scale forever
    • use them for papers
    • raster (.png) needs 300 dpi
  • Every detail - formats, URLs, Excel, script checklists - is in the Week 6 companion notebooks (sections 2-4) and Ch. 11/Ch. 12

Creating Data Frames in R and Python

  • Data can come from a file, or you can generate it
    • Directly or
    • With random draws - that’s the next section
  • For now, let’s make a small dummy dataset by hand, to get used to dealing with data frames
  • Set up three variables (hydrogel_concentration, compression and conductivity) as vectors
  • In Python the data frame comes from pandas (import pandas as pd); it has the same idea - rows are samples, columns are variables

Creating Data Frames in R and Python

hydrogel_concentration <- factor(c("low", "high", "high", "high", "medium", "medium", "medium","low"))
compression <- c(3.4, 3.4, 8.4, 3, 5.6, 8.1, 8.3, 4.5)
conductivity <- c(0, 9.2, 3.8, 5, 5.6, 4.1, 7.1, 5.3)
  • Create a data frame where vectors become columns
mydata <- data.frame(hydrogel_concentration, compression, conductivity)
row.names(mydata) <- c("Sample_1", "Sample_2", "Sample_3", "Sample_4",
                       "Sample_5", "Sample_6", "Sample_7", "Sample_8")
import pandas as pd

hydrogel_concentration = pd.Categorical(["low", "high", "high", "high", "medium", "medium", "medium", "low"])
compression = [3.4, 3.4, 8.4, 3, 5.6, 8.1, 8.3, 4.5]
conductivity = [0, 9.2, 3.8, 5, 5.6, 4.1, 7.1, 5.3]
  • Create a DataFrame from a dictionary of column name: values
mydata = pd.DataFrame({"hydrogel_concentration": hydrogel_concentration,
                       "compression": compression,
                       "conductivity": conductivity})
mydata.index = [f"Sample_{i}" for i in range(1, 9)]   # row names = the index
mydata.head(3)
#          hydrogel_concentration  compression  conductivity
# Sample_1                    low          3.4           0.0
# Sample_2                   high          3.4           9.2
# Sample_3                   high          8.4           3.8
  • Now you have a hand-made data frame with row names
  • A data frame is a list of equal-length vectors:
    • one name per column
    • one type per column
  • Take a look at it in Positron’s Variables pane, or open it in the Data Explorer (R and Python)

The stickleback table as a data frame

stickle <- read.delim("../Data_Folder/Stickle_RNAseq.tsv", check.names = FALSE)
dim(stickle)                          # 80 fish x 604 columns
[1]  80 604
stickle[1:4, 1:7]                     # first rows, first columns
  Individual Genotype   Microbiota Geno&Micro Gene1 Gene2 Gene3
1        AC1        A conventional      convA    16    15    71
2        AC2        A conventional      convA     7     6    72
3        AC3        A conventional      convA     9     8    22
4        AC4        A conventional      convA     8    18    15
table(stickle$Genotype, stickle$Microbiota)
   
    conventional mono
  A           20   20
  B           20   20
summary(stickle$Gene1)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  0.000   1.000   3.500   7.675  12.250  39.000 
import pandas as pd

stickle = pd.read_csv("../Data_Folder/Stickle_RNAseq.tsv", sep="\t")
stickle.shape                          # (80, 604)
stickle.iloc[:4, :7]
pd.crosstab(stickle["Genotype"], stickle["Microbiota"])
stickle["Gene1"].describe()
  • First questions for any new table: how big (dim / .shape), what’s in it (first rows), how do the groups break down (table / pd.crosstab), what does one column look like (summary / .describe)
  • check.names = FALSE keeps the column name Geno&Micro exactly as it is in the file

Summary statistics - overall and by group

aggregate(Gene1 ~ Genotype + Microbiota, data = stickle, FUN = median)
  Genotype   Microbiota Gene1
1        A conventional  11.5
2        B conventional   1.0
3        A         mono  13.5
4        B         mono   2.0
sapply(stickle[, c("Gene1", "Gene2", "Gene3")], mean)
   Gene1    Gene2    Gene3 
  7.6750  42.0375 168.4375 
stickle.groupby(["Genotype", "Microbiota"])["Gene1"].median()
# Genotype  Microbiota
# A         conventional    11.5
#           mono            13.5
# B         conventional     1.0
#           mono             2.0
stickle[["Gene1", "Gene2", "Gene3"]].mean()
  • aggregate()’s formula - response ~ groups - is the standard R way of saying “this, broken down by that”
  • The tidyverse’s group_by() |> summarize() and pandas’ .groupby().agg() generalize this to many statistics at once - coming up in a few slides

Random draws from known distributions

Why draw random numbers?

  • So far data have come from files or from typing - the third source is to generate them from a known distribution
  • Why a scientist does this constantly:
    • Test your analysis first: run it on data where you know the right answer, before trusting it on real data
    • Build a null model: what would the data look like if nothing were going on?
    • Plan an experiment: simulate it - how many samples before the effect you expect becomes visible?

Why draw random numbers?

  • Each named distribution is a different data-generating machine:
    • uniform - every value in a range equally likely (positions, backgrounds)
    • normal - measurements that scatter symmetrically around a mean (mass, OD, expression on a log scale)
    • Poisson - counts of events in a fixed window (sequencing reads, cells per field, spikes per second)
  • set.seed() (R) / np.random.default_rng(seed) (Python) make the draws reproducible:
    • same seed, same “random” numbers
    • no seed, different “random” numbers each time

The functions and their arguments

Distribution R Python (numpy generator)
Uniform runif(n, min = 0, max = 100) rng.uniform(0, 100, n)
Normal rnorm(n, mean = 50, sd = 10) rng.normal(50, 10, n)
Poisson rpois(n, lambda = 5) rng.poisson(5, n)
  • The first argument is always n, how many draws you want
  • The remaining arguments are the distribution’s parameters - the knobs on the machine:
    • uniform: the min and max of the range
    • normal: the mean (center) and sd (spread)
    • Poisson: lambda, the average count per window (which is also its variance!)
  • Set up once, draw everywhere: set.seed(2026) in R; rng <- np.random.default_rng(2026) in Python, then every draw comes from rng
  • Each R function has siblings you’ll meet in the statistics sequence: dnorm() (the density), pnorm() (cumulative probability), qnorm() (quantiles)

Three distributions, drawn and plotted

set.seed(2026); n <- 2000
dir.create("results", showWarnings = FALSE)                 # the folder must exist
svg("results/distributions.svg", width = 10, height = 3.6)   # open the file device
par(mfrow = c(1, 3), mar = c(4, 4, 2, 1))      # a 1 x 3 grid of panels
hist(runif(n, 0, 100),  breaks = 40, col = "#31688e", border = "white",
     main = "Uniform(0, 100)",  xlab = "value")
hist(rnorm(n, 50, 10),  breaks = 40, col = "#440154", border = "white",
     main = "Normal(50, 10)",   xlab = "value")
hist(rpois(n, 5),       breaks = seq(-0.5, 20.5, 1), col = "#35b779",
     border = "white", main = "Poisson(5)",    xlab = "count")
dev.off()                                       # close it - the file is written
import numpy as np
import matplotlib.pyplot as plt
from pathlib import Path

rng = np.random.default_rng(2026)
n = 2000
Path("results").mkdir(exist_ok=True)       # the folder must exist
draws = {"Uniform(0, 100)": (rng.uniform(0, 100, n), "#31688e"),
         "Normal(50, 10)":  (rng.normal(50, 10, n),  "#440154"),
         "Poisson(5)":      (rng.poisson(5, n),      "#35b779")}

fig, ax = plt.subplots(1, 3, figsize=(10, 3.6))    # the grid, as an array of axes
for a, (name, (d, c)) in zip(ax, draws.items()):
    a.hist(d, bins=40, color=c, edgecolor="white")
    a.set_title(name)
fig.tight_layout()
plt.savefig("results/distributions.svg")
plt.show()
  • Same machine, three settings: uniform spreads flat, normal mounds around its mean, Poisson is discrete counts piling near lambda
  • Base R saves differently from ggplot: open a file device → plot → dev.off() - forget dev.off() and the file is empty
  • par(mfrow = c(1, 3)) splits the plotting device into panels; each hist() call fills the next one (plt.subplots(1, 3) hands Python the grid as an array of axes)
  • Change n, the parameters and breaks - this one figure is a little simulation lab (notebook sections 8-10)

The Tidyverse for Data Wrangling

The tidyverse family of packages

  • A family of R packages that share one grammar: data frames in, data frames out
  • Loading tidyverse loads all of them at once

The tidyverse packages and their pandas counterparts

Package Purpose pandas / Python equivalent
dplyr Data manipulation (filter, select, mutate, summarise) pandas methods
ggplot2 Data visualization matplotlib, seaborn, plotnine
tidyr Reshaping data (pivot_longer, pivot_wider, separate) melt, pivot
readr Fast reading of CSV, TSV files pd.read_csv
stringr String manipulation str methods, re
purrr Functional programming tools (map) list comprehensions
  • Today we need three of them: readr (read), tidyr (reshape), and ggplot2 (draw) - with dplyr verbs in between

A taste of the tidyverse: the pipe |>

library(tidyverse)

mydata |>
  filter(compression > 4) |>                            # keep rows
  mutate(ratio = conductivity / compression) |>         # add a column
  group_by(hydrogel_concentration) |>                   # define groups
  summarise(n = n(), mean_ratio = mean(ratio))          # one row per group
import pandas as pd

(mydata
   .query("compression > 4")                                  # keep rows
   .assign(ratio=lambda d: d.conductivity / d.compression)    # add a column
   .groupby("hydrogel_concentration", observed=True)          # define groups
   .agg(n=("ratio", "size"), mean_ratio=("ratio", "mean"))    # one row per group
)

Note

  • Read |> as “and then”.
  • The older %>% pipe does the same thing.
  • In pandas the dot does the same job - wrap the chain in ( ) so you can put each step on its own line.

The verbs, side by side

Verb What it does
filter() Keep rows matching a condition
select() Keep or drop columns
mutate() Add or change columns
arrange() Sort rows
group_by() + summarise() Summaries per group
dplyr verb pandas method
filter() .query("...") or df[df.col > 4]
select() df[["a", "b"]] / .drop(columns=)
mutate() .assign(new=...)
arrange() .sort_values("col")
group_by() + summarise() .groupby("col").agg(...)
  • Five verbs cover most everyday wrangling
  • Each takes a data frame and returns a data frame, which is what makes them chainable

Wide to long, then summarize

library(tidyverse)
stickle <- read_tsv("../Data_Folder/Stickle_RNAseq.tsv", show_col_types = FALSE) |>
  rename(geno_micro = `Geno&Micro`)          # fix the awkward name once

stickle_long <- stickle |>
  pivot_longer(starts_with("Gene"), names_to = "gene", values_to = "count")
stickle_long                                  # 80 x 600 = 48,000 rows: one per fish x gene
# A tibble: 48,000 × 6
   Individual Genotype Microbiota   geno_micro gene   count
   <chr>      <chr>    <chr>        <chr>      <chr>  <dbl>
 1 AC1        A        conventional convA      Gene1     16
 2 AC1        A        conventional convA      Gene2     15
 3 AC1        A        conventional convA      Gene3     71
 4 AC1        A        conventional convA      Gene4    153
 5 AC1        A        conventional convA      Gene5      2
 6 AC1        A        conventional convA      Gene6     11
 7 AC1        A        conventional convA      Gene7      9
 8 AC1        A        conventional convA      Gene8     11
 9 AC1        A        conventional convA      Gene9    540
10 AC1        A        conventional convA      Gene10   104
# ℹ 47,990 more rows
gene_means <- stickle_long |>
  group_by(gene, Genotype, Microbiota) |>
  summarize(mean_count = mean(count), .groups = "drop")
gene_means |> filter(gene == "Gene1")
# A tibble: 4 × 4
  gene  Genotype Microbiota   mean_count
  <chr> <chr>    <chr>             <dbl>
1 Gene1 A        conventional      12.1 
2 Gene1 A        mono              15.8 
3 Gene1 B        conventional       0.6 
4 Gene1 B        mono               2.15
import pandas as pd

stickle = pd.read_csv("../Data_Folder/Stickle_RNAseq.tsv", sep="\t")

stickle_long = stickle.melt(                      # melt = pivot_longer
    id_vars=["Individual", "Genotype", "Microbiota", "Geno&Micro"],
    var_name="gene", value_name="count")
stickle_long.shape
# (48000, 6) - one row per fish x gene

gene_means = (stickle_long
    .groupby(["gene", "Genotype", "Microbiota"], as_index=False)
    .agg(mean_count=("count", "mean")))
gene_means.query("gene == 'Gene1'")

GGplot2: the grammar of graphics

The grammar of graphics: a plot is a stack of layers

Building a figure layer by layer: 1 - data, aesthetics, a geom

Figure 4: Layer 1 - every observation, visible. Gene1 read counts for all 80 fish by host genotype. Points are individual fish, jittered horizontally (width = 0.08) to prevent overplotting; no summaries are drawn yet.
gene1 <- stickle_long |> filter(gene == "Gene1")

ggplot(gene1, aes(x = Genotype, y = count)) +
  geom_jitter(width = 0.08)

ggsave("results/gene1_jitter.svg", width = 7, height = 5)
import seaborn as sns
import matplotlib.pyplot as plt

gene1 = stickle_long.query("gene == 'Gene1'")
sns.stripplot(data=gene1, x="Genotype", y="count", jitter=0.08)
plt.savefig("results/gene1_jitter.svg")
plt.show()
  • ggplot(data, aes(...)) declares the mapping: which column drives which visual channel
  • A geom draws it; geom_jitter() is geom_point() plus sideways noise
  • Nothing is styled yet - and it is already a readable figure
  • The save-line convention: every example in this deck ends by writing the figure to results/ as an SVG - ggsave() for ggplot, plt.savefig() in Python

Building a figure layer by layer: 2 - more aesthetics, more geoms

Figure 5: Layer 2 - distributions and raw data together. Gene1 read counts by host genotype, split by microbiota treatment (fill and color). Violins show each group’s density; overlaid points are individual fish, jitter-dodged so the two treatments sit side by side within each genotype.
ggplot(gene1, aes(x = Genotype, y = count, fill = Microbiota)) +
  geom_violin(alpha = 0.4, color = NA, position = position_dodge(0.8)) +
  geom_point(aes(color = Microbiota), alpha = 0.6,
             position = position_jitterdodge(jitter.width = 0.08, dodge.width = 0.8))

ggsave("results/gene1_violin.svg", width = 7, height = 5)
sns.violinplot(data=gene1, x="Genotype", y="count", hue="Microbiota",
               inner=None, alpha=0.4)
sns.stripplot(data=gene1, x="Genotype", y="count", hue="Microbiota",
              dodge=True, size=4, alpha=0.6, legend=False)
plt.savefig("results/gene1_violin.svg")
plt.show()
  • Mapping a column to fill / color (hue in seaborn) splits the groups and builds the legend - automatically, from the data
  • Layers draw in order: violin first, points on top
  • position_dodge places the two microbiota groups side by side

Building a figure layer by layer: 3 - facets, scales, labels, theme

Figure 6: Stickleback gut RNA-seq - the assembled panel. Read counts for Gene1-Gene4 (one panel per gene, independent y-axes) by host genotype and microbiota treatment in all 80 fish. Violins show group distributions; points are individual fish; colors are from the colorblind-safe viridis palette.
stickle_long |>
  filter(gene %in% paste0("Gene", 1:4)) |>
  ggplot(aes(Genotype, count, fill = Microbiota)) +
  geom_violin(alpha = 0.4, color = NA, position = position_dodge(0.8)) +
  geom_point(aes(color = Microbiota), alpha = 0.6, size = 1.2,
             position = position_jitterdodge(jitter.width = 0.08, dodge.width = 0.8)) +
  scale_fill_viridis_d(end = 0.7) + scale_color_viridis_d(end = 0.7) +
  facet_wrap(~ gene, scales = "free_y") +
  labs(x = "Host genotype", y = "Read count",
       title = "Stickleback gut RNA-seq",
       subtitle = "Each point is one fish; violins show the full distribution") +
  theme_minimal(base_size = 14) + theme(legend.position = "top")

ggsave("results/gene_panel.svg", width = 8, height = 5)
long4 = stickle_long[stickle_long["gene"].isin([f"Gene{i}" for i in range(1, 5)])]

g = sns.catplot(data=long4, x="Genotype", y="count", hue="Microbiota",
                col="gene", col_wrap=2, kind="violin", inner="point",
                palette=["#440154", "#35b779"], sharey=False, height=2.6, aspect=1.3)
g.set_axis_labels("Host genotype", "Read count")
g.set_titles("{col_name}")
g.figure.suptitle("Stickleback gut RNA-seq", y=1.02)
plt.savefig("results/gene_panel.svg", bbox_inches="tight")
plt.show()

The figure this code draws (pre-rendered)
  • facet_wrap(~ gene) - one panel per gene, for free; scales = "free_y" lets each gene keep its own axis
  • This is Figure 1 from the start of class - and the last line of the code writes it to results/gene_panel.svg

Common ggplot2 geoms - each with an example from our data

Geom Purpose Data type seaborn equivalent
geom_histogram() Histograms Single continuous sns.histplot()
geom_boxplot() Boxplots Continuous by category sns.boxplot()
geom_point() Scatterplots Two continuous sns.scatterplot()
geom_smooth() Trend lines Two continuous sns.regplot()
geom_line() Line plots Continuous over ordered sns.lineplot()
geom_bar() / geom_col() Bar charts Categorical counts / values sns.countplot() / sns.barplot()

Tip

The next six slides draw each one from the stickleback table. facet_wrap(~ group) splits any of them into one panel per group, and ggsave("figure.pdf") writes the last plot to a file. Full mechanics: Ch. 13.

Geom example 1: geom_histogram()

Figure 7: The distribution of Gene1 expression. Histogram of Gene1 read counts across all 80 fish (30 bins). The right-skewed shape - many fish with low counts and a long tail of high expressors - is typical of RNA-seq count data.
ggplot(gene1, aes(x = count)) +
  geom_histogram(bins = 30, fill = "#31688e", color = "white") +
  labs(x = "Gene1 read count", y = "Number of fish")

ggsave("results/gene1_histogram.svg", width = 7, height = 5)
sns.histplot(data=gene1, x="count", bins=30, color="#31688e")
plt.xlabel("Gene1 read count"); plt.ylabel("Number of fish")
plt.savefig("results/gene1_histogram.svg")
plt.show()
  • One continuous column mapped to x; the geom does the binning
  • bins changes the story - always try a few values
  • The smooth cousin is geom_density()

Geom example 2: geom_boxplot()

Figure 8: Gene2 expression by genotype and microbiota. Boxes show the median, interquartile range and 1.5 x IQR whiskers of Gene2 read counts per host genotype (n = 40 each), split by microbiota treatment (n = 20 per box); values beyond the whiskers are plotted as individual points.
stickle_long |>
  filter(gene == "Gene2") |>
  ggplot(aes(x = Genotype, y = count, fill = Microbiota)) +
  geom_boxplot(alpha = 0.7) +
  scale_fill_viridis_d(end = 0.7) +
  labs(x = "Host genotype", y = "Gene2 read count")

ggsave("results/gene2_boxplot.svg", width = 7, height = 5)
gene2 = stickle_long.query("gene == 'Gene2'")
sns.boxplot(data=gene2, x="Genotype", y="count", hue="Microbiota",
            palette=["#440154", "#35b779"])
plt.xlabel("Host genotype"); plt.ylabel("Gene2 read count")
plt.savefig("results/gene2_boxplot.svg")
plt.show()
  • Continuous by category: the five-number summary per group, at a glance
  • A second categorical variable maps to fill / hue and dodges automatically
  • When the shape of the distribution matters, swap in geom_violin() (as in Figure 1)

Geom example 3: geom_point()

Figure 9: Covariation of two genes. Gene2 versus Gene1 read counts; each point is one of the 80 fish, colored by host genotype. Fish with jointly elevated counts suggest shared regulation - or simply deeper sequencing.
ggplot(stickle, aes(x = Gene1, y = Gene2, color = Genotype)) +
  geom_point(size = 2.5, alpha = 0.7) +
  scale_color_viridis_d(end = 0.6) +
  labs(x = "Gene1 read count", y = "Gene2 read count")

ggsave("results/gene1_gene2_scatter.svg", width = 7, height = 5)
sns.scatterplot(data=stickle, x="Gene1", y="Gene2", hue="Genotype",
                s=50, alpha=0.7)
plt.savefig("results/gene1_gene2_scatter.svg")
plt.show()
  • Two continuous columns: the workhorse for relationships between variables
  • A third column rides along as color - no extra work
  • With thousands of points, lower alpha so density shows through

Geom example 4: geom_smooth()

Figure 10: A fitted trend as a layer. Gene2 versus Gene1 read counts for all 80 fish (points), with an ordinary least-squares line fitted by geom_smooth(method = ‘lm’); the gray ribbon is the 95 percent confidence interval of the fit.
ggplot(stickle, aes(x = Gene1, y = Gene2)) +
  geom_point(alpha = 0.5, color = "#31688e") +
  geom_smooth(method = "lm", color = "#440154") +
  labs(x = "Gene1 read count", y = "Gene2 read count")

ggsave("results/gene1_gene2_trend.svg", width = 7, height = 5)
sns.regplot(data=stickle, x="Gene1", y="Gene2",
            scatter_kws={"alpha": 0.5, "color": "#31688e"},
            line_kws={"color": "#440154"})
plt.savefig("results/gene1_gene2_trend.svg")
plt.show()
  • A trend is just another layer on the scatter - that is the grammar at work
  • method = "lm" fits a straight line (with its confidence ribbon); the default is a flexible loess curve
  • The fitted line previews the statistics sequence: this is a regression

Geom example 5: geom_line()

Figure 11: The ranked expression profile of the transcriptome. Mean read count per gene for all 600 genes, ordered from most to least expressed, on a log10 y-axis. A handful of highly expressed genes dominate, followed by a long shallow tail - the characteristic shape of an RNA-seq expression profile.
stickle_long |>
  group_by(gene) |>
  summarize(mean_count = mean(count)) |>
  arrange(desc(mean_count)) |>
  mutate(rank = row_number()) |>
  ggplot(aes(x = rank, y = mean_count)) +
  geom_line(color = "#440154", linewidth = 1) +
  scale_y_log10() +
  labs(x = "Gene rank (by mean expression)", y = "Mean read count (log)")

ggsave("results/gene_rank_means.svg", width = 7, height = 5)
genes = [c for c in stickle.columns if c.startswith("Gene")]
means = stickle[genes].mean().sort_values(ascending=False)

fig, ax = plt.subplots()
ax.plot(range(1, len(means) + 1), means, color="#440154", lw=2)
ax.set(yscale="log", xlabel="Gene rank (by mean expression)",
       ylabel="Mean read count (log)")
plt.savefig("results/gene_rank_means.svg")
plt.show()
  • Lines need an ordered x - time, position, or (here) rank; connecting unordered categories draws meaningless slopes
  • The shape is classic transcriptomics: a few genes dominate, then a long tail - which is why the log scale is essential
  • The whole pipeline is dplyr verbs feeding the plot: group_by |> summarize |> arrange |> mutate |> ggplot

Geom example 6: geom_col()

Figure 12: Group medians of Gene1 expression. Median Gene1 read count per host genotype x microbiota group (n = 20 fish per bar), with microbiota treatments dodged within each genotype. Bars begin at zero, keeping relative magnitudes honest.
gene1 |>
  group_by(Genotype, Microbiota) |>
  summarize(median_count = median(count), .groups = "drop") |>
  ggplot(aes(x = Genotype, y = median_count, fill = Microbiota)) +
  geom_col(position = "dodge") +
  scale_fill_viridis_d(end = 0.7) +
  labs(x = "Host genotype", y = "Median Gene1 read count")

ggsave("results/gene1_group_medians.svg", width = 7, height = 5)
import numpy as np

sns.barplot(data=gene1, x="Genotype", y="count", hue="Microbiota",
            estimator=np.median, errorbar=None,
            palette=["#440154", "#35b779"])
plt.xlabel("Host genotype"); plt.ylabel("Median Gene1 read count")
plt.savefig("results/gene1_group_medians.svg")
plt.show()
  • geom_bar() counts rows for you; geom_col() plots values you computed - here, the group medians from summarize()
  • Bars start at zero by construction, which keeps magnitudes honest
  • A bar hides the distribution behind one number - prefer boxplots or violins when the spread matters

Choosing the right kind of plot

Four genes in one figure

Figure 13: Example 1, rebuilt from the raw file. Read counts for Gene1-Gene4 in all 80 fish, by host genotype and microbiota treatment, produced by the complete script shown in the R and Python tabs: read the raw TSV, reshape to long format, filter to four genes, plot, and save.
library(tidyverse)

stickle <- read_tsv("data/Stickle_RNAseq.tsv", show_col_types = FALSE) |>
  rename(geno_micro = `Geno&Micro`)

stickle_long <- stickle |>
  pivot_longer(starts_with("Gene"), names_to = "gene", values_to = "count")

stickle_long |>
  filter(gene %in% paste0("Gene", 1:4)) |>
  ggplot(aes(Genotype, count, fill = Microbiota)) +
  geom_violin(alpha = 0.4, color = NA, position = position_dodge(0.8)) +
  geom_point(aes(color = Microbiota), alpha = 0.6, size = 1.2,
             position = position_jitterdodge(jitter.width = 0.08, dodge.width = 0.8)) +
  scale_fill_viridis_d(end = 0.7) + scale_color_viridis_d(end = 0.7) +
  facet_wrap(~ gene, scales = "free_y") +
  labs(x = "Host genotype", y = "Read count",
       title = "Stickleback gut RNA-seq",
       subtitle = "Each point is one fish; violins show the full distribution") +
  theme_minimal(base_size = 14) + theme(legend.position = "top")

ggsave("results/gene_panel.svg", width = 8, height = 6)
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

stickle = pd.read_csv("data/Stickle_RNAseq.tsv", sep="\t")

stickle_long = stickle.melt(
    id_vars=["Individual", "Genotype", "Microbiota", "Geno&Micro"],
    var_name="gene", value_name="count")

long4 = stickle_long[stickle_long["gene"].isin([f"Gene{i}" for i in range(1, 5)])]
g = sns.catplot(data=long4, x="Genotype", y="count", hue="Microbiota",
                col="gene", col_wrap=2, kind="violin", inner="point",
                palette=["#440154", "#35b779"], sharey=False, height=2.6, aspect=1.3)
g.set_axis_labels("Host genotype", "Read count")
g.set_titles("{col_name}")
g.figure.suptitle("Stickleback gut RNA-seq", y=1.02)

plt.savefig("results/gene_panel.svg", bbox_inches="tight")
plt.show()

The figure this code draws (pre-rendered)
  • This is Example 1 from the start of class, now as one complete, visible script: read the raw TSV → rename the awkward column → pivot_longer → filter → plot → save
  • Every line was derived earlier today: the reshape in the tidyverse section, each plot layer in the build-up slides
  • About 20 lines, exactly as promised - and swapping the four gene names reuses it for any genes (Ex 6 asks you to do exactly that)

All 600 genes in one figure

Figure 14: Mean versus variance for every gene, with a fitted trend. Each of the 600 points is one gene, placed by the mean and variance of its read counts across the 80 fish (log10 axes). Dashed line: variance = mean, the Poisson expectation; solid curve: loess trend through the observed genes. Essentially every gene lies above the line - the overdispersion that motivates negative-binomial models of RNA-seq.
stickle_long |>
  group_by(gene) |>
  summarize(mean = mean(count), var = var(count)) |>
  ggplot(aes(mean, var)) +
  geom_point(alpha = 0.4, color = "#31688e") +
  geom_abline(slope = 1, intercept = 0, linetype = "dashed") +
  geom_smooth(se = FALSE, color = "#440154") +
  scale_x_log10() + scale_y_log10() +
  labs(x = "Mean count per gene (log)", y = "Variance (log)",
       title = "All 600 genes at once",
       subtitle = "Dashed line: variance = mean (Poisson). Real RNA-seq runs hotter.") +
  theme_minimal(base_size = 14)

ggsave("results/mean_variance.svg", width = 7, height = 5)
import numpy as np

genes = [c for c in stickle.columns if c.startswith("Gene")]
mv = pd.DataFrame({"mean": stickle[genes].mean(), "var": stickle[genes].var()})

fig, ax = plt.subplots()
ax.scatter(mv["mean"], mv["var"], s=10, alpha=0.4, color="#31688e")
lims = np.array([mv["mean"].min() / 2, mv["mean"].max() * 2])
ax.plot(lims, lims, ls="--", color="black")              # variance = mean
ax.set(xscale="log", yscale="log",
       xlabel="Mean count per gene (log)", ylabel="Variance (log)",
       title="All 600 genes at once")
plt.savefig("results/mean_variance.svg")
plt.show()

The figure this code draws (pre-rendered)
  • group_by(gene) |> summarize() collapses 48,000 rows to 600 - one point per gene
  • The biology in one glance: variance outruns the mean, which is why RNA-seq models use the negative binomial rather than the Poisson - a preview of the statistics to come
  • This is Figure 2 from the start of class, plus a trend layer - you can now build both opening figures

Matching the figure to the question

Figure The question it answers R Python
Clustered heatmap + dendrograms which samples / genes behave alike? heatmap() sns.clustermap()
Time course with ribbons how do trajectories differ, and how noisy are they? geom_ribbon() + geom_line() plt.fill_between()
Dose-response curve where is the IC50? geom_point() + geom_smooth(), log-x plt.semilogx()
PCA scores plot what structures 600 dimensions? prcomp() + geom_point() sklearn PCA + scatter
Violin + jitter what does the whole distribution look like, by group? geom_violin() sns.violinplot()
  • The next four slides draw each one - two from synthesized data (so you can see the generating process), two from the real stickleback table
  • Full walk-throughs live in the book’s Ch. 13 figure gallery

Self-study this week: the rest of the toolbox

The lecture built the spine; these carry the rest. All of it is on the Examples page and in the book - budget a couple of hours before next Monday:

Topic Where it lives
Reading & writing files, saving figures, in full Notebook §2-4 · Ch. 11/Ch. 12
Random draws, honest density overlays, base graphics Notebook §8-11
Control flow: if, for, while - and when a vector beats a loop Notebook §16-18 · Parisa’s s2-s4 · Ch. 11/12
Logical operators and membership (%in% / .isin()) Notebook §15
Inspecting objects, common gotchas, reading error messages Notebook §14, §30-31 · Ch. 11/12
Writing your own functions Ch. 15 · Parisa’s s4
  • The companion notebooks (Lecture06_R_companion.ipynb, Lecture06_Python_companion.ipynb) are a matched R/Python pair with the same 31 sections - run them side by side
  • Installing packages and managing environments get their own lecture in Week 9
  • Ex 6 practices all of today plus the self-study - including rebuilding the four-gene panel for your own genes
  • Next Monday (L07): Quarto - your code, figures and writing become one reproducible document