Lecture 09

Reproducible Software Environments: venv, conda and renv

Bill Cresko

2026-11-23

Book reading for this lecture

The course book goes deeper on everything in this lecture. Read before or after class:

Goals for today

  • Why every project needs its own software toolbox - “dependency hell” and the reproducibility payoff
  • The three tools: venv (Python), conda (anything), renv (R) - and the record file each one leaves
  • conda in depth: channels, Miniforge, environment.yml
  • The laptop-cluster loop: build at home, rebuild on Talapas, use inside a batch job
  • A word on containers - the strongest guarantee of all

Environments: giving every project its own toolbox

Why environments?

  • By default your computer has one R and one Python - and every project installs packages into it
  • Project A needs pandas 2.2; reproducing last year’s analysis needs pandas 1.5. Upgrade for one and you silently break the other - “dependency hell”
  • From the bootcamp: “which Python did that install go into?” is the #1 Python headache
  • An environment is an isolated set of packages (and an interpreter) for one project
  • Three payoffs:
    • projects cannot break each other
    • you can record exactly which versions produced a result
    • a collaborator - or Talapas - can rebuild the same setup from that record

The three environment tools you will meet

Tool For Environments live… The record you share
venv (+ pip) Python in the project folder (.venv/) requirements.txt
conda / mamba Python plus compiled tools (samtools, even R) centrally, by name (conda env list) environment.yml
renv R in the project folder renv.lock
  • venv is the Python default on your laptop - you have used it since the bootcamp
  • conda is the standard in bioinformatics and on Talapas (next week’s cluster) - one tool for any software
  • renv for R projects: renv::init() once, renv::snapshot() to record, renv::restore() to rebuild
  • Whichever you use: commit the record file to Git - never the environment itself

conda in practice - on your own computer

# Install once: Miniforge, from https://conda-forge.org/download/

conda create -n rnaseq python=3.12 pandas matplotlib   # make a named environment - once
conda activate rnaseq                                  # turn it on - every session
conda install -c bioconda samtools                     # compiled genomics tools too!
conda env export > environment.yml                     # record it, to rebuild anywhere
conda deactivate                                       # turn it off
  • Environments are named and central, not per-folder: conda env list shows them all
  • Packages come from channels: conda-forge (general) and bioconda (genomics)
  • Install Miniforge, not Anaconda - Anaconda’s default channel requires a paid license at organizations like UO; Miniforge defaults to the free conda-forge
  • Modern conda uses the fast libmamba solver by default; mamba is its older drop-in cousin, still seen in cluster docs
  • In a few slides the same environment.yml rebuilds this environment on Talapas: conda env create -f environment.yml

venv in practice - the Python default

python3 -m venv .venv               # create, once - lives INSIDE the project folder
source .venv/bin/activate           # turn it on - every session; prompt shows (.venv)
pip install pandas matplotlib       # installs land in the environment, nowhere else
pip freeze > requirements.txt       # record it - the recipe a collaborator rebuilds from
deactivate                          # turn it off
  • The environment is the .venv/ folder - delete the folder, the environment is gone
  • requirements.txt is the pip-world counterpart of environment.yml: pip install -r requirements.txt rebuilds
  • A venv borrows your existing Python and manages Python packages only - for compiled tools (samtools, bwa), you need conda

The record file: environment.yml

With the environment active:

conda env export --from-history | sed '/prefix:/d' > environment.yml
name: rnaseq
channels:
  - bioconda
  - conda-forge
dependencies:
  - python=3.12
  - pandas
  - matplotlib
  • --from-history records only what you asked for, not all ~200 resolved dependencies - which keeps the file short and portable across operating systems
  • The sed drops the machine-specific prefix: line - this is the exact form the Talapas documentation recommends for sharing
  • The habit: keep an environment.yml in each project’s repository, next to the code. It is your collaborator instructions, your Talapas deployment recipe, and the environment section of your methods - all one file

renv: the same idea for R projects

renv::init()        # once per project: a project-local package library
renv::snapshot()    # record exact versions into renv.lock
renv::restore()     # rebuild the library from renv.lock - any machine, any collaborator
  • Per-project, like venv - the library and renv.lock live in the project folder
  • Manages R packages only; conda can provide R itself (pinned) on a cluster, and renv pins the packages inside it - the two compose
  • On a laptop where you already maintain R directly there is usually no rush to adopt it; it matters most for published pipelines and shared projects

R and Python packages on Talapas

module load R/4.3.3
R                                   # interactive, on an interactive node
install.packages("tidyverse")       # goes to your personal library in ~/R/ - once
library(tidyverse)                  # every session / every script
  • Modules give you R itself; you install packages into your home directory
  • For a reproducible project, renv::init() records exact package versions in the project folder
module load miniconda3/20240410

# Best: rebuild the environment you recorded on your laptop (previous slides)
conda env create -f environment.yml      # once
conda activate rnaseq                    # every session / job

# or create one from scratch on the cluster:
conda create -n my_env python=3.11 pandas matplotlib biopython

# or, with the plain python3 module and a virtual environment:
module load python3/3.11.4
python3 -m venv ~/envs/my_env && source ~/envs/my_env/bin/activate
  • conda (with the bioconda channel) installs compiled bioinformatics tools too - conda install -c bioconda samtools
  • Put the same module load + conda activate lines inside your sbatch script - every batch job starts with a clean shell
  • Conda environments live in your home directory and grow large: conda env list to see them, conda clean -a to reclaim cache space, and mind your quota

The laptop-cluster loop, end to end

The whole reason this lecture exists:

  1. Develop and test in a small environment on your laptop
  2. conda env export --from-history | sed '/prefix:/d' > environment.yml; commit it to the project repo
  3. On Talapas: module load miniconda3, then conda env create -f environment.yml
  4. Run at scale via SLURM - with the identical toolchain:
#!/bin/bash
#SBATCH --job-name=rnaseq_run
#SBATCH --account=<your_pirg>
#SBATCH --partition=compute
#SBATCH --time=02:00:00
#SBATCH --mem=8G

module load miniconda3/20240410     # every batch job starts with a clean shell,
conda activate rnaseq               # so BOTH lines go inside the script

python3 scripts/analysis.py data/Stickle_RNAseq.tsv results/

Tip

Last week’s rule still holds: request only what you need. This week’s addition: record what you ran it with. Together they are the recipe for results you can defend.

The wider ecosystem (reference)

Tool Family One-line verdict
mamba conda mostly obsolete - modern conda uses its fast solver
micromamba conda conda as one tiny binary, for containers and CI
pixi conda per-project conda with a lockfile - watch this one
uv PyPI-only very fast venv+pip replacement, pure Python only

Beyond environments: containers

Layer Isolates Travels between clusters?
Environment (venv/conda/renv) packages + interpreter yes, via the record file (rebuilt)
Container (Apptainer/Docker) an entire operating system + software stack yes, as one image file (bit-identical)
  • A container is the strongest reproducibility guarantee: the published pipeline runs exactly as shipped, OS and all
  • On HPC systems the tool is Apptainer (Docker needs root; Apptainer doesn’t)
  • We won’t need containers in this course - but when a paper ships one, you now know what it is: Ch. 20

Your turn

  1. This week, on your laptop: create a conda environment, export its environment.yml, destroy it, and rebuild it from the file (Ex 9 walks through every step) - then commit the file to last week’s repository
  2. Next week, on Talapas: rebuild that same environment on the cluster and prove it travels - a batch job that activates it and prints pandas.__version__ from a compute node (Ex 10 closes the loop)
  3. For your term project: create its environment now, commit the environment.yml, and do all remaining project computing inside it

Summary

  • One project, one environment - projects can’t break each other, and every result has a recorded toolchain
  • venv + requirements.txt for pure Python; conda + environment.yml for anything (and everything bioinformatic); renv + renv.lock for R
  • Commit the record file, never the environment - the environment is rebuilt, the record is shared
  • The same environment.yml works on your laptop and on Talapas - that loop is reproducible computing
  • Next Monday (L10): Talapas - taking the scripts and environments you now have to UO’s cluster. Check that your Talapas account works before class (Ex 7 ends with the account check)