Tutorial 10 — The Analysis Pipeline on Talapas (R scripts)

Streamlined, SLURM-callable R scripts that parallel the laptop learning notebooks — same ifnb dataset

Running the code in this tutorial

Every code chunk here is tagged with #| eval: false, so the published page shows the code without running it. To run it yourself, open the downloaded .qmd in RStudio:

Run one chunk: click the green ▶ (Run Current Chunk) at the chunk’s top-right, or press Ctrl/Cmd + Enter. This runs the chunk regardless of its eval setting — the easiest way to work through the tutorial interactively.
Run a chunk on render: change that chunk’s #| eval: false to #| eval: true (or delete the line) so it executes when the document is rendered.
Render the whole tutorial: click the Render button in RStudio (or run quarto render in a terminal) to execute the chunks top-to-bottom and knit a finished HTML. To run everything on render, set eval: true once in the YAML header at the top.

The laptop tutorials 01–08 are learning notebooks: rich prose, “Think about it” prompts, and inline output. This page is the fourth leg of the workshop’s 1:1:1:1 mapping — lecture → reading → laptop tutorial → Talapas script — the same workflow as streamlined R scripts you run on Talapas as SLURM jobs. The scripts carry only minimal comments that mark each step’s alignment with the matching notebook; the explanation lives in the notebook.

Note

Companion lecture: Lecture 10 — Running the Analysis Pipeline on Talapas · Companion reading: Chapter 10 — The Talapas Analysis Pipeline

The scripts run the same ifnb dataset (Kang et al. 2017) as the laptop tutorials, so this is the “run the identical analysis on the cluster” companion to the laptop series — no new dataset to learn. ifnb carries a real condition (stim: CTRL vs IFN-β) and real donors (ind: 8 lupus patients), so the differential-expression and differential-abundance steps produce real biology.

Note

Prerequisite. Work through Tutorial 09 — VS Code & SLURM Basics on Talapas first — it covers connecting VS Code to Talapas over Remote-SSH, navigating the file systems, running interactive sessions, and submitting batch jobs with sbatch. By the time you reach this module you should already be connected to Talapas with the green VS Code banner and have run at least one interactive job. You do not need raw FASTQs or Cell Ranger for this track: script 01 loads ifnb directly from Bioconductor. Want the full pipeline from raw reads instead? See the bonus upstream Tutorial 11 — Cell Ranger from raw FASTQs and the Full Talapas RNA-seq run from raw FASTQs roadmap.

Tip

Pre-cache ifnb on a login node. Script 01 downloads ifnb (~25 MB) through ExperimentHub on first use. Talapas compute nodes may not have outbound internet, so run Rscript -e 'invisible(muscData::Kang18_8vs8())' once on a login node (or in an interactive session with internet) to populate the cache before submitting the batch job.

The scripts ↔︎ notebook mapping

Every analysis script lives in scripts/ and is numbered to match its laptop notebook.

Core pipeline (ifnb — run in order)

These eight chain together on the shared ifnb object — each one’s output is the next one’s input.

Script	Parallels laptop notebook	Reads	Writes
`01_qc_preprocessing.R`	Tutorial 01 — QC & Preprocessing	`muscData::Kang18_8vs8()`	`data/ifnb_preprocessed.rds`
`02_dimreduction_clustering.R`	Tutorial 02 — DimReduction & Clustering	`ifnb_preprocessed.rds`	`ifnb_clustered.rds`
`03_markers_annotation.R`	Tutorial 03 — Markers & Annotation	`ifnb_clustered.rds`	`ifnb_annotated.rds`
`04_reference_annotation.R`	Tutorial 04 — Reference Annotation	`ifnb_annotated.rds`	`ifnb_annotated_final.rds`
`05_integration.R`	Tutorial 05 — Integration	`ifnb_annotated_final.rds`	`ifnb_integrated.rds`
`06_pseudobulk_de.R`	Tutorial 06 — Pseudobulk DE	`ifnb_integrated.rds`	`ifnb_pseudobulk_de.csv`
`07_functional_analysis.R`	Tutorial 07 — Functional Analysis	`ifnb_pseudobulk_de.csv`	`functional/*.csv`
`08_differential_abundance.R`	Tutorial 08 — Differential Abundance	`ifnb_integrated.rds`	`ifnb_milo_da.csv`

Each step reads and writes its .rds/.csv hand-off files in ../data/; the only external input is ifnb, fetched by script 01. The scripts run in order — each one’s output is the next one’s input.

Bonus tracks (standalone — Modules 13–17)

These parallel the downstream bonus laptop tutorials (run after the core). Scripts 13 (WGCNA) and 15 (scATAC) use their own datasets (downloaded into ../data/); 14 and 17 run on the annotated ifnb object the core pipeline produces, and 16 (Spatial) uses stxBrain. They do not need to be run in sequence. (The raw-data upstream bonus modules 11–12 are Cell Ranger pipelines, not R scripts — see Materials.)

Script	Parallels laptop notebook	Reads	Writes
`13_wgcna.R`	Tutorial 13 — WGCNA	`GSE152418_*_RawCounts.txt`	`wgcna_*.csv`
`14_trajectory_cellcomm.R`	Tutorial 14 — Trajectory & Cell–Cell Comm	`ifnb_annotated.rds`	`ifnb_slingshot_pseudotime.csv`
`15_scatac_signac.R`	Tutorial 15 — scATAC-seq (Signac)	PBMC 10k scATAC files in `data/`	`pbmc_atac_clustered.rds`
`16_spatial.R`	Tutorial 16 — Spatial Transcriptomics	`stxBrain` (SeuratData)	`brain_spatial_integrated.rds`
`17_fair_metadata.R`	Tutorial 17 — FAIR & Metadata	`ifnb_annotated_final.rds`	`ifnb_for_submission.h5ad`

Note

The bonus scripts mirror the runnable core of each notebook. Steps that are inherently interactive or Python-based — Monocle3 root selection, scVelo, CellChat plotting, the cellxgene-schema validator — are kept as commented sketches in the script, exactly as in the notebook.

Running the pipeline on Talapas

In Module 09 you connected to Talapas and worked interactively — starting srun sessions, exploring the file system, and running hello_world.R by hand. Here you submit the full analysis series as batch jobs (sbatch) so they run unattended and you can log out while they run.

The scripts are submitted with the generic wrapper run_rscript.sbatch, which loads R and runs the script you pass it. From the scripts/ directory:

#| label: M10-one_time_setup_slurm
#| eval: false
# One-time setup: SLURM writes job logs here (it will NOT create the dir for you)
mkdir -p logs

# Run the pipeline in order (wait for each to finish before the next)
sbatch --job-name=qc        run_rscript.sbatch 01_qc_preprocessing.R
sbatch --job-name=cluster   run_rscript.sbatch 02_dimreduction_clustering.R
sbatch --job-name=markers   run_rscript.sbatch 03_markers_annotation.R
sbatch --job-name=refannot  run_rscript.sbatch 04_reference_annotation.R
sbatch --job-name=integrate run_rscript.sbatch 05_integration.R
sbatch --job-name=pb_de  --mem=96G run_rscript.sbatch 06_pseudobulk_de.R
sbatch --job-name=fa     --mem=32G run_rscript.sbatch 07_functional_analysis.R
sbatch --job-name=da     --mem=96G run_rscript.sbatch 08_differential_abundance.R

Tip

Confirm each step before launching the next. Check the .out log for the Wrote ... line and run seff <jobid> to confirm State: COMPLETED (not OUT_OF_MEMORY/TIMEOUT) — a job killed mid-write leaves a corrupt .rds.

Note

Relocating outputs. Set the DATA_DIR environment variable to write the .rds/.csv hand-off files to /scratch/<PIRG>/... instead of ../data/, e.g. export DATA_DIR=/scratch/<PIRG>/<duckid>/ifnb/data.

Why ifnb gives real results

Unlike a single dissociated-tumor run, ifnb has a real biological condition (stim: control vs IFN-β–stimulated) and real biological replicates (ind: 8 lupus donors, each measured in both conditions). So scripts 05/06/08 use the genuine sample and donor structure — integration corrects the CTRL/STIM batch axis, and the pseudobulk DE and differential-abundance tests recover a real interferon-stimulated gene (ISG) response rather than a null result. This is the same dataset, and the same biology, as the laptop notebooks.

Credits

Adapted from the laptop tutorial series for the Talapas HPC environment, building on Shannon N. Snyder’s Talapas Basics materials.