--- title: "Tutorial 10 — The Analysis Pipeline on Talapas (R scripts)" subtitle: "Streamlined, SLURM-callable R scripts that parallel the laptop learning notebooks — same ifnb dataset" --- ::: {.callout-note title="Running the code in this tutorial"} Every code chunk here is tagged with `#| eval: false`, so the published page shows the code **without running it**. To run it yourself, open the downloaded `.qmd` in **RStudio**: - **Run one chunk:** click the green ▶ (*Run Current Chunk*) at the chunk's top-right, or press *Ctrl/Cmd + Enter*. This runs the chunk **regardless of its `eval` setting** — the easiest way to work through the tutorial interactively. - **Run a chunk on render:** change that chunk's `#| eval: false` to `#| eval: true` (or delete the line) so it executes when the document is rendered. - **Render the whole tutorial:** click the **Render** button in RStudio (or run `quarto render` in a terminal) to execute the chunks top-to-bottom and knit a finished HTML. To run everything on render, set `eval: true` once in the YAML header at the top. ::: The laptop **[tutorials 01–08](../Materials.html)** are *learning notebooks*: rich prose, "Think about it" prompts, and inline output. This page is the fourth leg of the workshop's **1:1:1:1** mapping — **lecture → reading → laptop tutorial → Talapas script** — the *same* workflow as **streamlined R scripts** you run on Talapas as SLURM jobs. The scripts carry only minimal comments that mark each step's alignment with the matching notebook; the explanation lives in the notebook. ::: callout-note **Companion lecture:** [Lecture 10 — Running the Analysis Pipeline on Talapas](../Lecture_Folder/Lecture_10_Talapas_Pipeline.html) · **Companion reading:** [Chapter 10 — The Talapas Analysis Pipeline](../Resources_Folder/Chapter_10_Talapas_Pipeline.html) ::: The scripts run the **same `ifnb` dataset** (Kang *et al.* 2017) as the laptop tutorials, so this is the "run the identical analysis on the cluster" companion to the laptop series — no new dataset to learn. `ifnb` carries a real condition (`stim`: CTRL vs IFN-β) and real donors (`ind`: 8 lupus patients), so the differential-expression and differential-abundance steps produce **real biology**. ::: callout-note **Prerequisite.** Work through [Tutorial 09 — VS Code & SLURM Basics on Talapas](Tutorial_09_SLURM_Basics.html) first — it covers connecting VS Code to Talapas over Remote-SSH, navigating the file systems, running interactive sessions, and submitting batch jobs with `sbatch`. By the time you reach this module you should already be connected to Talapas with the green VS Code banner and have run at least one interactive job. You do **not** need raw FASTQs or Cell Ranger for this track: script 01 loads `ifnb` directly from Bioconductor. **Want the full pipeline from raw reads instead?** See the bonus upstream [Tutorial 11 — Cell Ranger from raw FASTQs](Tutorial_11_CellRanger_RawData.html) and the [Full Talapas RNA-seq run from raw FASTQs](../Resources_Folder/Talapas_FullRun_FromFASTQs.html) roadmap. ::: ::: callout-tip **Pre-cache `ifnb` on a login node.** Script 01 downloads `ifnb` (~25 MB) through `ExperimentHub` on first use. Talapas compute nodes may not have outbound internet, so run `Rscript -e 'invisible(muscData::Kang18_8vs8())'` once on a **login node** (or in an interactive session with internet) to populate the cache before submitting the batch job. ::: ## The scripts ↔ notebook mapping Every analysis script lives in [`scripts/`](scripts/) and is numbered to match its laptop notebook. ### Core pipeline (ifnb — run in order) These eight chain together on the shared `ifnb` object — each one's output is the next one's input. | Script | Parallels laptop notebook | Reads | Writes | |--------|---------------------------|-------|--------| | [`01_qc_preprocessing.R`](scripts/01_qc_preprocessing.R) | [Tutorial 01 — QC & Preprocessing](Tutorial_01_QC_Preprocessing.html) | `muscData::Kang18_8vs8()` | `data/ifnb_preprocessed.rds` | | [`02_dimreduction_clustering.R`](scripts/02_dimreduction_clustering.R) | [Tutorial 02 — DimReduction & Clustering](Tutorial_02_DimReduction_Clustering.html) | `ifnb_preprocessed.rds` | `ifnb_clustered.rds` | | [`03_markers_annotation.R`](scripts/03_markers_annotation.R) | [Tutorial 03 — Markers & Annotation](Tutorial_03_Markers_Annotation.html) | `ifnb_clustered.rds` | `ifnb_annotated.rds` | | [`04_reference_annotation.R`](scripts/04_reference_annotation.R) | [Tutorial 04 — Reference Annotation](Tutorial_04_Reference_Annotation.html) | `ifnb_annotated.rds` | `ifnb_annotated_final.rds` | | [`05_integration.R`](scripts/05_integration.R) | [Tutorial 05 — Integration](Tutorial_05_Integration.html) | `ifnb_annotated_final.rds` | `ifnb_integrated.rds` | | [`06_pseudobulk_de.R`](scripts/06_pseudobulk_de.R) | [Tutorial 06 — Pseudobulk DE](Tutorial_06_DESeq2_DE.html) | `ifnb_integrated.rds` | `ifnb_pseudobulk_de.csv` | | [`07_functional_analysis.R`](scripts/07_functional_analysis.R) | [Tutorial 07 — Functional Analysis](Tutorial_07_FunctionalAnalysis.html) | `ifnb_pseudobulk_de.csv` | `functional/*.csv` | | [`08_differential_abundance.R`](scripts/08_differential_abundance.R) | [Tutorial 08 — Differential Abundance](Tutorial_08_DifferentialAbundance.html) | `ifnb_integrated.rds` | `ifnb_milo_da.csv` | Each step reads and writes its `.rds`/`.csv` hand-off files in `../data/`; the only external input is `ifnb`, fetched by script 01. The scripts run **in order** — each one's output is the next one's input. ### Bonus tracks (standalone — Modules 13–17) These parallel the **downstream bonus laptop tutorials** (run *after* the core). Scripts 13 (WGCNA) and 15 (scATAC) use their **own datasets** (downloaded into `../data/`); 14 and 17 run on the annotated `ifnb` object the core pipeline produces, and 16 (Spatial) uses `stxBrain`. They do not need to be run in sequence. (The raw-data *upstream* bonus modules 11–12 are Cell Ranger pipelines, not R scripts — see Materials.) | Script | Parallels laptop notebook | Reads | Writes | |--------|---------------------------|-------|--------| | [`13_wgcna.R`](scripts/13_wgcna.R) | [Tutorial 13 — WGCNA](Tutorial_13_WGCNA.html) | `GSE152418_*_RawCounts.txt` | `wgcna_*.csv` | | [`14_trajectory_cellcomm.R`](scripts/14_trajectory_cellcomm.R) | [Tutorial 14 — Trajectory & Cell–Cell Comm](Tutorial_14_Trajectory_CellCommunication.html) | `ifnb_annotated.rds` | `ifnb_slingshot_pseudotime.csv` | | [`15_scatac_signac.R`](scripts/15_scatac_signac.R) | [Tutorial 15 — scATAC-seq (Signac)](Tutorial_15_scATACseq_Signac.html) | PBMC 10k scATAC files in `data/` | `pbmc_atac_clustered.rds` | | [`16_spatial.R`](scripts/16_spatial.R) | [Tutorial 16 — Spatial Transcriptomics](Tutorial_16_Spatial_Transcriptomics.html) | `stxBrain` (SeuratData) | `brain_spatial_integrated.rds` | | [`17_fair_metadata.R`](scripts/17_fair_metadata.R) | [Tutorial 17 — FAIR & Metadata](Tutorial_17_FAIR_Metadata.html) | `ifnb_annotated_final.rds` | `ifnb_for_submission.h5ad` | ::: callout-note The bonus scripts mirror the runnable core of each notebook. Steps that are inherently interactive or Python-based — Monocle3 root selection, scVelo, CellChat plotting, the `cellxgene-schema` validator — are kept as commented sketches in the script, exactly as in the notebook. ::: ## Running the pipeline on Talapas In [Module 09](Tutorial_09_SLURM_Basics.html) you connected to Talapas and worked **interactively** — starting `srun` sessions, exploring the file system, and running `hello_world.R` by hand. Here you submit the full analysis series as **batch jobs** (`sbatch`) so they run unattended and you can log out while they run. The scripts are submitted with the generic wrapper [`run_rscript.sbatch`](scripts/run_rscript.sbatch), which loads R and runs the script you pass it. From the `scripts/` directory: ```{bash} #| label: M10-one_time_setup_slurm #| eval: false # One-time setup: SLURM writes job logs here (it will NOT create the dir for you) mkdir -p logs # Run the pipeline in order (wait for each to finish before the next) sbatch --job-name=qc run_rscript.sbatch 01_qc_preprocessing.R sbatch --job-name=cluster run_rscript.sbatch 02_dimreduction_clustering.R sbatch --job-name=markers run_rscript.sbatch 03_markers_annotation.R sbatch --job-name=refannot run_rscript.sbatch 04_reference_annotation.R sbatch --job-name=integrate run_rscript.sbatch 05_integration.R sbatch --job-name=pb_de --mem=96G run_rscript.sbatch 06_pseudobulk_de.R sbatch --job-name=fa --mem=32G run_rscript.sbatch 07_functional_analysis.R sbatch --job-name=da --mem=96G run_rscript.sbatch 08_differential_abundance.R ``` ::: callout-tip **Confirm each step before launching the next.** Check the `.out` log for the `Wrote ...` line and run `seff ` to confirm `State: COMPLETED` (not `OUT_OF_MEMORY`/`TIMEOUT`) — a job killed mid-write leaves a corrupt `.rds`. ::: ::: callout-note **Relocating outputs.** Set the `DATA_DIR` environment variable to write the `.rds`/`.csv` hand-off files to `/scratch//...` instead of `../data/`, e.g. `export DATA_DIR=/scratch///ifnb/data`. ::: ## Why ifnb gives real results Unlike a single dissociated-tumor run, `ifnb` has a **real biological condition** (`stim`: control vs IFN-β–stimulated) and **real biological replicates** (`ind`: 8 lupus donors, each measured in both conditions). So scripts 05/06/08 use the genuine sample and donor structure — integration corrects the CTRL/STIM batch axis, and the pseudobulk DE and differential-abundance tests recover a real **interferon-stimulated gene (ISG)** response rather than a null result. This is the same dataset, and the same biology, as the laptop notebooks. ## Credits Adapted from the laptop tutorial series for the Talapas HPC environment, building on **Shannon N. Snyder's** *Talapas Basics* materials.