Where this fits

• This is the second Friday-morning Talapas module — putting Module 09’s SLURM skills to work on the real analysis.
• In Module 09 you connected with VS Code Remote-SSH, explored file systems and modules, and learned interactive + batch submission. Here you submit the full pipeline as batch jobs.
• Companion reading: Chapter 10 — The Talapas Analysis Pipeline
• Companion tutorial: Tutorial 10 — Talapas Pipeline
• Prerequisite: Lecture 09 — VS Code & SLURM Basics on Talapas
• Goal: run the same ifnb analysis as the laptop notebooks, but as SLURM batch jobs.

Goals of this lecture

• See how the workshop’s 1:1:1:1 structure maps onto the cluster
• Understand the 01–08 scripts as a chained pipeline of batch jobs
• Use the run_rscript.sbatch wrapper and run_core_pipeline.sbatch
• Know why you pre-cache ifnb and what DATA_DIR is for
• Understand why ifnb yields real biology, not a null result
• Know where the bonus scripts 13–17 fit

Four legs, one workflow

• Every module of the workshop has four parallel legs:
• Lecture → concept slides (this deck)
• Reading → long-form prose chapter
• Laptop tutorial → rich learning notebook with output
• Talapas script → the same workflow, streamlined for SLURM

Notebooks vs scripts

• The laptop notebooks are teaching documents: prose, “Think about it” prompts, inline plots.
• The Talapas scripts are the same analysis, stripped to runnable R.
• Scripts carry only minimal comments that mark each step’s alignment with its notebook — the explanation lives in the notebook.
• Same dataset, same biology — no new analysis to learn, just a new place to run it.

01–08: each output feeds the next

• The eight core scripts chain on a shared ifnb object.
• Each step reads the previous step’s .rds/.csv and writes the next one’s input.
• Only external input is ifnb itself, fetched by script 01 (I have also already loaded it for you).
• They must run in order — a missing or corrupt hand-off breaks the chain.

The hand-off chain

Why .rds hand-offs?

• Each .rds is a checkpoint — a fully-formed Seurat object on disk.
• Re-run step 6 without re-running 1–5; just point it at the saved object.
• Steps can be right-sized independently (06 and 08 want more memory).
• A clean, inspectable contract between steps — exactly how reproducible pipelines are built.

Confirm before launching the next

• The steps are chained — don’t fire step 2 until step 1 truly finished.
• Check the .out log for the Wrote ... line.
• Run seff <jobid> and confirm State: COMPLETED (not OUT_OF_MEMORY / TIMEOUT).
• A job killed mid-write leaves a corrupt .rds that poisons the next step.

run_sc_pipeline.sbatch — the whole series at once

STEPS=(01_qc_preprocessing.R 02_dimreduction_clustering.R … 08_differential_abundance.R)
for step in "${STEPS[@]}"; do
  echo "==== RUNNING ${step} ===="
  Rscript "${step}"
done

• Runs all eight in order, in a single allocation (~12 h, ~96 GB).
• Submit once: sbatch run_sc_pipeline.sbatch
• Simplest way to reproduce the entire core analysis end-to-end.

Memory per step — not all steps are equal

• Steps 06 (pseudobulk_de) and 08 (differential_abundance) use more RAM — override with --mem=96G.
• Confirm each step is COMPLETED — a job killed mid-write leaves a corrupt .rds that poisons the chain.

Pre-caching ifnb

• Script 01 downloads ifnb (~25 MB) via ExperimentHub on first use.
• Compute nodes may have no outbound internet — the download would fail inside the job.
• So populate the cache once on a login node (which has internet) before submitting:
• If this doesn’t work for you, I have already download the data and intermediate data files so you don’t get stuck

DATA_DIR — where the hand-off files land

• By default the scripts write their .rds/.csv hand-offs to ../data/.
• Set DATA_DIR to redirect them — e.g. to fast, roomy /scratch:

• Keeps large intermediates off /home/ and makes the run reproducible and relocatable.

Why ifnb gives real results

• ifnb (Kang et al. 2017) has a real condition — stim: CTRL vs IFN-β.
• And real biological replicates — ind: 8 lupus donors, each in both conditions.
• So scripts 05/06/08 use genuine sample and donor structure.
• Integration corrects the CTRL/STIM axis; DE and DA recover a real interferon-stimulated gene (ISG) response — not a null.

Same dataset, same biology

• This is the identical dataset and biology as the laptop notebooks.
• The cluster run isn’t a toy demo — it reproduces the workshop’s real findings at scale.
• A single dissociated-tumor run would lack condition + replicate structure; ifnb has both by design.

Scripts 13–17 — standalone tracks

• Parallel the downstream bonus notebooks; run after the core, not in sequence.
• 13_wgcna — co-expression modules, its own GSE152418 dataset.
• 14_trajectory_cellcomm — runs on the annotated ifnb object.
• 15_scatac_signac — scATAC on PBMC 10k, its own data.
• 16_spatial — stxBrain spatial data.
• 17_fair_metadata — exports ifnb for submission (.h5ad).

What’s next

• You can now run the full ifnb analysis on Talapas as chained SLURM jobs.
• For the raw-FASTQ upstream path, see the Cell Ranger bonus modules and the full Talapas run roadmap.

Key takeaways

• The pipeline is a chain: .rds checkpoints hand off step to step, in order.
• run_sc_pipeline.sbatch runs all eight analysis scripts.
• Pre-cache ifnb on a login node; use DATA_DIR to relocate outputs.