Many nodes, one scheduler

• An HPC cluster is a network of computers (nodes) that share storage and are managed as a single system.
• You interact with a login node — a lightweight gateway machine. Heavy work runs on compute nodes you request from the scheduler.
• Nodes share a parallel file system so your files look the same from every node.
• The scheduler (SLURM at UO) queues and routes jobs fairly among all users.

Meet Talapas — UO’s cluster

• Talapas is managed by UO Research Advanced Computing Services (RACS).
• Thousands of CPU cores, GPUs, high-memory nodes, and petabytes of storage.
• Access is through a PIRG (PI Research Group) — your PI requests an account for the lab (for this workshop your PIRG is cbds)
• Once you have an account your Duck ID grants access via SSH and Open OnDemand (ask your PI to add you to their PIRG)

VS Code Remote-SSH overview

• The Remote-SSH extension (Microsoft) turns VS Code into a full remote IDE.
• You edit, browse, and run code on Talapas — but the interface lives on your laptop.
• One-time setup: add Talapas as an SSH host; after that, one click connects you.
• Install the extension: Ctrl/Cmd + Shift + X → search “Remote - SSH” → Install.

Add the SSH host

• Press F1 → Remote-SSH: Add New SSH Host → enter:

ssh <duckid>@login.talapas.uoregon.edu

• Save to ~/.ssh/config when prompted.
• The host then appears in the Remote Explorer panel on the left sidebar.

Connect with Duck ID + Duo

• Click the host in Remote Explorer (or F1 → Remote-SSH: Connect to Host).
• Enter your Duck ID password when prompted, then approve the Duo push.
• The bottom-left status bar turns green and shows SSH: login.talapas.uoregon.edu — you’re in.
• Open a folder (File → Open Folder) to browse the Talapas file system in the sidebar.

The integrated terminal is a shell on Talapas

• Open it: Terminal → New Terminal (or Ctrl/Cmd+`).
• This is not your laptop’s terminal — it is a shell running on the Talapas login node.
• The green SSH: login.talapas.uoregon.edu banner in the status bar confirms you’re remote.

hostname   # e.g. login1.talapas.uoregon.edu
whoami     # your Duck ID

• All the shell commands from P1 (Computer Systems & the Command Line) — pwd, ls, cd, pipes — work here, now on the cluster.
• module load, srun, and sbatch are all typed in this terminal.

The edit-in-editor / run-in-terminal loop

• Editor pane (left/top): open and edit .R, .qmd, or .sbatch files on Talapas — changes are saved directly on the cluster filesystem.
• Terminal pane (bottom): run the file, submit the job, or tail the log — all on the same machine.

# In the integrated terminal (on the login node):
module load R/4.3.2
Rscript scripts/01_qc_preprocessing.R   # quick smoke-test on login
sbatch run_sc_pipeline.sbatch # submit as a batch job

• For anything real (full data, long runs): always use sbatch — the job survives you closing the laptop.
• For .qmd rendering on the cluster: quarto render my_analysis.qmd in the terminal.

Open OnDemand — browser alternative

• Open OnDemand (ondemand.talapas.uoregon.edu) gives a full HPC portal in the browser — no SSH client needed.
• Launches interactive apps: RStudio Server, Jupyter, a desktop session.
• Useful when VS Code Remote-SSH isn’t an option or for one-off interactive work.
• For this workshop, VS Code Remote-SSH is the primary workflow.

Three places to put files

Navigating on login

• All commands run in the VS Code integrated terminal — the shell on the login node.
• Shell basics (pwd, ls, cd, pipes) were covered in P1 — Computer Systems & the Command Line; the focus here is what’s different on the cluster.
• Check your quotas: df -h /home/$USER and df -h /projects/<PIRG>.
• Your first visit: ls /projects/<PIRG> and ls /scratch/<PIRG>/ to see the lab space.
• Workshop data lives in /projects/<PIRG>/participants/<your-duck-id> — copy or symlink it into your working directory.

What are modules?

• Talapas does not have R (or most tools) in your default PATH.
• Lmod is the module system that adds software to your environment on demand.
• Each module load command sets PATH, LD_LIBRARY_PATH, etc. for one piece of software.
• Modules are temporary — they must be loaded at the start of every session and inside every batch script.

Common module commands

module avail              # list all available modules
module spider R            # filter by name
module load R/4.3.2       # load a specific version
module list               # see what is currently loaded
module purge              # unload everything (good start for scripts)

• Always load a specific version (e.g. R/4.3.2) — module load R picks whatever is default today, which may change.
• module purge at the top of every batch script ensures a clean, reproducible environment.

Explore before you batch

• Use an interactive session (srun --pty bash) to prototype, debug, and explore.
• VS Code Remote-SSH works seamlessly in interactive sessions: open your folder, run code in the integrated terminal.
• Prototype interactively on a small slice of data → then submit the same script as a batch job for the full run.

A cluster is a shared resource

• Talapas has thousands of CPU cores, hundreds of GPUs, terabytes of RAM.
• Dozens of users compete for those resources at any moment.
• If everyone just ran whatever they wanted, the machine would thrash and nobody’s work would finish.
• A scheduler turns chaos into an orderly queue.

What SLURM does

• SLURM = Simple Linux Utility for Resource Management.
• It decides when, where, and with what resources every job runs.
• You declare what you need — CPUs, memory, time, maybe GPUs.
• SLURM finds a compute node that fits and runs your job there.
• Every UO job names an account (your PIRG) for fair-share accounting.

Fair-share, briefly

• SLURM doesn’t run jobs first-come-first-served — it balances by fair-share.
• Groups that have used a lot recently are nudged down the queue; lighter users move up.
• Net effect: smaller, well-sized jobs tend to start sooner.
• You don’t game the queue — you size jobs honestly and let fair-share work.

Two very different machines

• You land on a login node when you ssh in (login.talapas.uoregon.edu).
• Login nodes are for editing files, short commands, and submitting jobs — nothing heavy.
• The real work happens on compute nodes, which SLURM hands you on request.
• Never run a Seurat pipeline directly on a login node — it hurts everyone sharing it.

The golden rule

Login nodes are the front desk. Compute nodes are where you should run jobs.

• Ask SLURM (via srun/salloc/sbatch) to send your work to a compute node.
• Modules you load on a login node do not follow you into a job — load them inside the job.

Which mode when?

• Interactive (srun --pty bash / salloc): you sit and watch — testing, debugging, exploring data, short runs.
• Batch (sbatch): you start it and walk away — long runs that keep going after you log out.
• Most real workshop analysis is batch; you’ll prototype interactively, then submit.

A minimal interactive session

srun --account=<myPIRG> \ 
     --partition=compute \
     --time=1:00:00 \
     --mem=4G \
     --cpus-per-task=2 \
     --pty bash

• When granted, your prompt changes to a compute node (e.g. n0123).
• salloc is the close cousin — request an allocation, prompt jumps to the node.
• Type exit the moment you’re done — idle sessions hoard the node.

Partitions = queues

• Every job targets a partition, which maps to a class of hardware and a time cap.
• compute / computelong — standard CPU, 1 day / 14 days.
• gpu / gpulong — GPU jobs.
• memory / memorylong — high-RAM nodes (up to 4 TB).
• interactive / interactivegpu — short sessions, fast to start.
• preempt — any node, but the owner can bump you off.

Run sinfo to see the cluster

sinfo

• Shows each partition and the state of its nodes (idle, allocated, down).
• QOS (quality-of-service) limits ride on partitions too — they cap how much one user can hold at once.

Key SLURM flags

• --account=<myPIRG> — PIRG to charge (required).
• --partition=<name> — which queue.
• --time=<HH:MM:SS> — wall-clock limit; over it, SLURM kills the job.
• --mem=<size> / --mem-per-cpu=<size> — memory.
• --cpus-per-task=<N> — cores for a threaded task.
• --gpus=<N> + --constraint=gpu-40gb — GPUs and memory tier.

Anatomy of an sbatch script

#!/bin/bash
#SBATCH --account=<myPIRG>
#SBATCH --partition=compute
#SBATCH --job-name=seurat_qc
#SBATCH --output=logs/seurat_qc_%j.out
#SBATCH --error=logs/seurat_qc_%j.err
#SBATCH --time=04:00:00
#SBATCH --cpus-per-task=8
#SBATCH --mem=64G

module purge
module load R/4.3.2
Rscript 01_qc_preprocessing.R

• All #SBATCH lines sit at the top, before any command.
• %j is the job ID; module load happens inside the job.

Create logs/ first — one time

• The shipped scripts write .out/.err to a logs/ subdirectory.
• SLURM will not create it for you — a missing dir means the job is rejected with no log to explain why.

mkdir -p logs

SLURM_ACCOUNT — set it once

echo 'export SLURM_ACCOUNT=<myPIRG>'  >> ~/.bash_profile
echo 'export SBATCH_ACCOUNT=<myPIRG>' >> ~/.bash_profile
source ~/.bash_profile

• Once set, you can omit --account= from every command
• Your PIRG name appears in the output of groups after you log in

Submit and watch

sbatch hostname.sbatch        # -> Submitted batch job 123456
squeue -u $USER               # your jobs
squeue -j 123456              # one job
scancel 123456                # cancel it

• State codes: PD pending, R running, CG completing.
• Two files appear when done: *_123456.out (stdout) and *_123456.err (stderr).

Monitoring a running job

# Live log tail (follow output as it writes)
tail -f logs/seurat_qc_*.out

• tail -f lets you watch progress without opening an interactive session
• After each step confirms State: COMPLETED (not OUT_OF_MEMORY/TIMEOUT), move to the next
• A job killed mid-write leaves a corrupt .rds — always confirm before chaining

Check what a job actually used

seff 123456
sacct -j 123456 --format=JobID,ReqMem,MaxRSS,Elapsed,State,ExitCode

• seff gives a friendly summary: CPU efficiency, peak memory, wall time.
• MaxRSS = peak memory used; Elapsed = wall time.
• Use both to right-size your next submission.

The size trade-off

• Smaller requests run sooner — SLURM slots them into gaps in the schedule.
• But ask for enough — exceed --mem and you get OUT_OF_MEMORY; exceed --time and you get TIMEOUT. Either way, the job dies.
• First run of a new pipeline: err slightly high, then trim using seff.
• Default per-core memory is ~3.7 GB; request more explicitly if you need it.

Why jobs sit in PD

• Priority / Resources — normal queueing; you’re waiting your turn.
• QOSMaxJobsPerUserLimit — you hit a per-user cap; let earlier jobs finish.
• ReqNodeNotAvail, Reserved for maintenance — your --time runs into a maintenance window; shorten it or add --time-min.

Being a good cluster citizen

• Never run heavy work on login nodes — always srun/sbatch.
• exit idle interactive sessions; don’t hoard nodes.
• Don’t request more CPUs/RAM/GPUs than you’ll use.
• Put big intermediates in /scratch/<PIRG>/, not /home/ (scratch purges at 90 days).
• Back up anything you care about — RACS does not back up Talapas.

Why reproducibility matters on a cluster

• On your laptop, R packages live in one user library — it just works.
• On a shared cluster, package versions drift between users and over time.
• Reproducible environments lock down exact versions so results are stable and shareable.

renv — per-project R libraries

• renv captures a snapshot of every package and its version in a renv.lock file.
• renv::restore() rebuilds the exact library on any machine (laptop or Talapas).
• The workshop project ships with a renv.lock — run renv::restore() once after cloning.

# In the R console or batch script, once per project
renv::restore()

Containers — Apptainer on Talapas

• Apptainer (formerly Singularity) runs OCI/Docker containers on HPC clusters.
• A container bundles the OS, R, and all libraries into a single portable image (.sif).
• Pull a container from Docker Hub; submit the .sif file in your batch script.
• Stronger isolation than renv — useful for complex dependency stacks or sharing workflows externally.

# Pull once on a login node
apptainer pull docker://bioconductor/bioconductor_docker:RELEASE_3_19
# Run R inside the container in a batch job
apptainer exec bioconductor_docker_RELEASE_3_19.sif Rscript scripts/01_qc_preprocessing.R

Environments — conda / mamba on Talapas

• conda (and its faster drop-in mamba) builds isolated environments that pin Python, R, and system libraries together — not just R packages.
• Ideal when a workflow mixes languages (e.g. scanpy in Python alongside Seurat in R) or needs non-R tools like samtools.
• Load the cluster module, create the environment once, then activate it at the top of every batch script.
• Lighter weight than a container, but less portable — the environment lives in your filesystem, not a single shareable image.

# Create once on a login node (mamba resolves faster than conda)
module load miniconda3
mamba create -n scrnaseq -c conda-forge -c bioconda r-seurat scanpy samtools
# Activate inside a batch job before running your script
mamba activate scrnaseq
Rscript scripts/01_qc_preprocessing.R

Key takeaways

• Connect with VS Code Remote-SSH (Duck ID + Duo) — green banner = you’re on Talapas.
• The VS Code integrated terminal (Ctrl/Cmd+`) is a shell on the login node — hostname confirms it. Shell skills from P1 all apply here.
• Edit in the editor, run in the terminal — both operate on the same remote filesystem.
• Files live in home (backed up, small), projects (lab-shared), and scratch (large, 90-day purge).
• Load software with module load; always module purge first in batch scripts.
• A scheduler exists because the cluster is shared — declare what you need.
• Login nodes edit and submit; compute nodes compute.
• Interactive to watch, batch to walk away.
• sbatch → squeue → seff, then right-size the next run.
• Lock dependencies with renv (or Apptainer) for reproducible results.

What’s next — Module 10

• You can now connect, navigate file systems, load modules, and submit jobs — the mechanics.
• Module 10 uses these to run the real scRNA-seq pipeline on Talapas: the numbered 01–08 scripts, chained through .rds hand-offs.
• Read the companion chapter for the conceptual deep-dive on schedulers and fair-share.