Chapter 9 — VS Code & SLURM Basics on Talapas

Author

Single Cell RNA-seq Workshop

Note

Where this chapter sits. This is the conceptual reading for the first of the two Friday-morning Talapas modules. Companion to Lecture 09 — VS Code & SLURM Basics on Talapas and Tutorial 09 — VS Code & SLURM Basics on Talapas. The lecture is the concept map and the tutorial is the command-by-command walkthrough; this chapter is the long-form prose that explains why a scheduler behaves the way it does. This chapter also covers first-time login, VS Code Remote-SSH setup, Lmod modules, Talapas file systems, and reproducible environments — everything you need before you submit your first job. Material draws on the Talapas Quick Start Guide and Shannon Snyder’s Talapas Basics.

9.1 What is Talapas, and how do you get an account?

Talapas is UO’s flagship high-performance computing (HPC) cluster. Named from the Chinook word for coyote — the educator and keeper of knowledge — it is managed by Research Advanced Computing Services (RACS) and located in the Allen Hall Data Center. It combines centrally funded open-use resources with researcher-funded (“condo”) resources.

Key resources: over 14,500 CPU cores, 129 TB of memory, roughly 89 GPUs (NVIDIA A40 through H100), and about 3 petabytes of storage. CPU generations span AMD Milan and Intel Broadwell/Cascadelake/Icelake/Sapphire Rapids; node memory ranges from 128 GB to 6 TB.

Documentation: Talapas Knowledge Base.

Talapas Architecture

Getting an account (Duck ID + PIRG)

To work on Talapas you need a UO Duck ID and a Talapas account tied to a research group called a PIRG (Principal Investigator Research Group).

  • A PIRG = Principal Investigator Research Group.
  • Contact your PI to be added to their PIRG, or contact RACS to create a new one.
  • Request an account at racs.uoregon.edu/request-access.
  • Your username is your Duck ID (the part of your UO email before @uoregon.edu).
  • Your password is your University-wide UO password, managed via UO’s password reset page.

9.2 Connecting to Talapas

The Talapas login host is login.talapas.uoregon.edu.

# Basic SSH connection
ssh yourduckid@login.talapas.uoregon.edu

# Or a specific login node (login1 ... login4)
ssh yourduckid@login1.talapas.uoregon.edu
  • For security, no characters appear while you type your password.
  • SSH logins use Duo two-factor authentication: when prompted, type 1 and press Enter to receive a Duo push (2 sends a text, 3 calls you).
  • Off campus? Connect to the UO VPN first, then SSH as normal.
  • Mac and Linux users can set up an ED25519 SSH key to avoid typing a password every session (Duo will still prompt when required by policy).
  • From Windows: use PuTTY, MobaXterm, or Windows Terminal.

RACS provides a web portal at ondemand.talapas.uoregon.edu (Chrome or Firefox recommended):

  • No command line or SSH client needed.
  • File browser for uploads/downloads.
  • Interactive applications: JupyterLab, RStudio, MATLAB, and a remote X11 desktop via VNC.
  • The easiest way to run RStudio or Jupyter sessions on Talapas, and great for beginners.

9.3 VS Code + Remote-SSH: working interactively on Talapas

The most productive day-to-day way to work on Talapas is from an editor that connects directly to the cluster. We recommend VS Code with Remote-SSH, but Open OnDemand (see §9.2) is an excellent browser-based alternative that requires no local installation.

VS Code (Visual Studio Code) is a free, cross-platform editor from Microsoft. Its Remote-SSH extension lets you browse Talapas file systems and edit files there exactly as you would locally, while the integrated terminal gives you a shell on the login node without leaving the editor. The local VS Code setup (installation, laptop-only extensions) is in Chapter P1; this section covers the Remote-SSH configuration that connects VS Code to Talapas.

Connecting VS Code to Talapas over Remote-SSH: install the remote extensions, add the login.talapas.uoregon.edu SSH host, then connect (Duck ID password + Duo) until the green remote banner appears.

Step 1 — Install remote extensions

In the left sidebar, click the Extensions tab, search for ssh, and install:

  • Remote — SSH
  • Remote Explorer
  • Remote — SSH: Editing Configuration Files (optional but recommended)

Also handy for cluster work: the Quarto extension (for editing .qmd files), an R extension, and a PDF viewer / CSV viewer so you can open plots and tables your jobs produce without leaving VS Code.

Step 2 — Add the Talapas SSH host

Open Remote Explorer from the left sidebar, click the + icon next to SSH, and enter the following, replacing [yourduckid] with your Duck ID:

ssh [yourduckid]@login.talapas.uoregon.edu

Press Enter, then Enter again to save to the default config file.

Step 3 — Connect to Talapas

In the Remote Explorer dropdown, expand SSH; you should see login.talapas.uoregon.edu. Click the → arrow to connect, then:

  1. Enter your Duck ID password when prompted (characters will not appear as you type).
  2. When asked for Duo confirmation, type 1 and press Enter for a Duo push (2 sends a text, 3 calls you).
  3. Once connected, a green banner reading SSH: login.talapas.uoregon.edu appears in the bottom-left corner.

Click Open Folder to browse directories on Talapas — open the base directory of your current project. You may also want a second window pointed at your home or project directory, e.g.:

/home/[yourduckid]/nereus

The integrated terminal is a shell on Talapas

This is the most important thing to understand about VS Code Remote-SSH: when you are connected to Talapas, the VS Code integrated terminal is not a terminal on your laptop — it is a shell running on the Talapas login node. Every command you type there executes on the cluster’s filesystem and hardware.

The shell commands you learned in P1 — Computer Systems & the Command Line (pwd, ls, cd, paths, pipes, redirection) all work exactly the same way here — the only difference is that they now run on a remote machine instead of your laptop.

Opening the integrated terminal:

# Keyboard shortcut
Ctrl + `      # Windows / Linux
Cmd  + `      # macOS

# Or via the menu: Terminal → New Terminal

Verify you are on Talapas — the green SSH: login.talapas.uoregon.edu banner in the bottom-left status bar is the primary indicator, but you can confirm with shell commands:

hostname   # e.g. login1.talapas.uoregon.edu  (you are on the login node)
whoami     # your Duck ID
pwd        # your current working directory on Talapas
Note🔑 Key concept — the integrated terminal is a remote shell

When VS Code is connected via Remote-SSH, its integrated terminal opens a shell on the Talapas login node, not on your laptop. The green remote indicator in the status bar confirms this. All shell skills from P1 apply — pwd, ls, cd, pipes — but now they operate on the cluster’s filesystem. module load, srun (interactive jobs), and sbatch (batch jobs) are all typed into this same terminal. This is the single workflow that connects everything in Module 09.

The edit-in-editor / run-in-terminal loop

The core VS Code + Talapas workflow is a tight loop between two panes in the same window:

  • Editor pane (left / main area): open and edit .R, .qmd, or .sbatch files directly on Talapas — your changes are saved to the cluster filesystem immediately, with no need to upload.
  • Terminal pane (bottom): run those files, submit jobs, or tail logs — all running on the same remote machine.

Because both panes operate on the same filesystem, there is no copy-step between “editing” and “running”:

# -- In the integrated terminal (on the login node) --

# 1. Load R (must be done each session — modules do not persist)
module load R/4.4.1

# 2. Quick smoke-test on the login node (small input only)
Rscript scripts/01_qc_preprocessing.R

# 3. Submit the same script as a proper batch job for the full data
sbatch run_rscript.sbatch 01_qc_preprocessing.R

# 4. Watch the job as it runs
tail -f logs/seurat_qc_*.out

# 5. Render a .qmd notebook on the cluster
quarto render my_analysis.qmd

Keep an R or .qmd file open in the editor while its job runs in the terminal — when something goes wrong, edit, save (on the cluster), resubmit, without ever leaving VS Code.

Editor tips for Talapas work

  • You can have multiple VS Code windows open — one per server or project.
  • The file explorer on the left updates in real time, making complex directory structures easy to navigate.
  • View outputs in place. With the PDF/CSV viewer extensions, double-click a .pdf plot, .csv summary, or .out/.err log your job wrote and it opens right in the editor.
  • Comment / uncomment selected lines: Ctrl + / (macOS Cmd + /).
  • Add more terminal tabs with the + at the top-right of the terminal pane, or split the terminal vertically to tail two job logs side by side.
  • Split-pane editing and search-across-files make it easy to work on a script and its log side by side.
Note🔑 Key concept — login node vs compute node

A login node is a small, shared gateway machine: every user on Talapas lands there when they SSH in (or when VS Code opens its integrated terminal), so it handles dozens of simultaneous sessions. Its purpose is navigation, editing, and job submission — not computation. A compute node is a large, dedicated machine that SLURM reserves exclusively for your job. Running a Seurat pipeline or any memory-hungry analysis on the login node does not just slow your own work; it degrades the experience for every other user trying to log in at that moment. Always route real computation through srun, salloc, or sbatch.

The full login-node/compute-node distinction and the SLURM mechanics are developed in §9.9 and §9.12.

9.4 Talapas file systems

Every Talapas user has access to several directories on the ~3 PB IBM Elastic Storage System (GPFS).

Storage Structure
Location Quota Purpose
/home/<duckid> ~20 GB (small) Personal files, scripts, dotfiles; small software installs — not large data or references
/projects/<PIRG> Shared (2 TB default) Large shared project data and analysis for your research group
/scratch/<PIRG> 20 TB High-throughput working space; files untouched for 90 days are purged
/tmp Local to node Temporary scratch space

All navigation commands below are typed in the VS Code integrated terminal — the shell on the login node covered in §9.3. The basic shell commands (pwd, ls, cd, paths, pipes) were introduced in P1 — Computer Systems & the Command Line; what is different here is that they operate on the cluster’s shared parallel filesystem, which looks the same from every login and compute node.

# Check your storage usage (run in the integrated terminal)
df -h ~
du -sh /projects/myPIRG/myusername

# First visit: see your lab space
ls /projects/<PIRG>
ls /scratch/<PIRG>/
Warning

You are responsible for backing up your data — RACS does NOT back up Talapas file systems. Some snapshotting exists for home and project directories, but it is not a substitute for your own backups. Use /projects for large files and clean up when done. The /scratch filesystem purges files not accessed in 90 days — it is for working data, not long-term archives.

File transfer options:

  • scp / sftp from the terminal.
  • FileZilla — cross-platform GUI for SCP/SFTP.
  • Globus — RACS’s recommended tool for large transfers, especially to/from the Large-Scale Research Storage (LSRS) system.

9.5 Software modules (Lmod)

Talapas uses Lmod to manage centrally installed software. Modules are loaded per shell — you must module load again in every new session (or in every SLURM job script).

module spider blast        # search for software by name
module spider R            # find available versions
module avail               # list modules loadable right now
module load R/4.4.1        # load a specific version
module load samtools       # load a module (default version)
module list                # show currently loaded modules
module unload samtools     # unload a module
module purge               # unload everything; reset to defaults
NoteModule tips
  • In batch scripts, always load modules after the #SBATCH directives.
  • Use module spider to find available versions.
  • module purge resets your environment to defaults.
  • Specify version numbers for reproducibility (e.g., R/4.4.1, not just R).
Note🔑 Key concept — modules do not follow you into a batch job

If you module load R on the login node and then submit a batch job, that job starts on a fresh compute node with none of your modules loaded. You must load the modules you need inside the job script itself, after the #SBATCH directives. This is one of the most common sources of “command not found” errors in first batch jobs. The fix is always module purge followed by explicit module load lines inside the script.

9.6 Reproducible compute environments

Getting onto Talapas is only half the battle; you also want your analysis to produce the same results next month and on a co-author’s machine. The workshop ships reproducible environments so you can skip the install dance (Docker / Apptainer), pin exact package versions (renv), or use Talapas’s centrally installed software (module load).

NoteWhy bother?

A pinned environment lets you reproduce a tutorial output exactly six months from now, get the same Seurat version a co-author was running, and avoid two days of BiocManager::install() debugging on Talapas.

Layer 1 — renv lockfile (R level)

renv records the exact version of every R package used in a project in an renv.lock at the repository root — the recommended pattern for a reproducible analysis. (If you adopt it for your own project, the steps below are how you would use it.)

First-time setup in a fresh clone:

install.packages("renv")
renv::restore()       # reads renv.lock and installs the pinned versions

Subsequent R sessions in the project will use the pinned library automatically.

Updating the lockfile (instructors) after upgrading a package:

renv::snapshot()      # rewrite renv.lock from the currently loaded versions

Commit renv.lock (and only the lockfile, not the entire renv/library/ cache).

Layer 2 — Docker / Apptainer image

For a fully self-contained environment (R + Bioconductor + system dependencies + the dataset paths), use the workshop image. Containers package the entire software stack into one portable, versioned artifact: Docker1 is the standard on laptops and in the cloud, while Singularity/Apptainer2 is its HPC-safe cousin (it runs without root, which is why clusters allow it). Combined with per-tool environments from Bioconda3, a container is the strongest reproducibility guarantee available.

Docker (laptop):

# pull the workshop image
docker pull ghcr.io/wcresko/scrnaseq_tutorial:latest

# run RStudio Server, mounting your data folder
docker run --rm -p 8787:8787 \
  -e PASSWORD=workshop \
  -v "$PWD/data:/home/rstudio/data" \
  -v "$PWD:/home/rstudio/scrnaseq_workshop" \
  ghcr.io/wcresko/scrnaseq_tutorial:latest

Then open http://localhost:8787 and log in as rstudio / workshop.

Apptainer (Talapas): Talapas does not run Docker, but it does run Apptainer (formerly Singularity). Convert and run the same image:

# on a login node, request an interactive node first
srun --account=<myPIRG> --partition=interactive --time=2:00:00 \
     --mem=16G --pty bash

# pull the image (one-time)
module load apptainer
apptainer pull docker://ghcr.io/wcresko/scrnaseq_tutorial:latest

# run an interactive R session inside the container
apptainer exec scrnaseq_tutorial_latest.sif R

For batch jobs, the relevant sbatch line becomes:

apptainer exec /path/to/scrnaseq_tutorial_latest.sif Rscript scripts/01_qc_preprocessing.R
Note

The image is rebuilt nightly from the project’s Dockerfile against the latest Bioconductor release; check the GitHub releases page for the SHA of the version used in a given workshop session.

Layer 3 — Talapas modules (no container)

If you would rather use Talapas’s centrally installed software, the modules used by the workshop are:

module purge
module load R/4.4.1            # the R version the workshop standardizes on
module load gcc/13.2.0         # for source builds
module load hdf5/1.14.2        # for HDF5 files (used by some Bioconductor pkgs)
module load apptainer          # only if you'll use the container layer

Add these to your ~/.bash_profile or load them in every job script. For a per-user R library that survives upgrades:

mkdir -p $HOME/Rlibs
echo 'R_LIBS_USER=$HOME/Rlibs' >> $HOME/.Renviron

Then install.packages() and BiocManager::install() will write into $HOME/Rlibs instead of the system library.

Which layer should I use?

Goal Use
“Match the slides exactly on my laptop” renv::restore() (Layer 1)
“Skip installing anything” Docker image (Layer 2)
“Run the tutorials on Talapas” Apptainer + renv::restore() (Layers 2 + 1), or module load (Layer 3)

The files that implement this pattern in a fully reproducible project are an renv.lock (Layer 1), a Dockerfile (Layer 2), and a CI workflow such as .github/workflows/build-image.yml that rebuilds the image on a schedule. For the data these environments are built around, see Datasets.

9.7 What a cluster is, and why it needs a scheduler

So far in this workshop you have run everything on your own laptop. That works for the ifnb dataset because it is small, but it stops working the moment your data grows: a full 10x run with tens of thousands of cells, an integration across many samples, or a pseudobulk model with bootstrapped confidence intervals can exhaust a laptop’s memory or run for many hours. The solution is to move the work to a high-performance computing (HPC) cluster — at the University of Oregon, that cluster is Talapas.

A cluster is not one big computer. It is a collection of many separate machines (nodes), each with its own CPUs, memory, and sometimes GPUs, wired together by a fast network and a shared filesystem. Talapas has thousands of CPU cores, hundreds of GPUs, and many terabytes of RAM spread across those nodes. Crucially, it is shared: dozens of researchers from across the university are competing for those resources at any given moment.

Sharing a finite pool of hardware among many users who all want it at once is fundamentally a scheduling problem. If everyone simply logged in and launched whatever they wanted, two things would go wrong. First, popular nodes would be oversubscribed — twenty jobs fighting for the cores and memory of one machine, each running far slower than it should, some crashing when memory ran out. Second, there would be no fairness: whoever happened to start their job first, or who ran the most jobs, would crowd everyone else out. A scheduler exists to prevent exactly this.

Note🔑 Key concept — what a scheduler is and why you declare resources

A scheduler is the software layer that translates your resource request (CPUs, memory, time, GPU) into an exclusive reservation on a specific compute node, ensuring no two jobs share hardware they did not agree to share. You do not take resources on a cluster — you ask for them by declaring what your job needs. The declaration is a contract: SLURM promises you that much exclusive capacity, and your job promises not to exceed it. Every SLURM skill in this chapter is really about understanding both sides of that contract.

9.8 What SLURM does

SLURM — the Simple Linux Utility for Resource Management4 — is the scheduler that runs Talapas. It is the dominant scheduler on academic and national-lab HPC systems worldwide, so the skills here transfer to almost any cluster you encounter. Its job is to decide when, where, and with what resources every job runs. The mental model is a request-and-grant transaction:

  1. You declare what your job needs: how many CPU cores, how much memory, how long it will run, and whether it needs a GPU.
  2. SLURM looks across all the nodes, finds one (or several) that can satisfy that request without conflicting with jobs already running there, and reserves those resources for you.
  3. Your job runs on that reserved slice of a compute node. When it finishes — or when it exceeds the limits you declared — SLURM reclaims the resources and hands them to the next job in line.

The key idea, and the one most worth internalizing, is that you do not take resources on a cluster — you ask for them. The declaration is a contract: you promise SLURM how much you will use, and SLURM promises you exclusive access to that much. The rest of this chapter is really about understanding both halves of that contract.

Note🔑 Key concept — what a scheduler allocates and why the declaration is a contract

When you submit a job, SLURM does not merely suggest a machine — it makes an exclusive reservation. No other job is permitted to use the CPUs, memory, or GPUs you declared for the duration your job runs. This is fundamentally different from a shared laptop where every process contends for the same RAM. The consequence is bidirectional: SLURM keeps its side (exclusive access), and your job must keep its side (not exceed the requested resources). Exceeding your declared memory (--mem) causes SLURM to terminate the job immediately with OUT_OF_MEMORY; exceeding your declared wall-clock time causes termination with TIMEOUT. Both are hard limits, not warnings. Understanding this is why the chapter repeatedly emphasizes sizing requests honestly — under-requesting kills jobs, over-requesting wastes shared capacity and delays your own start.

Every job at UO must also name an account — your PIRG (Principal Investigator Research Group). This is how usage is tracked and how fair-share accounting (below) knows whose turn it is. If you do not know your PIRG, run groups after logging in; your research group will be listed alongside system groups you cannot charge to. You can set it once in your shell profile so you never have to type --account= again:

echo 'export SLURM_ACCOUNT=<myPIRG>'  >> ~/.bash_profile
echo 'export SBATCH_ACCOUNT=<myPIRG>' >> ~/.bash_profile
source ~/.bash_profile

9.9 Login nodes vs compute nodes

When you ssh into Talapas you land on a login node (login.talapas.uoregon.edu). It is tempting to think of this as “the cluster,” but it is not where your analysis should run. A login node is the front desk: a small, shared machine meant for editing files, browsing your directories, running short commands, and — most importantly — submitting jobs. There are only a handful of login nodes, and every user on the cluster shares them simultaneously.

The actual computation happens on compute nodes, which you never touch directly. Instead, you ask SLURM for one, and SLURM hands you a slice of a compute node that matches your request. A useful metaphor: login nodes are the front desk of a workshop, and compute nodes are the workbenches in the back. You check in at the desk and you fill out a request slip, but you do your sawing and welding at a bench.

Important

Never run heavy computation on a login node. Launching a Seurat pipeline, an integration, or any multi-hour analysis directly on the login node degrades the machine for every user trying to log in, edit, or submit. It is the single most common — and most disruptive — beginner mistake on a shared cluster. Always route real work through srun, salloc, or sbatch.

One consequence of the login/compute split catches almost everyone the first time: software environments do not follow you from the login node into a job. If you module load R on the login node and then submit a batch job, that job starts on a fresh compute node with none of your modules loaded. You must load the modules you need inside the job script itself. We return to this in §9.11.

9.10 Fair-share: how SLURM decides whose turn it is

A naive scheduler might run jobs first-come, first-served. SLURM does something more sophisticated and more equitable: fair-share scheduling. Rather than ordering strictly by submission time, SLURM assigns each job a priority that depends in part on how much your group has used the cluster recently. Groups that have consumed a lot of compute in the recent past are gently nudged down the queue; groups that have used little are nudged up. Over time this balances access so that no single heavy user can monopolize the machine.

Fair-share has a practical, almost counterintuitive implication for how you should size your jobs. Because SLURM also tries to fill small gaps in the schedule (a technique called backfill), a small, well-sized job often starts sooner than a large one, even if the large one was submitted first. If you ask for 4 cores and 16 GB for one hour, SLURM can slot you into a gap between bigger jobs. If you ask for a whole node for two days, it has to wait until a node-sized gap opens. The lesson is not to game the queue — you genuinely cannot — but to size your jobs honestly to what they need, which both gets you running sooner and leaves resources for everyone else.

Note🔑 Key concept — fair-share scheduling and backfill

Fair-share is not first-come-first-served. SLURM tracks how much CPU time each PIRG has consumed in a recent window and uses that history to adjust job priorities: groups that have used a lot recently are deprioritized relative to those that have used little. This rebalances the cluster toward equal long-run access rather than rewarding whoever submits first. The backfill mechanism compounds this advantage for small jobs: SLURM actively looks for gaps in the forward schedule where a small, short request can fit without displacing any higher-priority job. A scRNA-seq QC step requesting 4 cores for 1 hour may start in minutes; a request for an entire node for 3 days waits for a node-sized window. Honest sizing is therefore not just ethical — it is pragmatically faster.

QOS (quality of service) limits ride alongside fair-share. A QOS caps how much one user can hold at once — for example, a maximum number of simultaneously running jobs, or a ceiling on total cores. These limits are why you will sometimes see a job sit in the queue with a reason like QOSMaxJobsPerUserLimit: you have not done anything wrong, you have simply hit a per-user cap, and earlier jobs must finish before the next one starts.

9.11 Partitions: matching jobs to the right hardware

Talapas’s nodes are not identical. Some are ordinary CPU nodes, some carry GPUs, some have enormous amounts of RAM. SLURM groups nodes into partitions — queues that each map to a class of hardware and a maximum allowed wall-clock time. Every job must target a partition, and choosing the right one is how you ask for the right kind of machine. The available partitions are summarized in Table 1.

Table 1: Talapas SLURM partitions. Choose compute for most scRNA-seq steps; use memory only when a step genuinely exceeds a standard node’s RAM; use interactive or interactivegpu for short exploratory sessions. Run sinfo to see the live state of each partition.
Partition What it is for Max time
compute Standard CPU jobs 1 day
computelong Long-running CPU jobs 14 days
gpu / gpulong GPU jobs, short / long 1 / 14 days
memory / memorylong High-memory CPU jobs (up to 4 TB RAM) 1 / 14 days
interactive / interactivegpu Short interactive sessions 12 / 8 hours
preempt Any node, but the owner can bump your job off Variable

Reading Table 1: the most important column is “Max time” — it sets the upper bound you can request with --time. If you request a time longer than the partition allows, SLURM rejects the job immediately. For the ifnb workshop pipeline, all eight core analysis steps finish within 4 hours on a standard compute node; compute is the right choice throughout. The memory partition is for edge cases — atlas-scale integration, very large pseudobulk DE — not a general precaution; its nodes are few and in high demand. The preempt partition gives you access to any node (including those in other partitions), but the owner of that partition can evict your job at any time; use it only for checkpointed workloads that can survive being killed.

Most of the core scRNA-seq pipeline runs comfortably on compute. You reach for memory only when a step genuinely needs more RAM than a standard node provides, for gpu when you run GPU-accelerated tools, and for interactive when you want a quick session that starts fast. Running sinfo shows you each partition and the live state of its nodes (idle, allocated, down) so you can see where capacity is currently free.

A common beginner mistake is to use memory or memorylong as a precaution — “just in case” — rather than because a step genuinely needs it. The high-memory nodes are few and heavily contended; using them unnecessarily prevents other users’ legitimate high-memory work from starting. Conversely, using compute for a step that truly needs 200 GB will result in an OUT_OF_MEMORY kill mid-run. The right calibration comes from empirical measurement: run the first instance of any new step with a modest over-estimate, then check seff afterward and right-size subsequent submissions (§9.14).

9.12 Interactive vs batch: two ways to use a compute node

There are two fundamentally different ways to get work onto a compute node, and choosing between them is mostly a question of whether you intend to watch the job or walk away from it. Use an interactive session (srun/salloc) when you’ll sit and watch: testing or debugging a script, exploring data by hand, or running something short. Use a batch job (sbatch) for anything long-running, or anything you want to start and walk away from — a batch job keeps running after you log out.

Interactive sessions (srun --pty bash or salloc) give you a live shell on a compute node. You type commands, you see output immediately, and the session stays alive as long as you keep it open or until its time limit is reached. This is the right mode when you are testing or debugging a script, exploring data by hand, or running something short enough to sit and wait for. A minimal interactive request looks like this:

For quick, short sessions the interactive partition is usually faster to start than compute:

srun --account=<myPIRG> --partition=interactive --time=2:00:00 --mem=8G --pty bash

If you need a GPU interactively, use --partition=interactivegpu and add --gpus=1 --constraint=gpu-10gb (or gpu-40gb/gpu-80gb for the appropriate memory tier). For graphical tools such as RStudio, the recommended path is Open OnDemand (https://ondemand.talapas.uoregon.edu/): sign in with your Duck ID and launch an RStudio or Jupyter session from the web portal — no X server or X11 forwarding required.

srun --account=<myPIRG> --partition=compute \
     --time=1:00:00 --mem=4G --cpus-per-task=2 --pty bash

When the resources are granted, your shell prompt changes to show the name of the compute node you landed on (for example n0123). You are now on that node, working directly. salloc is its close cousin: it requests an allocation from a login node, and when granted, moves your prompt onto the compute node so you can launch several steps into the same allocation. Either way, the discipline is the same — type exit the instant you are done, because an interactive session holds its reserved resources whether or not you are actively typing. Idle sessions are pure waste; they hoard a node that someone else could be using.

Batch jobs (sbatch) are the opposite. You write a script that describes the job and the commands to run, hand it to SLURM, and then you are free to log out. The job waits in the queue, runs when its turn comes, and writes its output to log files — all without any terminal attached. This is the mode for anything long-running, and it is how the real workshop analysis is done. The crucial advantage is durability: a batch job survives your laptop going to sleep, your network dropping, and you logging off for the night.

A batch script is an ordinary shell script with a header of #SBATCH directives — the resource declaration — at the very top, before any commands:

#!/bin/bash
#SBATCH --account=<myPIRG>
#SBATCH --partition=compute
#SBATCH --job-name=seurat_qc
#SBATCH --output=logs/seurat_qc_%j.out
#SBATCH --error=logs/seurat_qc_%j.err
#SBATCH --time=04:00:00
#SBATCH --cpus-per-task=8
#SBATCH --mem=64G

module purge
module load R/4.4.1
Rscript scripts/01_qc_preprocessing.R

Two details deserve emphasis. First, the module purge and module load lines live inside the script — as noted in §9.9, the environment does not carry over from your login session, so the job must build its own. Second, the --output path points into a logs/ subdirectory. SLURM will not create that directory for you. If logs/ does not exist, the job is rejected (sometimes silently, with no log to explain why, because the log is exactly what could not be written). The one-time fix, run once from the directory you submit from, is mkdir -p logs.

The most useful #SBATCH flags map directly onto the resource contract:

  • --account=<myPIRG> — the PIRG to charge (required).
  • --partition=<name> — which queue, and therefore which hardware.
  • --time=<HH:MM:SS> — the wall-clock limit; exceed it and SLURM kills the job (TIMEOUT).
  • --mem=<size> or --mem-per-cpu=<size> — memory, total or per core.
  • --cpus-per-task=<N> — cores for a multi-threaded step.
  • --gpus=<N> with --constraint=gpu-40gb — GPUs and the memory tier you need.
  • --mail-type=END,FAIL plus --mail-user=you@uoregon.edu — email notification when the job ends or fails; useful for long runs.
  • --array=1-N — submit N independent sub-jobs from one script, each reading a sample name from a list file; the index is available as $SLURM_ARRAY_TASK_ID. This is the standard pattern for running Cell Ranger (or any per-sample step) at scale without writing N separate scripts.
  • %j in a filename expands to the job ID, keeping each run’s logs distinct.

A related pair of flags controls behavior near maintenance windows: --time-min=<HH:MM:SS> tells SLURM the minimum time your job actually needs, allowing it to start the job in a shorter gap even if it cannot guarantee the full --time. This is the recommended fix for jobs sitting in the queue with reason ReqNodeNotAvail, Reserved for maintenance.

One additional point on environment setup: every batch script should include set -euo pipefail as the first non-comment command. This trio of shell options makes the script exit immediately if any command fails (-e), treats unset variables as errors (-u), and propagates errors through pipes (-o pipefail). Without it, a script can silently continue past a failed step — writing a corrupt output, loading the wrong module, or producing nothing at all — with no indication anything went wrong until a confusing downstream error surfaces. For example: if module load R/4.4.1 silently fails because the module name changed, and set -euo pipefail is absent, the script proceeds to run Rscript ... with no R loaded, producing a mysterious “command not found” or, worse, using a system R that has none of your packages. With set -euo pipefail, the failed module load terminates the script immediately and the .err log shows exactly where it stopped.

Table 2 summarizes the most-used #SBATCH directives in a quick-reference form.

Table 2: #SBATCH directive quick reference. All directives must appear at the top of the script, before any commands. The logs/ directory must exist before submission — SLURM does not create it.
Directive Purpose Example
--account= PIRG to charge (required) --account=cresko
--partition= Queue / hardware class --partition=compute
--job-name= Human-readable label in squeue --job-name=seurat_qc
--output= Stdout log path (%j → job ID) --output=logs/qc_%j.out
--error= Stderr log path --error=logs/qc_%j.err
--time= Wall-clock limit (HH:MM:SS); exceed → TIMEOUT kill --time=04:00:00
--mem= Total memory for the job --mem=64G
--mem-per-cpu= Memory per core (alternative to --mem) --mem-per-cpu=8G
--cpus-per-task= Cores for a multi-threaded step --cpus-per-task=8
--gpus= Number of GPUs --gpus=1
--constraint= GPU memory tier --constraint=gpu-40gb
--mail-type= Notification events --mail-type=END,FAIL
--mail-user= Email for notifications --mail-user=you@uoregon.edu
--array= Submit N independent sub-jobs; index is $SLURM_ARRAY_TASK_ID --array=1-8
--time-min= Minimum acceptable wall-clock for backfill scheduling --time-min=01:00:00

Reading Table 2 as a decision checklist: every job must have --account, --partition, and --time; the first two are required, and the third has a partition-dependent default that is often too long for small jobs and too short for large ones. The --output and --error directives keep stdout and stderr in separate log files, which is essential for debugging — R’s error messages go to stderr (the .err file), while any progress-printing the script does goes to stdout (the .out file). Using %j in the filename embeds the job ID, so logs from repeated runs of the same script do not overwrite each other. The --array directive (last row) is the secret weapon for multi-sample work and is described in depth in the key-parameter callout below.

Note⚙️ Key parameter — --mem

Default on Talapas: ~3.7 GB per core (derived from the node’s total memory divided by its cores). If you need more than one core’s worth of RAM and do not specify --mem or --mem-per-cpu, your job receives only this default and will be killed with OUT_OF_MEMORY the moment it needs more. For most scRNA-seq steps, specify --mem explicitly — for example --mem=64G for integration or --mem=96G for pseudobulk DE. Setting it too high delays queue placement; setting it too low kills the job mid-run. After each first run, read seff <jobid> and note MaxRSS; trim your next request to that value plus a 15–20% safety margin.

Note⚙️ Key parameter — --cpus-per-task

Default: 1. This is the number of CPU cores allocated to your job on a single node. For multi-threaded tools (Seurat’s FindAllMarkers, STAR inside Cell Ranger, Harmony) you raise this — for example --cpus-per-task=8. The important distinction is between --cpus-per-task (threads within one process) and --ntasks (separate MPI processes). Most bioinformatics tools are multi-threaded, not MPI programs, so you almost always want --cpus-per-task, not --ntasks. Check what seff reports as CPU Efficiency after a run: if you asked for 8 cores but efficiency is 12%, the tool is not parallelizing and you should drop back to 1–2 cores.

Note⚙️ Key parameter — --time

Default: 24 hours on compute, 14 days on computelong. When your job exceeds this wall-clock limit, SLURM kills it with TIMEOUT. The job is terminated at exactly that boundary — there is no grace period. For a new pipeline, run once with a generous time limit and note Elapsed from seff; set subsequent submissions to Elapsed × 1.2. Use the --array directive (§9.12) rather than requesting unrealistically long time for jobs that process many samples — arrays let each sub-job have its own fair time limit and restart independently if one fails.

Note⚙️ Key parameter — --partition

Default: none — required. You must specify a partition for every job; SLURM will not accept a submission without one. For standard scRNA-seq steps that finish within a day, --partition=compute is the correct choice. Reach for --partition=computelong only when a step genuinely requires more than 24 hours (rare for the ifnb workflow). Use --partition=memory or --partition=memorylong only when a step truly exceeds a standard node’s RAM — not as a precaution, because those nodes are scarce. The --partition=interactive is for short (≤12 hour) exploratory sessions that need to start quickly, not for production analysis.

Note⚙️ Key parameter — --array

Syntax: --array=1-N (or 1-N%M to limit M simultaneously running sub-jobs). A job array submits N independent sub-jobs from a single script, with each receiving a unique $SLURM_ARRAY_TASK_ID (1 through N). Inside the script you use that ID to select the sample to process — for example SAMPLE=$(sed -n "${SLURM_ARRAY_TASK_ID}p" samples.txt). This is the standard pattern for running Cell Ranger or any per-sample step on many samples without writing N separate scripts. The %M throttle is important when N is large: --array=1-50%4 allows at most 4 sub-jobs to run simultaneously, so you don’t monopolize the queue for everyone else.

9.13 Submitting, monitoring, and reading the result

The lifecycle of a batch job is short to describe. You submit it, SLURM queues it, it runs, and it finishes — and a small set of commands lets you follow it the whole way. Understanding the difference between these commands matters: squeue shows the current state of a job while it is in the system; once a job finishes and leaves the queue, you must use seff or sacct to retrieve information about it. Many beginners check squeue, see nothing, and conclude the job never ran — when in fact it completed successfully and left the queue before they checked. Always pair squeue (for live monitoring) with seff (for post-mortem accounting).

sbatch hostname.sbatch        # -> Submitted batch job 123456
squeue -u $USER               # all of your jobs
squeue -j 123456              # one specific job
scancel 123456                # cancel a queued or running job
scancel -u $USER -t PENDING   # cancel all your pending (not yet running) jobs

In squeue you will see a state code for each job. PD means pending — it is queued but not yet running. R means running. CG means completing — it has finished and SLURM is cleaning up. When the job finishes, the .out (stdout) and .err (stderr) files appear in the directory you submitted from; the .err file is where R errors land, so it is the first place to look when something went wrong.

After a job completes, you can ask SLURM what it actually used — which is the foundation of right-sizing future runs:

seff 123456
sacct -j 123456 --format=JobID,ReqMem,MaxRSS,Elapsed,State,ExitCode

seff prints a friendly summary: CPU efficiency, peak memory, and wall time, plus the final state. sacct gives the same facts in tabular detail. Table 3 describes the key fields from both commands. The two numbers to watch are MaxRSS (the peak memory the job actually touched) and Elapsed (how long it actually ran). If you requested 64 GB and MaxRSS says you used 12 GB, your next submission can safely ask for far less — and will start sooner for it.

Table 3: Key fields from seff, sacct, and squeue and what each tells you about a job. Run seff <jobid> after every job as a habit; the Memory and CPU efficiency numbers are the foundation of right-sizing future submissions.
Field / output line Source What it means How to use it
State seff, sacct Final job status: COMPLETED, FAILED, OUT_OF_MEMORY, TIMEOUT, CANCELLED Anything other than COMPLETED means the job did not finish cleanly
CPU Efficiency seff Fraction of allocated CPU time that was actually used Low efficiency on a single-threaded job is normal; low efficiency on a requested-8-core job means the code is not parallelizing
Memory Efficiency seff Peak memory used / memory requested If consistently < 50%, shrink --mem on the next run to improve queue priority
Job Wall-clock time seff Actual elapsed time Use with a 10–20% safety margin to set --time on repeated runs
MaxRSS sacct Peak resident set size (memory high-water mark) The definitive right-sizing number; convert MB to GB with ÷ 1024
Elapsed sacct Actual wall-clock time in HH:MM:SS Same as seff wall-clock; more scriptable from sacct output
ExitCode sacct Process exit code (format code:signal) 0:0 = clean exit; any non-zero code indicates an error in the job itself
PD squeue State Pending — queued but not yet running Normal; check the Reason column for why
R squeue State Running Job is actively executing
CG squeue State Completing — job finished, SLURM cleaning up Transient; will disappear shortly

Reading Table 3: the most immediately useful line from seff is Memory Efficiency. If it reads “19% of 64 GB,” your next submission should ask for roughly 64 × 0.19 × 1.2 ≈ 15 GB — a substantial reduction that will move you up the queue. CPU Efficiency deserves a different interpretation depending on the job: low efficiency on a single-threaded step is expected, because seff computes efficiency relative to the allocated cores; low efficiency on a step you requested 8 cores for is a signal that the tool is not actually multithreading and you should reduce --cpus-per-task. The ExitCode field is the clearest binary: 0:0 is clean; anything else is a failure. A job with State: COMPLETED and ExitCode: 1:0 means the job ran to the time limit but the R script inside it errored — you must look at the .err log to find the actual failure message.

9.14 Right-sizing: the central skill

Resource requests pull in two opposite directions, and learning to balance them is the single most valuable HPC skill.

On one side, smaller requests run sooner. SLURM’s backfill scheduler slots small jobs into gaps that a large job could never fit, so over-asking for memory or time directly delays your own work. On the other side, you must request enough. If a job exceeds its --mem, SLURM kills it with OUT_OF_MEMORY; if it exceeds its --time, SLURM kills it with TIMEOUT. In both cases the job dies, often after hours of progress, and you start over.

The practical resolution is empirical. The first time you run a new pipeline step, err slightly on the high side so it does not die. Then read seff to see what it really needed, and trim subsequent submissions toward that number with a modest safety margin. A couple of defaults are worth memorizing: the default memory per core is roughly 3.7 GB, so if your step needs more you must say so explicitly; and the compute partition’s default wall time is 24 hours, while computelong defaults to two weeks. In the scRNA-seq pipeline, most steps are comfortable on a standard node, but pseudobulk DE and differential abundance are memory-hungry and benefit from an explicit --mem=96G.

The right-sizing workflow in practice is a two-submission loop. First submission: set generous but not absurd limits — for an unknown Seurat step, --mem=64G --time=04:00:00 --cpus-per-task=4 is a reasonable starting point. Run seff <jobid> immediately after completion. Second and later submissions: set --mem to MaxRSS × 1.2 (20% safety margin), --time to Elapsed × 1.2, and --cpus-per-task to whatever the tool actually uses (check CPU Efficiency from seff). A step that requested 64 GB but used only 12 GB has a MaxRSS-based new request of roughly 15 GB — a fourfold reduction that significantly improves its queue priority. Over the course of running the full 01–08 pipeline twice, you will have accurate resource profiles for all eight steps, and subsequent runs start promptly because each step requests only what it genuinely needs.

9.15 Why is my job stuck in PD?

A job sitting in the pending state is normal far more often than it is broken. The reason SLURM gives (visible in squeue’s reason column or via scontrol show job <id>) tells you which it is; the most common reasons are collected in Table 4.

Priority or Resources indicates ordinary queueing: either there are jobs ahead of you in the fair-share ordering, or the specific resources you asked for are not free yet. There is nothing to fix — you are simply waiting your turn. QOSMaxJobsPerUserLimit means you have hit a per-user cap on simultaneous jobs; let some earlier jobs finish and the next will start automatically. The most disruptive reason is ReqNodeNotAvail, Reserved for maintenance: your requested --time would run past an upcoming quarterly maintenance window, so SLURM cannot guarantee the node will be free for the whole duration. The fix is to shorten --time, or to add a --time-min so SLURM can start the job sooner in a shorter gap:

#SBATCH --time=0-08:00:00
#SBATCH --time-min=0-01:00:00
Table 4: Common squeue pending reasons, their meaning, and the recommended action. The Reason column appears in squeue -u $USER; for more detail run scontrol show job <id>.
Pending reason What it means Action
Priority Jobs with higher fair-share priority are ahead of you Wait; nothing to fix
Resources The resources you requested are not currently free Wait; consider reducing --mem or --cpus-per-task
QOSMaxJobsPerUserLimit You have hit the per-user simultaneous-job cap Wait for earlier jobs to finish
ReqNodeNotAvail, Reserved for maintenance Your --time extends past a maintenance window Add --time-min or shorten --time
AssocGrpCPUMinutesLimit Your PIRG account has exhausted its CPU-minute allocation Contact RACS
Dependency The job was submitted with --dependency= and the upstream job has not yet completed Normal in dependency chains; nothing to fix

Reading Table 4: the most important lesson is that most pending states require no action — they resolve when resources free up. The only reasons that require active intervention are ReqNodeNotAvail, Reserved for maintenance (shorten --time or add --time-min) and AssocGrpCPUMinutesLimit (contact RACS, because your PIRG has exhausted its allocation). Attempting to “fix” a Priority-pending job by cancelling it and resubmitting with shorter resource requests is logical in theory but rarely helps in practice: you move to the back of a queue whose front is also held up by the same resource contention. The most efficient response to Priority or Resources is to wait — and to note from seff on past runs whether your requests are larger than necessary.

The official Knowledge Base FAQ, Why is my SLURM job not running?, catalogs the less common reasons.

9.16 Being a good cluster citizen

Because Talapas is shared across the whole university, the etiquette of using it is not just politeness — it is what keeps the machine usable for everyone, including you.

Never run intensive work on login nodes; always route real computation through srun or sbatch. Do not leave idle interactive sessions open — type exit the moment you are done so the reserved node returns to the pool. Do not request more CPUs, memory, or GPUs than you will actually use: over-requesting wastes shared capacity and, counter-intuitively, delays your own jobs by making them harder for the backfill scheduler to place.

Put large output — caches and intermediate files — in /scratch/<PIRG>/, not in /home/. The scratch filesystem has generous per-PIRG quotas that home directories lack, but note that /scratch purges files not accessed in 90 days, so it is for working data, not long-term archives. Back up anything you care about: RACS does not back up Talapas, and a disk failure or an accidental deletion is final without your own copy. If you have a genuine deadline that needs special scheduling, email RACS rather than trying to outmaneuver the queue.

Do

  • Request only the resources you need.
  • Test with small datasets first.
  • Use --time conservatively.
  • Clean up old files regularly.
  • Use /projects for large data.

Don’t

  • Run heavy computation on login nodes.
  • Leave jobs running indefinitely.
  • Store sensitive data without encryption.
  • Forget to acknowledge RACS in publications.

Acknowledgment text

“The authors acknowledge Research Advanced Computing Services (RACS) at the University of Oregon for providing computing resources that have contributed to the research results reported within this publication.”

9.17 Reading the output: what a finished batch job leaves behind

After a batch job completes, check two things in combination before treating the run as successful.

First, look at the .out log (stdout). For jobs that print a “Wrote …” or “Done” line when they finish successfully (as the workshop R scripts do), the presence of that line is the true signal of completion — a job killed mid-run by an OOM or timeout may still write a file, but the log will not contain the success message.

Second, run seff <jobid> and verify that State: COMPLETED. The states OUT_OF_MEMORY and TIMEOUT mean SLURM killed the job before it finished. Both can leave a partial output file on disk that appears valid but is corrupt — the next step in the pipeline will then fail with a confusing downstream error rather than the root cause.

The two fields to read in seff output are CPU Efficiency (for future tuning), Memory Efficiency (if you requested 64 GB and used 12 GB, your next run can ask for far less), and Job Wall-clock time (to judge whether the time limit was adequate). If seff shows zero for CPU efficiency, the job was almost certainly killed before it could run.

9.18 Workshop scripts: hello_world.R and the generic wrappers

The Talapas scripts folder ships two items that the tutorial uses directly:

  • hello_world.R — a minimal base-R demo (no extra packages) that prints the compute node it ran on and writes a PDF plot and a CSV summary. It is the first thing you run on an interactive compute node to confirm the environment works: load R (module load R), then Rscript hello_world.R. Note: module load R must be run on the compute node, inside the interactive session or batch script — the module does not carry over from the login node.
  • run_rscript.sbatch — a generic batch wrapper whose essential line is Rscript "${SCRIPT}". Rather than writing a separate .sbatch file for every R script, you pass the R script as an argument and override --job-name per run: sbatch --job-name=qc run_rscript.sbatch 01_qc_preprocessing.R. This is the pattern used throughout the Module 10 pipeline. After submission, follow the job with squeue -u $USER and read the .out/.err logs in logs/ when it finishes; the .err file is where R error messages and warnings land.
  • run_core_pipeline.sbatch — runs all eight core analysis scripts (01–08) in sequence inside a single SLURM allocation (~12 hours, 96 GB). Submit once with sbatch run_core_pipeline.sbatch after mkdir -p logs and pre-caching the ifnb dataset; confirm completion with seff <jobid>.

All three require a logs/ subdirectory in the directory you submit from. SLURM will not create it for you; run mkdir -p logs once before submitting any job.

9.19 Where this leads

This chapter gave you everything you need to start working on Talapas: how to connect with VS Code Remote-SSH, how to manage files and load software with Lmod, how to set up a reproducible environment, and the full mechanics of the SLURM scheduler — how it thinks, the difference between login and compute nodes, how to size and submit and monitor a job, and how to behave on a shared system. None of the SLURM mechanics are specific to scRNA-seq — these same skills apply to any analysis you ever run on Talapas.

Chapter 10 — The Talapas Analysis Pipeline puts them to work. There, the numbered 01–08 analysis scripts become a chain of SLURM batch jobs, each one’s output feeding the next, reproducing the entire ifnb workflow on the cluster. For the official UO references behind this chapter, see the Talapas Quick Start Guide, the partition list, and the canonical SLURM documentation.

Going further

The original design of the scheduler itself is described by Yoo, Jette & Grondona4.

References

1.
Merkel, D. Docker: Lightweight linux containers for consistent development and deployment. Linux Journal 2014, 2 (2014).
2.
Kurtzer, G. M., Sochat, V. & Bauer, M. W. Singularity: Scientific containers for mobility of compute. PLoS ONE 12, e0177459 (2017).
3.
Grüning, B. et al. Bioconda: Sustainable and comprehensive software distribution for the life sciences. Nature Methods 15, 475–476 (2018).
4.
Yoo, A. B., Jette, M. A. & Grondona, M. SLURM: Simple linux utility for resource management. Lecture Notes in Computer Science 2862, 44–60 (2003).
Back to top