--- title: "Tutorial 09 — VS Code & SLURM Basics on Talapas" subtitle: "Connect to Talapas with VS Code Remote-SSH, explore the cluster interactively, and run your first SLURM jobs" author: "Single Cell RNA-seq Workshop" format: html: toc: true toc-depth: 3 code-fold: false code-overflow: wrap highlight-style: github embed-resources: true execute: eval: false echo: true warning: false message: false editor: visual --- ::: {.callout-note title="Running the code in this tutorial"} Every code chunk here is tagged with `#| eval: false`, so the published page shows the code **without running it**. To run it yourself, open the downloaded `.qmd` in **RStudio**: - **Run one chunk:** click the green ▶ (*Run Current Chunk*) at the chunk's top-right, or press *Ctrl/Cmd + Enter*. This runs the chunk **regardless of its `eval` setting** — the easiest way to work through the tutorial interactively. - **Run a chunk on render:** change that chunk's `#| eval: false` to `#| eval: true` (or delete the line) so it executes when the document is rendered. - **Render the whole tutorial:** click the **Render** button in RStudio (or run `quarto render` in a terminal) to execute the chunks top-to-bottom and knit a finished HTML. To run everything on render, set `eval: true` once in the YAML header at the top. ::: ## About this tutorial This is the **first of the two Friday-morning Talapas modules** (the second is [Tutorial 10 — the analysis pipeline](Tutorial_10_Talapas_Pipeline.html)). It has two parts: first you **connect VS Code to Talapas** over Remote-SSH, explore the Lmod module system, and navigate the file systems; then you learn the **basics of SLURM** (the job scheduler on the University of Oregon's Talapas HPC cluster) --- both in **interactive mode** and through **batch scripts** --- and end with an example batch script that runs the whole `01`–`08` analysis series. It is aimed at workshop participants who have a Duck ID and Talapas account and are connecting for the first time on Friday morning. ::: callout-note **Companion lecture:** [Lecture 09 — VS Code & SLURM Basics on Talapas](../Lecture_Folder/Lecture_09_SLURM_Basics.html) · **Companion reading:** [Chapter 9 — VS Code & SLURM Basics on Talapas](../Resources_Folder/Chapter_09_SLURM_Basics.html) ::: All examples are specific to UO and Talapas. The VS Code Remote-SSH connection to Talapas is the **first step of this module** (Section 2 below) — you do not need any prior cluster login to start here. Material is adapted from the [Talapas Quick Start Guide](https://uoracs.github.io/talapas2-knowledge-base/docs/quickstart_guide.html) and the [How-To articles](https://uoracs.github.io/talapas2-knowledge-base/) in the Talapas Knowledge Base, and from **Shannon Snyder's *Talapas Basics*** workshop guide (the `hello_world.R` / `salloc` / job-monitoring walkthroughs below). ::: {.callout-tip title="How to use this page"} Read this page top to bottom the first time. Later, treat it as a reference --- the commands here cover \~95% of what you will do in the workshop. ::: ------------------------------------------------------------------------ ## 1. Why SLURM? Talapas is a shared cluster with thousands of CPU cores, hundreds of GPUs, and many terabytes of RAM. Dozens of users compete for those resources at any given time. **SLURM** (Simple Linux Utility for Resource Management) is the scheduler that decides *when*, *where*, and *with what resources* each job runs. At UO, this means: - You **never** run heavy computation directly on a login node (`login.talapas.uoregon.edu`). Login nodes are for editing, short commands, and submitting jobs only. - Instead, you ask SLURM for resources --- CPUs, GPUs, memory, time --- and SLURM allocates a **compute node** for you. - Every job at UO must specify an **account** (your PIRG, Principal Investigator Research Group) for correct service charging. ## 2. Prerequisites Before you start this module, you need: 1. **The P1 and P2 (Tuesday) material done on your laptop** — [P1 — Computer Systems & the Command Line](../Resources_Folder/Chapter_P1_Computer_Systems.html) and [P2 — R & RStudio](../Resources_Folder/Chapter_P2_R_and_CommandLine.html). You should have VS Code installed, the Quarto and R extensions installed, and have run R locally. 2. A **UO Duck ID** and your University-wide UO password. 3. A **Talapas account** tied to a **PIRG** (Principal Investigator Research Group). Ask your PI to add you to their PIRG, or request access at [racs.uoregon.edu/request-access](https://racs.uoregon.edu/request-access). 4. If you are **off campus**, connect to the **UO VPN** first — the Talapas login nodes are only reachable from the campus network or the VPN. Throughout this module, replace `` with your Duck ID and `` with your PIRG name. If you are not sure which PIRG you belong to, you will find out in Section 4 by running `groups` once you are logged in. ::: {.callout-note title="Talapas account not set up yet?"} If you do not yet have a Talapas account, request one at [racs.uoregon.edu/request-access](https://racs.uoregon.edu/request-access) before Friday morning. Account provisioning can take 1–2 business days. ::: ------------------------------------------------------------------------ ## 3. Connect VS Code to Talapas ### 3.1 Install the Remote-SSH extension In VS Code's **Activity Bar** (left edge), click the **Extensions** icon (four-squares) and search for and install: - **Remote — SSH** — lets you open folders and run terminals on Talapas directly from VS Code. - **Remote Explorer** — the sidebar panel for managing SSH hosts. After installing, a new **Remote Explorer** icon appears in the Activity Bar. ::: {.callout-note title="Quarto and R extensions from P1"} You installed the **Quarto** and **R** extensions on your laptop in the [Chapter P1](../Resources_Folder/Chapter_P1_Computer_Systems.html) session. VS Code installs extensions *per host*, so once you connect to Talapas you will be prompted to install them on the remote side as well — do so when asked. ::: ### 3.2 Add the Talapas SSH host 1. Open **Remote Explorer** from the Activity Bar. 2. Make sure the dropdown at the top is set to **SSH** (sometimes labelled *SSH Targets*). 3. Click the **+** ("Add New") icon, and in the prompt that appears type the SSH command below, replacing `` with your Duck ID: ```{bash} #| label: M9-ssh_yourduckid_login_talapas #| eval: false ssh @login.talapas.uoregon.edu ``` 4. Press **Enter**, then **Enter** again to save it to the default SSH config file (`~/.ssh/config`). `login.talapas.uoregon.edu` now appears as a host under **SSH** in the Remote Explorer. ### 3.3 Connect (Duck ID + Duo) 1. In **Remote Explorer**, find `login.talapas.uoregon.edu` and click the **→** (Connect) arrow — choose to connect in the current window or a new one. 2. If asked about the platform of the remote host, choose **Linux**. 3. Enter your **Duck ID password** when prompted. For security, **no characters appear as you type** — that is expected. 4. When prompted for **Duo**, type `1` and press **Enter** to receive a Duo push (typing `2` sends a text, `3` calls you). Approve the push on your phone. 5. Once connected, a **green banner** reading **SSH: login.talapas.uoregon.edu** appears in the bottom-left corner of the window. If the connection hangs or is refused: confirm you are on the **UO VPN** (if off campus), that your Duck ID password is current, and that your Talapas account is active. ::: {.callout-important title="Think about it"} You are now connected, and the green banner says `login.talapas.uoregon.edu`. Which kind of node are you on — and what are you allowed to do here?
Show answers You are on a **login (head) node**. Login nodes are **shared by everyone** and are for *light* tasks only: browsing and editing files, managing data, and submitting jobs. They are **not** for running real analysis — heavy processes on a login node will be throttled or killed, and they slow the node down for every other user. To do actual computing you must move onto a **compute node**, which you do through SLURM in Sections 7–8.
::: ------------------------------------------------------------------------ ## 4. Open a terminal and explore the cluster ### 4.1 Identity and groups Open VS Code's integrated terminal with **Terminal → New Terminal** (or `` Ctrl+` ``). The prompt will show one of the Talapas login nodes (e.g. `login1`). ```{bash} #| label: M9-whoami_duck_id #| eval: false whoami # your Duck ID groups # your PIRG(s) appear alongside system groups pwd # where am I? -> /home/ ``` ::: {.callout-important title="Think about it"} Why do you need a PIRG just to run a job? What is it used for?
Show answers A PIRG is your research group's **account** on Talapas. SLURM uses it to track and charge resource usage to the right group, and to enforce each group's share of the cluster. Every job you submit — interactive or batch — must name an account with `--account=`, or it will be rejected.
::: ### 4.2 Lmod modules Talapas manages centrally installed software with **Lmod** modules. Software is **not** available until you `module load` it — and modules are loaded **per shell**, so you reload them in every new session and in every job script. ```{bash} #| label: M9-module_avail_list_modules #| eval: false module avail # list modules you can load right now module spider R # search for R and see which versions exist module load R # load the default R module list # show what is currently loaded R --version # confirm R is now on your PATH module purge # unload everything (reset) when you want a clean slate ``` ::: {.callout-important title="Think about it"} You ran `R --version` before `module load R` and got "command not found", but after loading it worked. Why isn't R just always available?
Show answers On a shared cluster, many versions of many tools coexist, and different projects need different (sometimes conflicting) versions. Rather than putting everything on everyone's `PATH` at once — which would cause clashes — Talapas keeps software **modular**: nothing analysis-related is loaded by default, and you opt in to exactly what you need with `module load`. Because this happens **per shell**, a fresh terminal (or a new job) starts clean, so you must load your modules again. For reproducibility, pin the version — `module load R/4.4.1` rather than bare `R` — so you get the same R every time.
::: ### 4.3 Navigate the Talapas file systems Talapas gives you several places to put files, each with a different purpose and quota: ```{bash} #| label: M9-personal_home_directory_250 #| eval: false # Your personal home directory (250 GB) — scripts, small files, dotfiles cd /home/ pwd ls -la # Your group's shared project space (large, ~2 TB default) — shared data & analysis ls /projects/ # High-throughput scratch (20 TB) — fast working space. # WARNING: files untouched for 90 days are PURGED. Not for anything you can't regenerate. ls /scratch/ # Check how much space you're using df -h ~ du -sh /home/ ``` ::: {.callout-warning title="Scratch purge warning"} Files in `/scratch` that have not been accessed for **90 days are automatically deleted**. Never store anything there that you cannot regenerate. Keep scripts and final outputs in `/home` or `/projects`. ::: ::: {.callout-important title="Think about it"} You have a 40 GB intermediate file you'll re-read many times this week, the small R scripts that generate it, and the final figures you want to keep. Where does each belong — `/home`, `/projects`, or `/scratch`?
Show answers - **Scripts and final figures → `/home`** (or `/projects` if the group should share them). `/home` is small (250 GB) but persistent and snapshotted; it is the right place for code and small keepers. - **The 40 GB intermediate → `/scratch`**. Scratch is large and fast and exactly meant for big working files — but remember the **90-day purge**, so never leave anything there you can't recreate. - **Anything the whole group needs → `/projects`** (large, shared, more durable than scratch). And in all cases: **RACS does not back up Talapas** — keep your own backups of anything irreplaceable.
::: Optionally, set your SLURM account once in `~/.bash_profile` so you don't type `--account=` every time: ```{bash} #| label: M9-echo_export_slurm_account #| eval: false echo 'export SLURM_ACCOUNT=' >> ~/.bash_profile echo 'export SBATCH_ACCOUNT=' >> ~/.bash_profile source ~/.bash_profile ``` Replace `` with your actual PIRG name. ------------------------------------------------------------------------ ## 5. Key SLURM commands at a glance | Command | Purpose | |----|----| | `sinfo` | Show partitions and node states. | | `squeue -u $USER` | Show *your* queued/running jobs. | | `sbatch script.sh` | Submit a batch job. | | `srun ... --pty bash` | Start an **interactive** session on a compute node. | | `scancel ` | Cancel a queued or running job. | | `sacct -j ` | Show accounting info (state, memory, runtime) for past jobs. | | `seff ` | Friendly summary of resources used by a finished job. | ::: {.callout-note title="Key SLURM flags you will use constantly"} - `--account=` --- PIRG / account for charging (required). - `--partition=` --- queue (e.g. `compute`, `gpu`, `interactive`). - `--time=` or `--time=` --- wall-clock time limit. - `--mem=` --- total memory per node (e.g. `8G`, `100M`). - `--mem-per-cpu=` --- memory per core. - `--cpus-per-task=` --- cores for a multi-threaded task. - `--gpus=` --- request GPUs (paired with `--partition=gpu`). - `--constraint=gpu-10gb` / `gpu-40gb` / `gpu-80gb` --- GPU memory tier. ::: ------------------------------------------------------------------------ ## 6. Talapas partitions (queues) Every job must target a **partition**. Common ones at UO: | Partition | What it's for | Max time | |----|----|----| | `compute` | Standard CPU jobs. | 1 day | | `computelong` | Long-running CPU jobs. | 14 days | | `gpu` / `gpulong` | GPU jobs, short / long. | 1 / 14 d | | `memory` / `memorylong` | High-memory CPU jobs (up to 4 TB RAM). | 1 / 14 d | | `interactive` / `interactivegpu` | Short interactive sessions. | 12 / 8 h | | `preempt` | All nodes; your job may be **preempted** by its owner. | Variable | Run `sinfo` to see the current state of each partition on the cluster. ------------------------------------------------------------------------ ## 7. Interactive mode --- `srun` Use an interactive session when you want a shell **on a compute node** --- for testing scripts, exploring data, or running applications like RStudio or Python interactively. This is the mode you will use most often during the workshop. ::: {.callout-note title="Interactive vs. batch — which one?"} Use an **interactive** session (`srun` / `salloc`) when you'll **sit and watch** something run: testing or debugging a script, running a few commands by hand, exploring data, or running something very short. Use a **batch** job (`sbatch`, §8) for anything long-running, or anything you want to **start and walk away from** — a batch job keeps running after you log out. ::: ### 7.1 A minimum interactive session From a Talapas login node, run: ```{bash} #| label: M9-srun_account_mypirg #| eval: false srun --account= \ --partition=compute \ --time=1:00:00 \ --mem=4G \ --cpus-per-task=2 \ --pty bash ``` What each flag does: - `--account` --- your PIRG (required on Talapas; omit if you set `SLURM_ACCOUNT` in your profile). - `--partition=compute` --- standard CPU queue. - `--time=1:00:00` --- 1 hour; after this the session is killed. - `--mem=4G` --- 4 GB of RAM total on that node. - `--cpus-per-task=2` --- two CPU cores. - `--pty bash` --- give me a pseudo-terminal running bash. When the resources are available, your prompt will change to show the hostname of a compute node (e.g. `n0123`). You are now running on a compute node. When you're done, type `exit` to release the resources --- do not leave idle sessions running. **`salloc` — the close cousin of `srun --pty bash`.** Run `salloc` from a **login node** to request an interactive allocation; when it's granted, your shell **prompt changes from the login node to the compute node** (e.g. `n0135`) and you work there directly: ```{bash} #| label: M9-allocation_granted_prompt_changes #| eval: false salloc --account= --partition=compute --time=1:00:00 --mem=4G --cpus-per-task=2 # allocation granted → prompt changes, e.g. [duckid@n0135 ~]$ ``` `srun --pty bash` does the request *and* hands you a shell in one step; `salloc` is handy when you want a persistent allocation you can launch several `srun` steps into. Either way, `exit` releases the node. ### 7.2 Using the short `interactive` partition For short, quick sessions the `interactive` partition is usually faster to start: ```{bash} #| label: M9-srun_account_mypirg_partition #| eval: false srun --account= --partition=interactive --time=2:00:00 --mem=8G --pty bash ``` ### 7.3 Interactive GPU session ```{bash} #| label: M9-srun_account_mypirg_2 #| eval: false srun --account= \ --partition=interactivegpu \ --time=1:00:00 \ --nodes=1 --ntasks=1 --cpus-per-task=2 \ --mem=16G \ --gpus=1 --constraint=gpu-10gb \ --pty bash ``` `--constraint=gpu-10gb` (or `gpu-40gb` / `gpu-80gb`) pins the job to a GPU of that memory tier. ### 7.4 Graphical interactive jobs If you need a GUI (e.g. RStudio, MATLAB), two options: 1. **X11 forwarding** --- SSH in with `ssh -Y` and then: ```{bash} #| eval: false srun --account= --x11 --partition=compute \ --time=2:00:00 --mem=4G --pty bash ``` You need an X server running on your laptop (XQuartz on macOS, VcXsrv on Windows). 2. **Open OnDemand** (recommended and simpler) --- open in Chrome or Firefox, sign in with your Duck ID, and launch an RStudio, Jupyter, or full desktop session from the web portal. No X server needed. ::: {.callout-warning title="Don't hoard nodes"} Interactive jobs reserve their resources even when you are not actively typing. As soon as you are done, `exit` the shell so other users can use the node. ::: ------------------------------------------------------------------------ ### 7.5 Hands-on: run `hello_world.R` on an interactive node The Talapas scripts folder ships a tiny demo, [`hello_world.R`](scripts/hello_world.R) — open it in VS Code to read it (note the `#!/usr/bin/env Rscript` shebang on the first line). It uses only base R, prints which compute node it ran on, and writes a PDF plot and a summary CSV. From an interactive session (above), load R and run it: ```{bash} #| label: M9-module_load_r #| eval: false module load R Rscript hello_world.R ``` It reports the node it ran on and writes two files **in the directory you ran it from**: ``` hello_world_plot.pdf hello_world_summary.csv ``` ::: {.callout-warning title="Be in the script's directory — or give the full path"} `Rscript hello_world.R` only finds the script if you are **in the same directory** as it. Otherwise supply the path, e.g. `Rscript /projects///scrnaseq/scripts/hello_world.R`. The same is true for `sbatch