21  Parallel Computing in R

NoteLearning Objectives

After completing this chapter, you will be able to:

  • Understand when parallel computing provides performance benefits
  • Use the future ecosystem for parallel programming in R
  • Apply future_lapply() and future_map() for parallel iteration
  • Choose between forking and socket-based parallelization
  • Understand the differences between explicit and implicit parallelization
  • Avoid common pitfalls like resource competition
  • Debug functions and handle errors in parallel contexts
  • Cache results to avoid recomputation

21.1 Why Parallel Computing?

Modern computers have multiple processor cores, but by default R only uses one. This means that when you run a computationally intensive analysis — like bootstrapping, permutation tests, or processing thousands of samples — most of your computer’s power sits idle.

Parallel computing lets you use all available cores simultaneously, potentially reducing computation time by a factor equal to your core count. A task that takes 8 hours on one core might take just 1 hour across 8 cores.

In biosciences, parallel computing is especially valuable for:

  • Bootstrapping for confidence intervals
  • Permutation tests for significance testing
  • Simulation studies (e.g., population genetics)
  • Processing many samples (e.g., analyzing hundreds of RNA-seq files)
  • Cross-validation in machine learning
  • Parameter sweeps for computational models

Tasks like these share a structure: the same computation is repeated many times, and no repetition depends on the result of any other. Such problems are called embarrassingly parallel, and they are the best kind of parallel problem to have—each bootstrap replicate, gene, simulation, or sample can be handed to a different core with no coordination among the workers and the results simply collected at the end. Nearly everything in this chapter targets exactly this situation. When iterations do depend on one another (each step of a Markov chain needs the previous step), parallelizing is far harder, and the techniques here do not directly apply.

21.2 How Many Cores Do You Have?

Before parallelizing, find out how many cores are available:

parallel::detectCores()
#> [1] 14

This number includes both physical cores and “logical” cores (from technologies like Intel’s hyperthreading). For CPU-intensive tasks, your effective speedup is typically limited by the number of physical cores.

21.3 A Motivating Example

Let’s start with a simple example to demonstrate the power of parallelization. We’ll create a deliberately slow function that simulates a computationally expensive operation:

# Simulate analyzing a biological sample (takes 2 seconds each)
analyze_sample = function(sample_id) {
  # Simulate processing time
  Sys.sleep(2)

  # Return some fake results
  tibble(
    sample_id = sample_id,
    gene_count = sample(5000:20000, 1),
    quality_score = runif(1, 0.8, 1.0)
  )
}

Serial Execution

First, let’s process 8 samples the traditional way:

library(tictoc)

tic()
serial_results = lapply(1:8, analyze_sample) %>% bind_rows()
toc()
#> 16.063 sec elapsed

As expected, processing 8 samples at 2 seconds each takes about 16 seconds.

Parallel Execution

Now let’s do the same thing in parallel using the future.apply package:

library(future.apply)
plan(multisession)  # Enable parallel processing

tic()
parallel_results = future_lapply(1:8, analyze_sample) %>% bind_rows()
toc()
#> 2.326 sec elapsed

The parallel version is dramatically faster! With 8 cores, we can process all 8 samples simultaneously, reducing the time from ~16 seconds to ~2 seconds.

Figure 21.1 shows where that speedup comes from.

Figure 21.1: Timeline of the motivating example: eight 2-second samples processed one after another on a single core take about 16 seconds, while the same eight samples spread across eight workers all finish after about 2 seconds.

Let’s verify the results are identical:

# Results have same structure (values differ due to random generation)
all.equal(names(serial_results), names(parallel_results))
#> [1] TRUE

21.4 The Future Ecosystem

The examples above use the future package ecosystem, which provides a unified, elegant approach to parallel programming in R. The key packages are listed in Table 21.1:

Table 21.1: The future ecosystem
Package Description
future Core package defining how code is evaluated
future.apply Parallel versions of apply functions
furrr Parallel versions of purrr functions
NoteBase R’s parallel package

Older code, and many package internals, use the parallel package that ships with R rather than future. The pattern is: start a set of worker processes, hand them work with a parallel lapply(), and shut them down:

library(parallel)
n_cores <- detectCores() - 1          # leave one core for the OS
cl <- makeCluster(n_cores)            # start the workers
results <- parLapply(cl, data_list, analysis_function)
stopCluster(cl)                       # always shut them down

future’s plan(multisession) does the same thing behind the scenes, but it also manages the workers for you, ships the variables and packages your function needs, and handles random seeds correctly. You should be able to read the parallel version when you meet it; write new code with future and furrr.

Setting Up Parallel Execution

Before running parallel code, you must set a “plan” that determines how futures are resolved:

library(future)

# Sequential (no parallelization - useful for debugging)
plan(sequential)

# Parallel using multiple R sessions (works on all systems)
plan(multisession)

# Parallel using forked processes (faster, but Unix/Mac only)
plan(multicore)

For most users, plan(multisession) is the safest choice as it works on all operating systems.

future_lapply: Parallel Apply

The future_lapply() function is a drop-in replacement for lapply() (Section 15.6.2):

# Serial version
# lapply(1:4, function(x) x^2)

# Parallel version - just add "future_"
future_lapply(1:4, function(x) x^2)
#> [[1]]
#> [1] 1
#> 
#> [[2]]
#> [1] 4
#> 
#> [[3]]
#> [1] 9
#> 
#> [[4]]
#> [1] 16

furrr: Parallel purrr

If you prefer purrr’s syntax (Section 15.6.3), furrr provides parallel versions of all map functions:

library(furrr)

# Serial: map_dbl(1:4, function(x) x^2)
# Parallel:
future_map_dbl(1:4, function(x) x^2)
#> [1]  1  4  9 16

# Returns a data frame directly
future_map_dfr(1:4, function(x) {
  tibble(value = x, squared = x^2)
})
value squared
1 1
2 4
3 9
4 16

21.5 A Real Example: Bootstrapping Gene Expression

Let’s apply parallel computing to a realistic bioinformatics task: bootstrapping to estimate confidence intervals for differential expression.

Setup

set.seed(1234)

# Simulate gene expression data for 1000 genes, 50 samples
n_genes = 1000
n_samples = 50

expression_data = tibble(
  gene_id = rep(paste0("gene_", 1:n_genes), each = n_samples),
  sample_id = rep(paste0("sample_", 1:n_samples), n_genes),
  group = rep(rep(c("control", "treatment"), each = n_samples/2), n_genes),
  expression = rnorm(n_genes * n_samples, mean = 10, sd = 2)
) %>%
  # Add a treatment effect to some genes
  mutate(
    expression = if_else(
      gene_id %in% paste0("gene_", 1:100) & group == "treatment",
      expression + rnorm(n(), 2, 0.5),  # Upregulated genes
      expression
    )
  )

head(expression_data)
gene_id sample_id group expression
gene_1 sample_1 control 7.585869
gene_1 sample_2 control 10.554858
gene_1 sample_3 control 12.168882
gene_1 sample_4 control 5.308605
gene_1 sample_5 control 10.858249
gene_1 sample_6 control 11.012112

Bootstrap Function

# Function to bootstrap fold change for one gene
bootstrap_fold_change = function(gene_data) {
  # Resample within each group
  boot_sample = gene_data %>%
    group_by(group) %>%
    slice_sample(prop = 1, replace = TRUE) %>%
    ungroup()

  # Calculate mean expression per group
  means = boot_sample %>%
    group_by(group) %>%
    summarise(mean_expr = mean(expression), .groups = "drop")

  # Calculate fold change (treatment / control)
  fc = means$mean_expr[means$group == "treatment"] /
       means$mean_expr[means$group == "control"]

  return(fc)
}

# Function to run many bootstrap iterations for one gene
run_bootstrap = function(gene_id, data, n_boot = 1000) {
  gene_data = data %>% filter(gene_id == !!gene_id)

  # Run bootstrap iterations
  boot_fcs = replicate(n_boot, bootstrap_fold_change(gene_data))

  # Return summary statistics
  tibble(
    gene_id = gene_id,
    mean_fc = mean(boot_fcs),
    lower_ci = quantile(boot_fcs, 0.025),
    upper_ci = quantile(boot_fcs, 0.975)
  )
}

Compare Serial vs Parallel

Let’s bootstrap confidence intervals for 20 genes:

genes_to_test = paste0("gene_", 1:20)

# Serial version
tic()
serial_boot = map_dfr(genes_to_test, run_bootstrap,
                      data = expression_data, n_boot = 500)
toc()
#> 19.321 sec elapsed

# Parallel version
tic()
parallel_boot = future_map_dfr(genes_to_test, run_bootstrap,
                                data = expression_data, n_boot = 500,
                                .options = furrr_options(seed = TRUE))
toc()
#> 3.206 sec elapsed

The parallel version should be substantially faster, with speedup proportional to your core count.

View Results

parallel_boot %>%
  mutate(
    significant = lower_ci > 1 | upper_ci < 1,
    direction = case_when(
      lower_ci > 1 ~ "upregulated",
      upper_ci < 1 ~ "downregulated",
      TRUE ~ "not significant"
    )
  ) %>%
  head(10)
gene_id mean_fc lower_ci upper_ci significant direction
gene_1 1.131773 1.024564 1.238272 TRUE upregulated
gene_2 1.172248 1.048010 1.295856 TRUE upregulated
gene_3 1.186548 1.088120 1.289473 TRUE upregulated
gene_4 1.183704 1.054100 1.320405 TRUE upregulated
gene_5 1.127931 1.012363 1.237380 TRUE upregulated
gene_6 1.131323 1.031547 1.240307 TRUE upregulated
gene_7 1.176178 1.069463 1.302288 TRUE upregulated
gene_8 1.178690 1.031968 1.313715 TRUE upregulated
gene_9 1.273898 1.166305 1.398772 TRUE upregulated
gene_10 1.265714 1.126358 1.420870 TRUE upregulated

21.6 Forking vs Sockets

There are two main approaches to parallelization:

Socket-based (multisession)

  • Creates new R sessions for each worker
  • Works on all operating systems including Windows
  • Requires copying data to each worker (slower startup)
  • Selected with plan(multisession)

Fork-based (multicore)

  • Clones the current R session
  • Works on Unix/Mac only (not Windows)
  • Shares memory with parent process (faster)
  • Selected with plan(multicore)
  • May cause issues inside an IDE console (RStudio in particular); run from the terminal for best results
# Safe choice for all systems
plan(multisession)

# Faster on Unix/Mac (run scripts from terminal)
plan(multicore)

# Specify number of workers
plan(multisession, workers = 4)
Warning

On Windows, plan(multicore) silently falls back to plan(sequential). Always use plan(multisession) for cross-platform code.

21.7 Handling Errors in Parallel Code

Errors in parallel code can be tricky to debug. The purrr package provides safely() and possibly() to handle errors gracefully:

# A function that sometimes fails
risky_analysis = function(x) {
  if (x == 5) stop("Error processing sample 5!")
  tibble(sample = x, result = x^2)
}

# Wrap in safely() to catch errors
safe_analysis = safely(risky_analysis, otherwise = NULL)

# Run in parallel - won't crash if one fails
results = future_map(1:10, safe_analysis)

# Extract successful results
successful = map(results, "result") %>%
  compact() %>%  # Remove NULLs

  bind_rows()

# See which failed
errors = map(results, "error") %>%
  compact()

cat("Successful:", nrow(successful), "\n")
#> Successful: 9
cat("Failed:", length(errors), "\n")
#> Failed: 1

For simpler error handling, use possibly():

# Returns NA for failed cases
robust_analysis = possibly(risky_analysis, otherwise = tibble(sample = NA, result = NA))

future_map_dfr(1:10, robust_analysis)
sample result
1 1
2 4
3 9
4 16
NA NA
6 36
7 49
8 64
9 81
10 100

21.8 Caching Results

For expensive computations, cache intermediate results so you don’t have to recompute them:

library(memoise)

# Create a slow function
slow_gc_analysis = function(sequence) {
  Sys.sleep(1)  # Simulate slow computation
  tibble(
    sequence = sequence,
    gc_content = str_count(toupper(sequence), "[GC]") / nchar(sequence)
  )
}

# Create memoised version
memo_gc_analysis = memoise(slow_gc_analysis)

# First call - slow
tic()
result1 = memo_gc_analysis("ATGCGATCGATCG")
toc()
#> 1.01 sec elapsed

# Second call with same input - instant (cached)
tic()
result2 = memo_gc_analysis("ATGCGATCGATCG")
toc()
#> 0.01 sec elapsed

Persistent Caching

To cache results across R sessions (useful for long analyses):

# Create a cache directory
cache_dir = "analysis_cache"
if (!dir.exists(cache_dir)) dir.create(cache_dir)

# Create memoised function with persistent cache
memo_gc_persistent = memoise(
  slow_gc_analysis,
  cache = cache_filesystem(cache_dir)
)

# Results will be saved to disk and available in future sessions

21.9 When to Use Parallel Computing

Parallel computing isn’t always faster. Consider:

Good Candidates for Parallelization

  • Independent iterations: Each iteration doesn’t depend on others
  • CPU-intensive tasks: Complex calculations, simulations
  • Many iterations: Overhead is amortized across many runs
  • Long-running operations: Minutes or hours of total computation

Poor Candidates

  • Fast operations: Overhead exceeds computation time
  • Few iterations: Not enough work to distribute
  • Memory-intensive: Each worker needs its own copy of data
  • I/O bound: Waiting for disk/network, not CPU

Amdahl’s Law

The maximum speedup from parallelization is limited by the sequential portion of your code. If 50% of your code must run sequentially, you can never achieve more than 2× speedup, regardless of core count (Figure 21.2).

Figure 21.2: Amdahl’s law: speedup as a function of core count for different parallel fractions

21.10 Best Practices

1. Start Sequential, Then Parallelize

Always develop and debug your code in sequential mode first:

# Debug with sequential execution
plan(sequential)

# Test with a small subset
test_results = map_dfr(1:3, my_analysis_function)

# Once working, switch to parallel
plan(multisession)
full_results = future_map_dfr(1:1000, my_analysis_function)

2. Set Random Seeds

For reproducible parallel simulations:

# future.apply
future_lapply(1:10, my_function, future.seed = 123)

# furrr
future_map(1:10, my_function, .options = furrr_options(seed = 123))

3. Monitor Progress

Use progress bars to track long-running parallel jobs:

library(progressr)

handlers("txtprogressbar")

with_progress({
  p = progressor(steps = 100)

  results = future_map(1:100, function(x) {
    p()  # Signal progress
    slow_function(x)
  })
})

4. Manage Memory

Each parallel worker needs memory for: - A copy of R - Any data passed to the function - Results being computed

If you run out of memory, reduce the number of workers:

# Use fewer workers for memory-intensive tasks
plan(multisession, workers = 2)

5. Know When to Leave Your Laptop

Everything in this chapter parallelizes within one machine: plan(multisession, workers = 8) cannot use more cores than your laptop has, and every worker shares its memory. When a job needs more cores, more memory, or more hours than that, the next step is a cluster (Chapter 20), where the SLURM scheduler (Yoo et al., 2003) hands your script a slice of a compute node. The R code needs almost no change, but two habits matter:

  • Ask future for the cores SLURM gave you, not the cores the node has. On a shared node parallel::detectCores() reports the whole machine (often 128 cores), so use the allocation instead (see Table 20.5): plan(multisession, workers = as.integer(Sys.getenv("SLURM_CPUS_PER_TASK", "1"))). Use multisession, not multicore, inside a batch job unless you have verified forking works with your packages.
  • Scale out with job arrays rather than up with more workers. Many independent samples are better submitted as a SLURM job array (Section 20.14.3), one R process per sample, each with its own memory limit and log, than as one giant future_map() on one node. To make that R process reproducible on any machine, package its dependencies with renv or run it inside an Apptainer container (Kurtzer et al., 2017), the cluster-safe counterpart of Docker (Merkel, 2014).

21.11 Summary

Parallel computing can dramatically speed up computationally intensive analyses:

  • The future ecosystem provides a simple, unified approach to parallelization
  • Use plan(multisession) for cross-platform compatibility
  • Replace lapply() with future_lapply() or map() with future_map()
  • Always develop in sequential mode first, then parallelize
  • Use safely() or possibly() to handle errors gracefully
  • Cache expensive results with memoise for persistent storage
  • Consider Amdahl’s law: not everything benefits from parallelization

Parallel programming in R has never been easier. With just a few lines of code, you can harness all your computer’s cores for faster, more efficient analyses.

Tip

For reference on R functions used in parallel computing workflows, see Appendix E.

Exercises

TipPractice Exercises

Exercise 1: Basic Parallelization

Write a function that simulates processing a biological sample (include a Sys.sleep(1) to simulate processing time). Use future_lapply() to process 10 samples in parallel and compare the time to serial execution.

Exercise 2: Bootstrap Analysis

Using the gene expression simulation from this chapter: 1. Write a function to bootstrap the mean expression of a single gene 2. Apply it in parallel to 50 genes 3. Calculate 95% confidence intervals for each gene

Exercise 3: Error Handling

Create a function that analyzes DNA sequences but fails if the sequence contains invalid characters. Use safely() to process a list of sequences where some are invalid, and report which ones failed.

Exercise 4: Parameter Sweep

Write a function that simulates bacterial growth with parameters for growth rate (k), carrying capacity (K), and initial population (n0). Use parallel computing to run simulations across a grid of parameter combinations.

Exercise 5: Caching

Create a memoised version of a “slow” function that calculates protein properties. Demonstrate that repeated calls with the same input are instant.

Exercise 6: Benchmarking

For a computationally intensive task of your choice: 1. Time the serial version 2. Time the parallel version with 2, 4, and 8 workers 3. Calculate the speedup factor for each 4. Does the speedup match what you’d expect from Amdahl’s law?

Additional Reading