5  Unix Fundamentals

NoteLearning Objectives

After completing this chapter, you will be able to:

  • Explain what Unix is and why it’s important for scientific computing
  • Access the shell on your operating system
  • Navigate the file system using command-line tools
  • Understand the anatomy of shell commands
  • Create, move, copy, and delete files and directories

Unix is the operating system of choice in data science and scientific computing. In this chapter, we introduce you to the Unix way of thinking using a practical example: how to keep a data analysis project organized. Along the way, you will learn the most commonly used commands, though we won’t cover every detail—Unix is vast, and mastery comes through practice.

We encourage you to keep learning, especially when you find yourself reaching for the mouse too often or performing repetitive tasks. In those moments, there is almost certainly a more efficient way to accomplish your goal using Unix. As you work through this book and beyond, consider exploring additional resources such as the interactive tutorials at Codecademy, edX, or Coursera. For reference material, Bite Size Linux and Bite Size Command Line by Julia Evans are particularly clear and accessible.

When searching for Unix resources, keep in mind that you may encounter terms like Linux, the shell, and the command line. These all refer to aspects of what we’re learning here: a series of commands and a philosophy that enables you to organize files and automate tasks without relying on graphical interfaces.

5.1 What Is Unix?

Unix is a family of operating systems originally developed at Bell Labs in 1969 and publicly released in 1973 (Kernighan & Pike, 1984). Its command-line tools remain the everyday working environment of computational biology (Buffalo, 2015). What began as a project for a single computer has become the foundation of modern computing infrastructure.

Today, Unix and its derivatives power:

  • Most web servers on the internet
  • Scientific computing clusters (including Talapas)
  • macOS (Apple’s desktop operating system)
  • Android phones (via the Linux kernel)
  • Embedded systems in everything from routers to cars

Linux is an open-source implementation of Unix principles that has become the dominant operating system for scientific computing. When we refer to “Unix commands” in this book, the same commands work on Linux, macOS, and Windows Subsystem for Linux (WSL).

Figure 5.1: A simplified Unix family tree. The original Bell Labs Unix (1969) gave rise to BSD—an ancestor of today’s macOS—and to the commercial System V line, while Linux (1991) reimplemented Unix’s design without inheriting its code. Whichever branch you sit on, the same commands work everywhere.
Tip

Learning Unix is an investment that pays dividends throughout your career. The commands you learn today will work on systems you encounter decades from now—Unix’s longevity is remarkable in the fast-changing world of technology.

The Unix Philosophy

The tools we use in Unix embody a design philosophy that emerged from Bell Labs in the 1970s (Kernighan & Pike, 1984):

Do One Thing And Do It Well

Each Unix command is designed to perform a single task excellently. By combining simple, well-designed tools, you can build powerful and complex workflows. This “minimalist and modular” approach means:

  • Tools are easier to learn (each does one thing)
  • Tools are easier to combine (standardized input/output)
  • Complex tasks become chains of simple operations

You’ll see this philosophy throughout Unix: cat just displays files, grep just searches for patterns, sort just sorts lines. The magic happens when you combine them, which is the subject of Chapter 7.

5.2 The Shell: Your Interface to Unix

Instead of clicking, dragging, and dropping to organize our files and folders, we will be typing Unix commands into the terminal. The way we do this is similar to how you might type commands into an R console, but instead of generating plots and statistical summaries, we will be organizing files on our system and building powerful data pipelines.

The shell is the program that interprets your commands and communicates with the operating system. Think of it as a translator between human-readable instructions and the computer’s internal operations. When you open a terminal, you’re interacting with the shell.

While graphical user interfaces (GUIs) like Finder on Mac or File Explorer on Windows are intuitive for casual use, the shell offers:

  • Speed: Type commands faster than clicking through menus
  • Power: Access to thousands of specialized tools
  • Automation: Script repetitive tasks (see Chapter 10)
  • Remote Access: Work on distant computers identically to local ones
  • Reproducibility: Document exactly what you did

Bash: The Default Shell

Bash (Bourne Again SHell) is the default shell on most Linux systems, including Ubuntu under WSL and the Talapas cluster, and it is the shell this course is written for. Modern macOS defaults to zsh instead. For everything we do in this course the two are nearly identical—the commands, pipes, and scripts you learn here work the same in either—so Mac users can simply use zsh. Other shells exist (fish, tcsh), but Bash is the most widely used and what you’ll encounter on computing clusters.

Accessing the Shell

How you access the shell depends on your operating system:

macOS
Open the Terminal application (Applications → Utilities → Terminal) or use a third-party terminal like iTerm2.
Linux
Open your distribution’s terminal application, often called “Terminal” or “Console.”
Windows
After installing WSL (see Section 1.6.3), search for “Ubuntu” or your installed Linux distribution in the Start menu.

When you open the terminal, you’ll see a prompt indicating the system is ready for your command:

username@computer:~$

The prompt typically shows your username, the computer name, your current directory (~ means your home directory), and a $ symbol indicating you’re a regular user (a # would indicate administrator/root access).

Once you have a terminal open, you can start typing commands. You should see a blinking cursor at the spot where what you type will appear. This position is called the command line. Once you type something and press Enter (or Return on Mac), Unix will try to execute the command. Try this simple example:

$ echo "hello world"
hello world

The echo command is similar to print() in R—it simply displays the text you provide. Notice that you can’t use the mouse to move around in the terminal the way you might in a word processor. You have to use the keyboard. To recall a command you previously typed, use the up arrow key. This is one of many conveniences that make the command line efficient once you learn its patterns.

TipCopying and pasting in the terminal

The shortcuts you know from other programs change here: Ctrl+C does not copy—it cancels the running command (Table 5.1). On macOS, Cmd+C and Cmd+V work as usual. In most Linux terminals, and in Ubuntu on Windows, use Ctrl+Shift+C and Ctrl+Shift+V, or right-click to paste. Two habits keep pasting safe: paste commands from the web one line at a time and read each before pressing Enter, and beware text that has passed through a word processor or slide deck—its typographic “smart quotes” look like quotes but are different characters the shell does not understand.

Note

Throughout this book, we show Unix commands with a $ prefix to indicate the prompt. You don’t type the $—it’s just showing where the command begins. Lines without $ represent output from commands.

Essential Keyboard Shortcuts

Learning these shortcuts (Table 5.1) will dramatically speed up your command-line work:

Table 5.1: Essential shell keyboard shortcuts
Shortcut Action
Tab Auto-complete commands, file names, and paths
↑ / ↓ Scroll through command history
Ctrl+A Move cursor to beginning of line
Ctrl+E Move cursor to end of line
Ctrl+U Delete everything before cursor
Ctrl+K Delete everything after cursor
Ctrl+W Delete word before cursor
Ctrl+R Search through command history
Ctrl+C Cancel current command
Ctrl+L or clear Clear the terminal screen

For a complete list of keyboard shortcuts, see Appendix A.

Tip

Tab completion is your best friend! Start typing a file name and press Tab—the shell will complete it for you. If there are multiple matches, press Tab twice to see all options. This prevents typos and saves enormous amounts of typing.

5.3 Anatomy of a Shell Command

Every shell command follows a consistent structure (Figure 5.2):

command -options arguments
Figure 5.2: The parts of a shell command. The prompt is printed by the shell to show it is ready; you type the command, any options that modify its behavior, and the arguments it should operate on.

Let’s break this down:

Prompt
Printed by the shell, not typed by you. It signals that the computer is ready to accept a new command ($ for a regular user, # for the administrator).
Command
The program or built-in function you want to run. This is required.
Options (also called flags)
Modify the command’s behavior. Usually preceded by - (single letter) or -- (full word). Optional.
Arguments
What the command should operate on—typically file or directory names. Sometimes optional.

For example:

ls -l Documents

Here, ls is the command (list directory contents), -l is an option (use long format with details), and Documents is the argument (which directory to list).

Note

Options can often be combined. For example, ls -l -h -a can be written as ls -lha. The order of combined options usually doesn’t matter.

Getting Help: The man Command

Every Unix command has a manual page accessible via the man command:

$ man ls

This opens the manual for ls, showing all available options and usage examples. Navigation within man:

  • Space or f — Move forward one page
  • b — Move back one page
  • /pattern — Search for “pattern” (press n for next match)
  • h — Show help for navigating the manual
  • q — Quit and return to the shell

For a quicker summary, many commands support --help:

$ ls --help
Tip

When you encounter an unfamiliar command, man should be your first stop. The manuals are comprehensive and authoritative—they’re written by the people who created the commands.

5.4 Understanding the File System

We refer to all the files, folders, and programs on your computer as the filesystem. In Unix, everything is organized as files within a hierarchical directory structure (Figure 5.3). Keep in mind that folders and programs are technically also files, but this is a detail we rarely think about. Understanding this structure is essential for navigation.

Figure 5.3: A computer’s storage is a system of directories (folders) and files. The same folders you click through in Finder or File Explorer are what you will navigate from the shell.

The Directory Tree

The first concept you need to grasp to become a Unix user is how your filesystem is organized. You should think of it as a series of nested folders, each containing files, folders, and executable programs.

In Unix, we refer to folders as directories. Directories that are inside other directories are called subdirectories. The entire file system branches from a single root directory, represented by /:

/
├── Users/
│   └── wcresko/
│       ├── Documents/
│       ├── Downloads/
│       └── projects/
├── Applications/
├── Library/
└── tmp/

Figure 5.4 shows the same idea as a tree: the root directory / at the top, system directories and user home directories branching below it, and your own files nested deeper still.

Figure 5.4: The Unix file hierarchy. Every file and directory on the system descends from the root directory /; user home directories sit under /Users (macOS) or /home (Linux).

Key directories include:

/
The root directory. Everything else is inside this.
/home or /Users
Contains user home directories. Your files live here.
~
Shorthand for your home directory (e.g., /home/wcresko).
/tmp
Temporary files that may be deleted on reboot.

The home directory is where all your personal files are kept, as opposed to system files that come with your computer. Your home directory is typically named after your username. For example, if your username is wcresko, your home directory might be /home/wcresko on Linux or /Users/wcresko on macOS.

To understand the hierarchical nature of the filesystem, imagine you want to delete a file called cv.tex that lives in a resumes folder, which is inside a docs folder, which is inside your home directory. In a graphical file manager, you would double-click docs, then resumes, then drag cv.tex to the trash. This sequence of clicks mirrors the hierarchical path: cv.tex is inside resumes, which is inside docs, which is inside your home directory.

In Unix, we express this same relationship as a path: ~/docs/resumes/cv.tex. The forward slashes separate each level of the directory tree, and ~ is shorthand for your home directory.

NoteNote for Windows Users

When using Windows Subsystem for Linux (WSL), your Unix home directory is different from your Windows home directory. The Unix home will be something like /home/yourusername, while your Windows Documents folder is typically accessible at /mnt/c/Users/yourusername/Documents. Keep this distinction in mind when organizing files.

Absolute vs. Relative Paths

A path specifies the location of a file or directory. The name comes from the fact that the path string spells out the route you need to follow to get to a location from some starting point. Paths come in two forms:

Absolute Paths
Start from the root directory (/). They work from anywhere because they specify the complete location from the root of the filesystem. An absolute path always begins with / or ~.
/Users/wcresko/Documents/analysis.R
~/Documents/analysis.R
Relative Paths
Start from your current directory. Shorter but depend on where you are. If a path does not start with / or ~, Unix assumes it is relative to your current working directory. Figure 5.5 shows how an absolute path is built up from the root of the tree.
Documents/analysis.R
Figure 5.5: Understanding file system paths. An absolute path is built by following the directory tree from the root (/) down through each level to the file; the highlighted route spells out the path one directory at a time.

Unix provides shorthand symbols for common directory references:

Table 5.2: Special directory symbols
Symbol Meaning
. Current directory (where you are now)
.. Parent directory (one level up)
~ Your home directory
- Previous directory (where you were before)

These symbols can be combined in paths. For example, ../.. means “go up two directories,” and ~/projects means “the projects folder in my home directory.”

The Special Entries . and ..

The first two symbols in Table 5.2 are not just shorthand: they are real entries that exist in every directory. You can see them with ls -a, which shows hidden files (anything whose name begins with a dot):

$ ls -a
.  ..  .bashrc  analysis.R  data  README.md

The entry . (dot) refers to the directory itself, and .. (dot dot) refers to its parent (Figure 5.6). This is why cd .. moves you up one level: you are literally entering the .. entry of your current directory (Figure 5.7). It is also why you run a script in the current directory as ./script.sh, meaning “the file script.sh inside the directory I am in right now.”

Figure 5.6: Every directory contains two hidden entries: ., which points to the directory itself, and .., which points to its parent.
Figure 5.7: Navigating with ..: cd .. follows the parent link up one level in the tree, and a path such as ../other_dir goes up one level and then down into a sibling directory.
# These three commands are equivalent
$ cd data
$ cd ./data
$ cd ././data

# Up one level, then into a sibling directory
$ cd ../scripts
Tip

Use Tab completion! Start typing a file or directory name and press Tab to autocomplete. This saves typing and prevents errors.

5.5 Essential Navigation Commands

In a point-and-click system, you always know where you are—the window shows the folder contents and the title bar displays the location. In the terminal, we don’t have the same visual cues, but the concept of a current location is equally important. We call this the working directory, and every terminal window has one.

Where Am I? pwd

The pwd command (print working directory) shows your current location:

$ pwd
/Users/wcresko/projects/analysis

When you first open a terminal, it typically starts in your home directory. The output of pwd will vary depending on your system—you might see /Users/yourusername on macOS or /home/yourusername on Linux, including Ubuntu running under WSL on Windows.

Always know where you are before running commands—especially destructive ones like rm!

What’s Here? ls

In a graphical file manager, you know what’s in a directory because you see icons representing files and folders. In the terminal, we use the ls command (list) to see directory contents:

$ ls
data/  results/  analysis.R  README.md

By default, ls provides a compact listing. Common options make it more informative:

Table 5.3: Common options for the ls command
Option Description
-l Long format (permissions, size, date)
-a Show hidden files (those starting with .)
-h Human-readable file sizes (KB, MB, GB)
-t Sort by modification time (newest first)
-S Sort by file size (largest first)
-r Reverse sort order
-F Add symbols to show file types (/ for directories, * for executables)

Example with multiple options:

$ ls -lah
total 32K
drwxr-xr-x  4 wcresko staff 128B Nov 15 14:30 .
drwxr-xr-x 12 wcresko staff 384B Nov 14 09:15 ..
-rw-r--r--  1 wcresko staff 2.1K Nov 15 14:30 analysis.R
drwxr-xr-x  3 wcresko staff  96B Nov 15 10:00 data
-rw-r--r--  1 wcresko staff 512B Nov 14 09:15 README.md
drwxr-xr-x  2 wcresko staff  64B Nov 15 14:00 results

The long format shows: permissions, links, owner, group, size, modification date, and name.

Moving Around: cd

The cd command (change directory) moves you to a different location. When you use cd, you change your working directory just as you would double-click a folder in a graphical file manager.

# Go to a specific directory (relative path)
$ cd Documents

# Go up one directory (to parent)
$ cd ..

# Go to home directory (three equivalent ways)
$ cd ~
$ cd $HOME
$ cd

# Go to the previous directory (like "back" in a browser)
$ cd -

# Go to an absolute path
$ cd /Users/wcresko/projects
TipPro Tip: Tab Completion

In Unix, you can auto-complete paths by pressing Tab. This is essential for efficient command-line work. Type cd Do and press Tab—if Documents is the only directory starting with “Do”, Unix will complete it for you. If there are multiple matches, press Tab twice to see all options. Using Unix without Tab completion is painfully slow; make it a habit from the start.

When using cd, you can type either a full path (starting with / or ~) or a relative path. If the path you type doesn’t start with / or ~, Unix assumes you’re typing a relative path and looks for the directory within your current working directory. So if you’re in your home directory and type cd projects, Unix looks for a projects subdirectory inside your home. But if you type cd Users, you’ll get an error—there’s no Users directory inside your home directory.

Caution

Unlike GUIs, cd gives no feedback when successful. No news is good news! If you don’t see an error message, the command worked. Use pwd to verify where you ended up.

You can chain directory moves with forward slashes. To navigate to a nested directory in one command:

# Move through multiple directories
$ cd projects/analysis/data

# Move up two levels
$ cd ../..

# Move up one level and into a sibling directory
$ cd ../other_project

Two more useful cd tricks: typing cd - returns you to whatever directory you were in before your last cd command, which is handy if you typed a long path and want to go back. And typing just cd with no arguments always returns you to your home directory.

NotePaths with spaces need quotes

The shell separates arguments with spaces, so cd University of Oregon Dropbox hands cd three separate arguments and fails with “too many arguments” or “No such file or directory.” Either wrap the path in quotes—cd "University of Oregon Dropbox"—or escape each space with a backslash: cd University\ of\ Oregon\ Dropbox. Tab completion inserts the backslashes for you, one more reason to lean on it. You will meet such paths immediately, because OneDrive and Dropbox folders have spaces in their names (Chapter 3); this is also why Section 5.7 asks you never to put spaces in names you create yourself.

Navigation Examples

Let’s work through some examples to solidify these concepts. Consider this directory structure:

~
├── docs/
│   ├── reports/
│   └── resumes/
│       └── cv.tex
└── projects/
    ├── project-1/
    │   ├── data/
    │   └── figs/
    └── project-2/
        └── analysis.R

Example 1: Starting from ~/projects, navigate to figs in project-1:

# Using a relative path
$ cd project-1/figs

# Verify we're in the right place
$ pwd
/home/wcresko/projects/project-1/figs

Example 2: From ~/projects, navigate to reports in docs:

# Using a relative path (go up, then into docs)
$ cd ../docs/reports

# Or using an absolute path
$ cd ~/docs/reports

Example 3: From ~/projects/project-1/figs, navigate to project-2:

# Go up two levels, then into project-2
$ cd ../../project-2

# Or use the full path
$ cd ~/projects/project-2

Notice how relative paths require you to think about where you are, while absolute paths work from anywhere but are longer to type. Both approaches are useful—Tab completion makes either one efficient.

5.6 Working with Files and Directories

Now that you can navigate, let’s learn to manipulate files and directories.

Creating Directories: mkdir

# Create a single directory
$ mkdir analysis

# Create nested directories (use -p for parents)
$ mkdir -p project/data/raw

# Create multiple directories at once
$ mkdir results figures tables

Creating Files: touch and nano

The touch command creates an empty file (or updates the timestamp of an existing file):

$ touch notes.txt

For files with content, use a text editor. nano is a simple terminal-based editor:

$ nano notes.txt

In nano:

  • Type your content normally
  • Ctrl+O saves the file (press Enter to confirm)
  • Ctrl+X exits nano
  • Ctrl+K cuts the current line; Ctrl+U pastes it back
  • Ctrl+W searches (“where is”)

The help bar along the bottom of nano’s screen writes ^ for Ctrl, so the ^O it shows is the Ctrl+O above; the full set of nano shortcuts is in Section A.4. It is worth getting comfortable with nano even if you prefer editing files in your IDE, because it is the editor you will have on Talapas (Chapter 20), where there is no point-and-click and a quick fix to a job script happens in the terminal.

Copying Files: cp

# Copy a file
$ cp original.txt backup.txt

# Copy a file to a directory
$ cp analysis.R scripts/

# Copy a directory (requires -r for recursive)
$ cp -r data/ data_backup/

# Copy multiple files to a directory
$ cp *.txt documents/

Moving and Renaming: mv

The mv command both moves and renames files:

# Rename a file
$ mv oldname.txt newname.txt

# Move a file to a directory
$ mv analysis.R scripts/

# Move and rename simultaneously
$ mv data/raw.csv backup/raw_2024.csv

# Move multiple files
$ mv *.R scripts/
CautionThe shell trusts you

mv and cp will silently overwrite an existing file if you give two things the same name, and rm will delete whatever you tell it to. There is no “Are you sure?” dialog. Add -i (mv -i, cp -i, rm -i) to be asked before anything is overwritten or removed.

Deleting Files: rm and rmdir

WarningDanger Zone

The shell has no trash can! Deleted files are gone forever. Always double-check before using rm, especially with wildcards.

# Remove a file
$ rm unnecessary.txt

# Remove multiple files
$ rm *.tmp

# Remove a directory and all contents (DANGEROUS!)
$ rm -r old_project/

# Interactive mode (asks for confirmation)
$ rm -i important.txt

# Remove empty directories only
$ rmdir empty_folder/

The -r (recursive) option with rm is powerful but dangerous. Consider using -i (interactive) to confirm each deletion when learning.

Viewing Files: cat, head, tail, less

Four commands let you look inside a file without opening an editor: cat prints the whole file, head and tail show its first or last lines, and less pages through it a screen at a time (press q to quit). For now, that one-line summary is enough to keep moving:

$ cat notes.txt          # the whole file
$ head -n 5 notes.txt    # the first 5 lines
$ tail -n 5 notes.txt    # the last 5 lines
$ less big_file.txt      # page through; q quits, /text searches

Each of these tools repays closer study, and Section 7.3 gives them a full treatment—including how to choose the right one for files too large to open at all.

File Information: wc

The wc (word count) command provides file statistics:

$ wc document.txt
  100   850  5420 document.txt

This shows: lines, words, and bytes. Useful options:

$ wc -l data.csv   # Lines only
$ wc -w essay.txt  # Words only
$ wc -c file.bin   # Bytes only

Finding Files: find

The find command searches for files and directories based on various criteria:

# Find by name (case-insensitive with -iname)
$ find . -iname "*.csv"

# Find files modified in the last 7 days
$ find ~/projects -mtime -7

# Find files larger than 100MB
$ find . -size +100M

# Find and do something with results
$ find . -name "*.tmp" -delete

The find command is powerful for locating files across complex directory structures. Common options:

Table 5.4: Common find options
Option Description
-name Match filename (case-sensitive)
-iname Match filename (case-insensitive)
-type f Find files only
-type d Find directories only
-size +10M Larger than 10 megabytes
-mtime -7 Modified within 7 days

5.7 Naming Conventions

Before you start organizing projects with Unix, you should adopt a naming convention that you will use systematically for your files and directories. Good naming practices prevent countless problems and help you find files quickly, even months or years after creating them.

The Smithsonian Data Management Best Practices outlines five principles of file naming and organization:

  1. Have a distinctive, human-readable name that gives an indication of the content
  2. Follow a consistent pattern that is machine-friendly
  3. Organize files into directories that follow a consistent pattern
  4. Avoid repetition of semantic elements among file and directory names
  5. Have a file extension that matches the file format (never change extensions!)

For specific coding conventions, we recommend following the Tidyverse Style Guide, which provides consistent patterns for naming files and variables in R projects.

ImportantFile Naming Rules

DO:

  • Use lowercase letters, numbers, underscores (_), and hyphens (-)
  • Use meaningful, descriptive names that indicate content
  • Include dates in ISO format when relevant: data_2024-01-15.csv
  • Use appropriate file extensions that match the format

DON’T:

  • Use spaces (use _ or - instead)
  • Start names with - (confused with command options)
  • Use special characters: * ? [ ] ' " \ / < > |
  • Use extremely long names

Examples:

Table 5.5: File naming examples
Bad Good Why
my data.csv my_data.csv Spaces cause problems in shell commands
-results.txt results.txt Leading dash is confused with options
data (copy).csv data_backup.csv Parentheses require escaping
Joe's file.txt joes_file.txt Apostrophe causes quoting issues
FINAL_v2_FINAL.csv analysis_2024-01-15.csv Date-based versioning is clearer
NoteCase matters—except when it doesn’t

Unix filenames are case-sensitive: Data.csv, data.csv, and DATA.csv are three different files on Linux, including Talapas. macOS is the confusing exception—its default file system preserves case but treats names that differ only in case as the same file—so a script that runs happily on your Mac, opening stickle_rnaseq.tsv when the file is really Stickle_RNAseq.tsv, will break the moment it runs on the cluster. Treat every filename as case-sensitive, spell names exactly as they are, and prefer all-lowercase names for files you create.

5.8 Getting Help

Manual Pages

Every Unix command has a manual page accessible via man:

$ man ls

Manual pages can be dense. Look for:

  • NAME: What the command does
  • SYNOPSIS: How to use it
  • DESCRIPTION: Detailed explanation
  • OPTIONS: Available flags
  • EXAMPLES: If you’re lucky!

Press q to exit the manual.

Quick Help

Many commands support a --help option:

$ ls --help

Searching for Commands

If you know what you want to do but not the command:

# Search man page descriptions
$ man -k "search term"
$ apropos "search term"

# Find where a command is located
$ which python

5.9 Preparing for a Data Science Project

Now let’s put these commands into practice by setting up a proper directory structure for a data science project. A well-organized project directory makes your work reproducible and easier to navigate.

We recommend creating a dedicated directory to hold all your projects. This keeps your work organized and separate from other files in your home directory:

$ cd ~
$ mkdir projects
$ cd projects

For this example, let’s create a project analyzing gun violence data (based on a real project on GitHub). We’ll create subdirectories for raw data and processed data:

$ mkdir murders
$ cd murders
$ mkdir data rdas
$ ls
data  rdas
$ pwd
/home/wcresko/projects/murders

Our project structure now looks like this:

~/projects/
└── murders/
    ├── data/      # Raw data files
    └── rdas/      # Processed R data files
TipProject Organization Tips
  • Create a projects directory in your home folder to contain all your work
  • Give each project its own directory with a descriptive name
  • Use consistent subdirectory names across projects (e.g., data, results, scripts, figures)
  • Keep raw data separate from processed data
  • Consider adding a README.md file to describe the project

From any terminal, you can navigate to this project by typing:

$ cd ~/projects/murders

As your projects grow, you might expand this structure:

~/projects/
└── murders/
    ├── data/           # Raw, unmodified data
    ├── rdas/           # Processed R data objects
    ├── scripts/        # Analysis scripts
    ├── results/        # Analysis output
    ├── figures/        # Generated plots
    └── README.md       # Project description

This organization keeps related files together, makes your analysis reproducible, and helps collaborators (including your future self) understand the project structure.

The Course Project Skeleton

The murders example above grew its structure as it needed it. For your own work, it is worth adopting one standard skeleton from the start and reusing it for every project (Noble, 2009; Wilson et al., 2017). The one used throughout this course separates data, code, results, and writing:

my_project/
├── data/
│   ├── raw/          # original data, exactly as it arrived - never modify
│   └── processed/    # cleaned or intermediate data, made by scripts
├── scripts/          # .sh, .R, .py, and .qmd files
├── output/
│   ├── figures/      # plots made by your scripts
│   └── tables/       # result tables made by your scripts
└── docs/             # notes, reports, manuscript drafts

Build it with a few mkdir -p calls:

$ mkdir -p my_project/data/raw my_project/data/processed
$ mkdir -p my_project/scripts my_project/output/figures my_project/output/tables
$ mkdir -p my_project/docs

(Section 6.1.1 introduces brace expansion, which collapses all of this into a single command, and the course’s make_project.sh script on the Example Files page builds the same skeleton complete with template files.)

Two rules give the skeleton its power:

  • data/raw/ is read-only. Scripts read from it; nothing ever writes to it. Once you can change permissions (Section 6.2.2), enforce the rule with chmod a-w data/raw/*.
  • output/ is expendable. Everything in it can be regenerated by running the scripts on the raw data, so deleting it costs nothing. If deleting it would cost something, a result is being made by hand that should be made by a script.
Figure 5.8: The course project skeleton with its two rules made visible: data/raw/ is locked read-only so nothing can silently overwrite the original data, while everything in output/ is regenerated by the scripts and is therefore safe to delete.

Two habits complete the picture. Use relative paths inside your scripts (data/raw/samples.csv, never /Users/you/my_project/data/raw/samples.csv), so the whole project can move to another machine—or to Talapas—and still run. And number your scripts in the order they run (01_clean.R, 02_analyze.R, 03_plot.R), so that a directory listing reads as the pipeline’s documentation.

5.10 Summary

This chapter covered the fundamentals of Unix navigation and file manipulation:

  • The shell interprets commands and interfaces with the operating system
  • Commands follow the pattern: command -options arguments
  • The file system is a hierarchical tree starting at /
  • Navigation uses pwd, ls, and cd
  • File operations include cp, mv, rm, mkdir, and touch
  • Every directory contains . (itself) and .. (its parent); cd .. moves up one level
  • Good naming conventions prevent problems
  • Manual pages (man) provide detailed help

These commands form the foundation for everything else in Unix. Practice them until they become automatic!

The second half of the shell story—wild cards, file permissions, and configuring your shell environment—continues in Chapter 6.

Tip

For a comprehensive reference of Unix commands with syntax and examples, see Appendix B.

Additional Reading

To further develop your Unix skills:

For macOS-specific Unix guidance:

Exercises

TipPractice Exercises

Exercise 1: Navigation Practice

Without using a GUI file browser:

  1. Open your terminal
  2. Print your current directory
  3. Navigate to your home directory
  4. List all files including hidden ones
  5. Navigate to your Documents folder (or create it if needed)
  6. Return to your home directory using the shortcut

Exercise 2: File Operations

  1. Create a new directory called practice
  2. Navigate into it
  3. Create an empty file called notes.txt
  4. Create a subdirectory called backup
  5. Copy notes.txt to the backup directory
  6. Rename the original notes.txt to my_notes.txt
  7. List the contents of both directories to verify your work
  8. Clean up by removing the entire practice directory

Exercise 3: Exploration

Use manual pages to answer:

  1. What does the ls -R option do?
  2. How can you create a directory and all necessary parent directories in one command?
  3. What command shows disk usage, and what option makes the output human-readable?