5 Unix Fundamentals
After completing this chapter, you will be able to:
- Explain what Unix is and why it’s important for scientific computing
- Access the shell on your operating system
- Navigate the file system using command-line tools
- Understand the anatomy of shell commands
- Create, move, copy, and delete files and directories
Unix is the operating system of choice in data science and scientific computing. In this chapter, we introduce you to the Unix way of thinking using a practical example: how to keep a data analysis project organized. Along the way, you will learn the most commonly used commands, though we won’t cover every detail—Unix is vast, and mastery comes through practice.
We encourage you to keep learning, especially when you find yourself reaching for the mouse too often or performing repetitive tasks. In those moments, there is almost certainly a more efficient way to accomplish your goal using Unix. As you work through this book and beyond, consider exploring additional resources such as the interactive tutorials at Codecademy, edX, or Coursera. For reference material, Bite Size Linux and Bite Size Command Line by Julia Evans are particularly clear and accessible.
When searching for Unix resources, keep in mind that you may encounter terms like Linux, the shell, and the command line. These all refer to aspects of what we’re learning here: a series of commands and a philosophy that enables you to organize files and automate tasks without relying on graphical interfaces.
5.1 What Is Unix?
Unix is a family of operating systems originally developed at Bell Labs in 1969 and publicly released in 1973 (Kernighan & Pike, 1984). Its command-line tools remain the everyday working environment of computational biology (Buffalo, 2015). What began as a project for a single computer has become the foundation of modern computing infrastructure.
Today, Unix and its derivatives power:
- Most web servers on the internet
- Scientific computing clusters (including Talapas)
- macOS (Apple’s desktop operating system)
- Android phones (via the Linux kernel)
- Embedded systems in everything from routers to cars
Linux is an open-source implementation of Unix principles that has become the dominant operating system for scientific computing. When we refer to “Unix commands” in this book, the same commands work on Linux, macOS, and Windows Subsystem for Linux (WSL).
Learning Unix is an investment that pays dividends throughout your career. The commands you learn today will work on systems you encounter decades from now—Unix’s longevity is remarkable in the fast-changing world of technology.
The Unix Philosophy
The tools we use in Unix embody a design philosophy that emerged from Bell Labs in the 1970s (Kernighan & Pike, 1984):
Do One Thing And Do It Well
Each Unix command is designed to perform a single task excellently. By combining simple, well-designed tools, you can build powerful and complex workflows. This “minimalist and modular” approach means:
- Tools are easier to learn (each does one thing)
- Tools are easier to combine (standardized input/output)
- Complex tasks become chains of simple operations
You’ll see this philosophy throughout Unix: cat just displays files, grep just searches for patterns, sort just sorts lines. The magic happens when you combine them, which is the subject of Chapter 7.
5.2 The Shell: Your Interface to Unix
Instead of clicking, dragging, and dropping to organize our files and folders, we will be typing Unix commands into the terminal. The way we do this is similar to how you might type commands into an R console, but instead of generating plots and statistical summaries, we will be organizing files on our system and building powerful data pipelines.
The shell is the program that interprets your commands and communicates with the operating system. Think of it as a translator between human-readable instructions and the computer’s internal operations. When you open a terminal, you’re interacting with the shell.
While graphical user interfaces (GUIs) like Finder on Mac or File Explorer on Windows are intuitive for casual use, the shell offers:
- Speed: Type commands faster than clicking through menus
- Power: Access to thousands of specialized tools
- Automation: Script repetitive tasks (see Chapter 10)
- Remote Access: Work on distant computers identically to local ones
- Reproducibility: Document exactly what you did
Bash: The Default Shell
Bash (Bourne Again SHell) is the default shell on most Linux systems, including Ubuntu under WSL and the Talapas cluster, and it is the shell this course is written for. Modern macOS defaults to zsh instead. For everything we do in this course the two are nearly identical—the commands, pipes, and scripts you learn here work the same in either—so Mac users can simply use zsh. Other shells exist (fish, tcsh), but Bash is the most widely used and what you’ll encounter on computing clusters.
Accessing the Shell
How you access the shell depends on your operating system:
- macOS
- Open the Terminal application (Applications → Utilities → Terminal) or use a third-party terminal like iTerm2.
- Linux
- Open your distribution’s terminal application, often called “Terminal” or “Console.”
- Windows
- After installing WSL (see Section 1.6.3), search for “Ubuntu” or your installed Linux distribution in the Start menu.
When you open the terminal, you’ll see a prompt indicating the system is ready for your command:
The prompt typically shows your username, the computer name, your current directory (~ means your home directory), and a $ symbol indicating you’re a regular user (a # would indicate administrator/root access).
Once you have a terminal open, you can start typing commands. You should see a blinking cursor at the spot where what you type will appear. This position is called the command line. Once you type something and press Enter (or Return on Mac), Unix will try to execute the command. Try this simple example:
The echo command is similar to print() in R—it simply displays the text you provide. Notice that you can’t use the mouse to move around in the terminal the way you might in a word processor. You have to use the keyboard. To recall a command you previously typed, use the up arrow key. This is one of many conveniences that make the command line efficient once you learn its patterns.
The shortcuts you know from other programs change here: Ctrl+C does not copy—it cancels the running command (Table 5.1). On macOS, Cmd+C and Cmd+V work as usual. In most Linux terminals, and in Ubuntu on Windows, use Ctrl+Shift+C and Ctrl+Shift+V, or right-click to paste. Two habits keep pasting safe: paste commands from the web one line at a time and read each before pressing Enter, and beware text that has passed through a word processor or slide deck—its typographic “smart quotes” look like quotes but are different characters the shell does not understand.
Throughout this book, we show Unix commands with a $ prefix to indicate the prompt. You don’t type the $—it’s just showing where the command begins. Lines without $ represent output from commands.
Essential Keyboard Shortcuts
Learning these shortcuts (Table 5.1) will dramatically speed up your command-line work:
| Shortcut | Action |
|---|---|
Tab |
Auto-complete commands, file names, and paths |
↑ / ↓
|
Scroll through command history |
Ctrl+A |
Move cursor to beginning of line |
Ctrl+E |
Move cursor to end of line |
Ctrl+U |
Delete everything before cursor |
Ctrl+K |
Delete everything after cursor |
Ctrl+W |
Delete word before cursor |
Ctrl+R |
Search through command history |
Ctrl+C |
Cancel current command |
Ctrl+L or clear
|
Clear the terminal screen |
For a complete list of keyboard shortcuts, see Appendix A.
Tab completion is your best friend! Start typing a file name and press Tab—the shell will complete it for you. If there are multiple matches, press Tab twice to see all options. This prevents typos and saves enormous amounts of typing.
5.3 Anatomy of a Shell Command
Every shell command follows a consistent structure (Figure 5.2):
Let’s break this down:
- Prompt
-
Printed by the shell, not typed by you. It signals that the computer is ready to accept a new command (
$for a regular user,#for the administrator). - Command
- The program or built-in function you want to run. This is required.
- Options (also called flags)
-
Modify the command’s behavior. Usually preceded by
-(single letter) or--(full word). Optional. - Arguments
- What the command should operate on—typically file or directory names. Sometimes optional.
For example:
Here, ls is the command (list directory contents), -l is an option (use long format with details), and Documents is the argument (which directory to list).
Options can often be combined. For example, ls -l -h -a can be written as ls -lha. The order of combined options usually doesn’t matter.
Getting Help: The man Command
Every Unix command has a manual page accessible via the man command:
This opens the manual for ls, showing all available options and usage examples. Navigation within man:
- Space or f — Move forward one page
- b — Move back one page
-
/pattern — Search for “pattern” (press
nfor next match) - h — Show help for navigating the manual
- q — Quit and return to the shell
For a quicker summary, many commands support --help:
When you encounter an unfamiliar command, man should be your first stop. The manuals are comprehensive and authoritative—they’re written by the people who created the commands.
5.4 Understanding the File System
We refer to all the files, folders, and programs on your computer as the filesystem. In Unix, everything is organized as files within a hierarchical directory structure (Figure 5.3). Keep in mind that folders and programs are technically also files, but this is a detail we rarely think about. Understanding this structure is essential for navigation.
The Directory Tree
The first concept you need to grasp to become a Unix user is how your filesystem is organized. You should think of it as a series of nested folders, each containing files, folders, and executable programs.
In Unix, we refer to folders as directories. Directories that are inside other directories are called subdirectories. The entire file system branches from a single root directory, represented by /:
/
├── Users/
│ └── wcresko/
│ ├── Documents/
│ ├── Downloads/
│ └── projects/
├── Applications/
├── Library/
└── tmp/
Figure 5.4 shows the same idea as a tree: the root directory / at the top, system directories and user home directories branching below it, and your own files nested deeper still.
/; user home directories sit under /Users (macOS) or /home (Linux).
Key directories include:
/- The root directory. Everything else is inside this.
/homeor/Users- Contains user home directories. Your files live here.
~-
Shorthand for your home directory (e.g.,
/home/wcresko). /tmp- Temporary files that may be deleted on reboot.
The home directory is where all your personal files are kept, as opposed to system files that come with your computer. Your home directory is typically named after your username. For example, if your username is wcresko, your home directory might be /home/wcresko on Linux or /Users/wcresko on macOS.
To understand the hierarchical nature of the filesystem, imagine you want to delete a file called cv.tex that lives in a resumes folder, which is inside a docs folder, which is inside your home directory. In a graphical file manager, you would double-click docs, then resumes, then drag cv.tex to the trash. This sequence of clicks mirrors the hierarchical path: cv.tex is inside resumes, which is inside docs, which is inside your home directory.
In Unix, we express this same relationship as a path: ~/docs/resumes/cv.tex. The forward slashes separate each level of the directory tree, and ~ is shorthand for your home directory.
When using Windows Subsystem for Linux (WSL), your Unix home directory is different from your Windows home directory. The Unix home will be something like /home/yourusername, while your Windows Documents folder is typically accessible at /mnt/c/Users/yourusername/Documents. Keep this distinction in mind when organizing files.
Absolute vs. Relative Paths
A path specifies the location of a file or directory. The name comes from the fact that the path string spells out the route you need to follow to get to a location from some starting point. Paths come in two forms:
- Absolute Paths
-
Start from the root directory (
/). They work from anywhere because they specify the complete location from the root of the filesystem. An absolute path always begins with/or~.
- Relative Paths
-
Start from your current directory. Shorter but depend on where you are. If a path does not start with
/or~, Unix assumes it is relative to your current working directory. Figure 5.5 shows how an absolute path is built up from the root of the tree.
/) down through each level to the file; the highlighted route spells out the path one directory at a time.
Unix provides shorthand symbols for common directory references:
| Symbol | Meaning |
|---|---|
. |
Current directory (where you are now) |
.. |
Parent directory (one level up) |
~ |
Your home directory |
- |
Previous directory (where you were before) |
These symbols can be combined in paths. For example, ../.. means “go up two directories,” and ~/projects means “the projects folder in my home directory.”
The Special Entries . and ..
The first two symbols in Table 5.2 are not just shorthand: they are real entries that exist in every directory. You can see them with ls -a, which shows hidden files (anything whose name begins with a dot):
The entry . (dot) refers to the directory itself, and .. (dot dot) refers to its parent (Figure 5.6). This is why cd .. moves you up one level: you are literally entering the .. entry of your current directory (Figure 5.7). It is also why you run a script in the current directory as ./script.sh, meaning “the file script.sh inside the directory I am in right now.”
., which points to the directory itself, and .., which points to its parent.
Use Tab completion! Start typing a file or directory name and press Tab to autocomplete. This saves typing and prevents errors.
5.6 Working with Files and Directories
Now that you can navigate, let’s learn to manipulate files and directories.
Creating Directories: mkdir
Creating Files: touch and nano
The touch command creates an empty file (or updates the timestamp of an existing file):
For files with content, use a text editor. nano is a simple terminal-based editor:
In nano:
- Type your content normally
-
Ctrl+Osaves the file (press Enter to confirm) -
Ctrl+Xexits nano -
Ctrl+Kcuts the current line;Ctrl+Upastes it back -
Ctrl+Wsearches (“where is”)
The help bar along the bottom of nano’s screen writes ^ for Ctrl, so the ^O it shows is the Ctrl+O above; the full set of nano shortcuts is in Section A.4. It is worth getting comfortable with nano even if you prefer editing files in your IDE, because it is the editor you will have on Talapas (Chapter 20), where there is no point-and-click and a quick fix to a job script happens in the terminal.
Copying Files: cp
Moving and Renaming: mv
The mv command both moves and renames files:
mv and cp will silently overwrite an existing file if you give two things the same name, and rm will delete whatever you tell it to. There is no “Are you sure?” dialog. Add -i (mv -i, cp -i, rm -i) to be asked before anything is overwritten or removed.
Deleting Files: rm and rmdir
The shell has no trash can! Deleted files are gone forever. Always double-check before using rm, especially with wildcards.
The -r (recursive) option with rm is powerful but dangerous. Consider using -i (interactive) to confirm each deletion when learning.
Viewing Files: cat, head, tail, less
Four commands let you look inside a file without opening an editor: cat prints the whole file, head and tail show its first or last lines, and less pages through it a screen at a time (press q to quit). For now, that one-line summary is enough to keep moving:
Each of these tools repays closer study, and Section 7.3 gives them a full treatment—including how to choose the right one for files too large to open at all.
File Information: wc
The wc (word count) command provides file statistics:
This shows: lines, words, and bytes. Useful options:
Finding Files: find
The find command searches for files and directories based on various criteria:
The find command is powerful for locating files across complex directory structures. Common options:
find options
| Option | Description |
|---|---|
-name |
Match filename (case-sensitive) |
-iname |
Match filename (case-insensitive) |
-type f |
Find files only |
-type d |
Find directories only |
-size +10M |
Larger than 10 megabytes |
-mtime -7 |
Modified within 7 days |
5.7 Naming Conventions
Before you start organizing projects with Unix, you should adopt a naming convention that you will use systematically for your files and directories. Good naming practices prevent countless problems and help you find files quickly, even months or years after creating them.
The Smithsonian Data Management Best Practices outlines five principles of file naming and organization:
- Have a distinctive, human-readable name that gives an indication of the content
- Follow a consistent pattern that is machine-friendly
- Organize files into directories that follow a consistent pattern
- Avoid repetition of semantic elements among file and directory names
- Have a file extension that matches the file format (never change extensions!)
For specific coding conventions, we recommend following the Tidyverse Style Guide, which provides consistent patterns for naming files and variables in R projects.
DO:
- Use lowercase letters, numbers, underscores (
_), and hyphens (-) - Use meaningful, descriptive names that indicate content
- Include dates in ISO format when relevant:
data_2024-01-15.csv - Use appropriate file extensions that match the format
DON’T:
- Use spaces (use
_or-instead) - Start names with
-(confused with command options) - Use special characters:
* ? [ ] ' " \ / < > | - Use extremely long names
Examples:
| Bad | Good | Why |
|---|---|---|
my data.csv |
my_data.csv |
Spaces cause problems in shell commands |
-results.txt |
results.txt |
Leading dash is confused with options |
data (copy).csv |
data_backup.csv |
Parentheses require escaping |
Joe's file.txt |
joes_file.txt |
Apostrophe causes quoting issues |
FINAL_v2_FINAL.csv |
analysis_2024-01-15.csv |
Date-based versioning is clearer |
Unix filenames are case-sensitive: Data.csv, data.csv, and DATA.csv are three different files on Linux, including Talapas. macOS is the confusing exception—its default file system preserves case but treats names that differ only in case as the same file—so a script that runs happily on your Mac, opening stickle_rnaseq.tsv when the file is really Stickle_RNAseq.tsv, will break the moment it runs on the cluster. Treat every filename as case-sensitive, spell names exactly as they are, and prefer all-lowercase names for files you create.
5.8 Getting Help
Manual Pages
Every Unix command has a manual page accessible via man:
Manual pages can be dense. Look for:
- NAME: What the command does
- SYNOPSIS: How to use it
- DESCRIPTION: Detailed explanation
- OPTIONS: Available flags
- EXAMPLES: If you’re lucky!
Press q to exit the manual.
Quick Help
Many commands support a --help option:
Searching for Commands
If you know what you want to do but not the command:
5.9 Preparing for a Data Science Project
Now let’s put these commands into practice by setting up a proper directory structure for a data science project. A well-organized project directory makes your work reproducible and easier to navigate.
We recommend creating a dedicated directory to hold all your projects. This keeps your work organized and separate from other files in your home directory:
For this example, let’s create a project analyzing gun violence data (based on a real project on GitHub). We’ll create subdirectories for raw data and processed data:
Our project structure now looks like this:
~/projects/
└── murders/
├── data/ # Raw data files
└── rdas/ # Processed R data files
- Create a
projectsdirectory in your home folder to contain all your work - Give each project its own directory with a descriptive name
- Use consistent subdirectory names across projects (e.g.,
data,results,scripts,figures) - Keep raw data separate from processed data
- Consider adding a
README.mdfile to describe the project
From any terminal, you can navigate to this project by typing:
As your projects grow, you might expand this structure:
~/projects/
└── murders/
├── data/ # Raw, unmodified data
├── rdas/ # Processed R data objects
├── scripts/ # Analysis scripts
├── results/ # Analysis output
├── figures/ # Generated plots
└── README.md # Project description
This organization keeps related files together, makes your analysis reproducible, and helps collaborators (including your future self) understand the project structure.
The Course Project Skeleton
The murders example above grew its structure as it needed it. For your own work, it is worth adopting one standard skeleton from the start and reusing it for every project (Noble, 2009; Wilson et al., 2017). The one used throughout this course separates data, code, results, and writing:
my_project/
├── data/
│ ├── raw/ # original data, exactly as it arrived - never modify
│ └── processed/ # cleaned or intermediate data, made by scripts
├── scripts/ # .sh, .R, .py, and .qmd files
├── output/
│ ├── figures/ # plots made by your scripts
│ └── tables/ # result tables made by your scripts
└── docs/ # notes, reports, manuscript drafts
Build it with a few mkdir -p calls:
(Section 6.1.1 introduces brace expansion, which collapses all of this into a single command, and the course’s make_project.sh script on the Example Files page builds the same skeleton complete with template files.)
Two rules give the skeleton its power:
-
data/raw/is read-only. Scripts read from it; nothing ever writes to it. Once you can change permissions (Section 6.2.2), enforce the rule withchmod a-w data/raw/*. -
output/is expendable. Everything in it can be regenerated by running the scripts on the raw data, so deleting it costs nothing. If deleting it would cost something, a result is being made by hand that should be made by a script.
data/raw/ is locked read-only so nothing can silently overwrite the original data, while everything in output/ is regenerated by the scripts and is therefore safe to delete.
Two habits complete the picture. Use relative paths inside your scripts (data/raw/samples.csv, never /Users/you/my_project/data/raw/samples.csv), so the whole project can move to another machine—or to Talapas—and still run. And number your scripts in the order they run (01_clean.R, 02_analyze.R, 03_plot.R), so that a directory listing reads as the pipeline’s documentation.
5.10 Summary
This chapter covered the fundamentals of Unix navigation and file manipulation:
- The shell interprets commands and interfaces with the operating system
- Commands follow the pattern:
command -options arguments - The file system is a hierarchical tree starting at
/ - Navigation uses
pwd,ls, andcd - File operations include
cp,mv,rm,mkdir, andtouch - Every directory contains
.(itself) and..(its parent);cd ..moves up one level - Good naming conventions prevent problems
- Manual pages (
man) provide detailed help
These commands form the foundation for everything else in Unix. Practice them until they become automatic!
The second half of the shell story—wild cards, file permissions, and configuring your shell environment—continues in Chapter 6.
For a comprehensive reference of Unix commands with syntax and examples, see Appendix B.
Additional Reading
To further develop your Unix skills:
- The Linux Command Line by William Shotts — A comprehensive, free book on Linux command-line fundamentals
- Software Carpentry: The Unix Shell (Software Carpentry, 2024) — An excellent interactive tutorial designed for scientists
- “The Unix Programming Environment” (Kernighan & Pike, 1984) by Kernighan and Pike — A classic text on Unix philosophy and tools
- Explain Shell — An interactive tool that breaks down complex shell commands into understandable parts
- Linux Pocket Guide by Daniel Barrett — A handy quick reference for essential commands
For macOS-specific Unix guidance:
- macOS Terminal User Guide — Apple’s official documentation for the Terminal application
Exercises
Exercise 1: Navigation Practice
Without using a GUI file browser:
- Open your terminal
- Print your current directory
- Navigate to your home directory
- List all files including hidden ones
- Navigate to your Documents folder (or create it if needed)
- Return to your home directory using the shortcut
Exercise 2: File Operations
- Create a new directory called
practice - Navigate into it
- Create an empty file called
notes.txt - Create a subdirectory called
backup - Copy
notes.txtto thebackupdirectory - Rename the original
notes.txttomy_notes.txt - List the contents of both directories to verify your work
- Clean up by removing the entire
practicedirectory
Exercise 3: Exploration
Use manual pages to answer:
- What does the
ls -Roption do? - How can you create a directory and all necessary parent directories in one command?
- What command shows disk usage, and what option makes the output human-readable?