6  Permissions, Wild Cards, and the Shell Environment

NoteLearning Objectives

After completing this chapter, you will be able to:

  • Match many files at once with wild cards (*, ?, [ ]) safely
  • Read a permission string from ls -l and change permissions with chmod
  • Make a script executable and run it with ./
  • Use environment variables and understand how the PATH finds programs
  • Customize your shell with configuration files, and connect to remote computers with ssh

Chapter 5 introduced the file system and the core commands for moving around it and managing files. This chapter is the second half of that story: the shell features that turn those commands into real workflows. Wild cards let one command act on many files; permissions control who can read, change, and—crucially for the scripts you are about to start writing—run a file; and environment variables, the PATH, and configuration files determine how your shell behaves every time it starts. These are exactly the pieces you need in hand before Chapter 7 chains commands into pipelines and Chapter 10 turns pipelines into programs.

6.1 Wild Cards

One of the most powerful features of Unix is wild cards—special characters that match multiple files at once. Suppose you want to remove all temporary HTML files from a project. Removing them one by one would be tedious, but with wild cards you can match them all:

# List all HTML files
$ ls *.html

# Remove all HTML files
$ rm *.html

The * wild card matches any number of any characters (including none). The ? wild card matches exactly one character:

# All CSV files, all files starting with "data", all text and markdown files
$ ls *.csv
$ ls data*
$ ls *.txt *.md

# Match files like file-001.html, file-002.html, file-999.html
$ ls file-???.html

# sample1.txt, sample2.txt, ... but not sample10.txt
$ ls sample?.txt

# Remove all files starting with "temp"
$ rm temp*

# Match files with any extension
$ ls data.*

Square brackets [ ] match any single character from a set or range:

# sample1.txt, sample2.txt, or sample3.txt
$ ls sample[123].txt

# data_0.csv through data_9.csv
$ ls data_[0-9].csv

# fileA.txt through fileZ.txt
$ ls file[A-Z].txt

Wild cards are expanded by the shell before the command runs, so they work with any command, not just ls:

# Count lines in every CSV file
$ wc -l *.csv

# Copy all R scripts to a backup folder
$ cp *.R backup/

# Search for "gene" in all text files
$ grep "gene" *.txt
WarningWild Card Danger

Combining rm with * can be dangerous! The command rm * deletes everything in the current directory without asking for confirmation. Always use ls first to see what a wild card pattern matches before using it with rm, and consider rm -i for interactive confirmation when deleting with wild cards.

Brace Expansion

A close cousin of the wild cards is brace expansion. A comma-separated list inside { } expands into one copy of the surrounding text per alternative, and {start..end} expands a numeric or alphabetic sequence:

# Both the .fastq and the .md files, in one command
$ ls *.{fastq,md}

# A quick backup copy: expands to  cp notes.txt notes.txt.bak
$ cp notes.txt{,.bak}

# sample_1 through sample_5
$ echo sample_{1..5}
sample_1 sample_2 sample_3 sample_4 sample_5

Unlike the wild cards above, brace expansion never looks at which files exist—the shell simply generates the strings and hands them to the command. That makes it the tool of choice for creating things rather than matching them. Braces nest, so a single mkdir -p can build an entire project directory tree (Section 5.9) in one line:

$ mkdir -p my_project/{data/{raw,processed},scripts,output/{figures,tables},docs}
$ ls -R my_project

Keep the distinction straight and the two features cover complementary jobs: a wild card such as sample_*.txt selects from names already present (and matches nothing if no file fits), while sample_{1..5}.txt generates all five names whether or not the files exist. Wild cards select; braces generate.

6.2 File Permissions and Ownership

Unix is a multi-user system, and every file has permissions controlling who can read, write, or execute it. Understanding permissions is essential for security and for running scripts (Section 10.2.2).

Understanding Permission Strings

When you run ls -l, you see permission strings like:

-rwxr-xr-x 1 wcresko staff 2048 Jan 15 10:30 analysis.sh
drwxr-xr-x 3 wcresko staff   96 Jan 15 09:00 data

Let’s decode -rwxr-xr-x (Table 6.1):

Table 6.1: Decoding a permission string such as -rwxr-xr-x
Position Meaning
1st character Type: - (file), d (directory), l (link)
Characters 2-4 Owner permissions: rwx
Characters 5-7 Group permissions: r-x
Characters 8-10 Others permissions: r-x

The three permission types are:

  • r (read): View file contents or list directory
  • w (write): Modify file or add/remove files in directory
  • x (execute): Run file as program or enter directory
  • -: Permission denied for that operation

So -rwxr-xr-x means: this is a regular file; the owner can read, write, and execute it; members of the group can read and execute it; and everyone else can read and execute it.

Figure 6.1: Decomposing the permission string -rwxr-xr-x. The first character gives the file type; the next nine form three triplets—owner, group, and others—each flagging read, write, and execute. Summing read = 4, write = 2, and execute = 1 within each triplet gives the octal value 755.

Changing Permissions: chmod

The chmod command changes file permissions. There are two notations:

Symbolic notation uses letters for who (u user/owner, g group, o others, a all) and operators for what (+ add, - remove, = set exactly):

# Add execute permission for owner
$ chmod u+x script.sh

# Remove write permission for group and others
$ chmod go-w data.csv

# Set exact permissions (owner: rwx, group: rx, others: rx)
$ chmod u=rwx,g=rx,o=rx script.sh

Octal notation assigns each permission a value—read = 4, write = 2, execute = 1—and adds them up for each of the three sets. For example, rwx = 4 + 2 + 1 = 7 and r-x = 4 + 0 + 1 = 5 (Table 6.2):

Table 6.2: Octal permission values used by chmod
Number Permission
7 rwx (read + write + execute)
6 rw- (read + write)
5 r-x (read + execute)
4 r– (read only)
0 — (no permissions)
# Common permission sets
$ chmod 755 script.sh    # rwxr-xr-x (executable script)
$ chmod 644 data.csv     # rw-r--r-- (readable file)
$ chmod 700 private/     # rwx------ (private directory)
$ chmod 600 secrets.txt  # rw------- (private file)

A few patterns cover almost every situation: 755 for scripts (you can modify, everyone can run), 644 for data files (you can modify, everyone can read), and 700 or 600 for directories and files that only you should touch.

Figure 6.2: The octal permission calculator: read = 4, write = 2, and execute = 1 are summed within each triplet to give one digit. A handful of values—755 for scripts, 644 for data files, 600 for private files—cover most situations, and 444 locks raw data read-only for everyone.

One pattern deserves to become a habit: making your raw data read-only. Raw data files should never change after they arrive (Section 5.9), and removing write permission enforces that—no script, yours or anyone else’s, can silently overwrite them:

# Nobody can write: protect your raw data
$ chmod a-w data/raw/*

# The same idea in octal: r--r--r--, read-only for everyone
$ chmod 444 data/raw/*.tsv

# While you are at it: make every script in a project runnable at once
$ chmod u+x scripts/*.sh

# Always verify what changed
$ ls -l data/raw

With the raw files locked down, a mistake costs you a prompt instead of your data: rm asks before deleting a write-protected file, and output redirection (>) aimed at one refuses outright. You can always restore write permission with chmod u+w when a file genuinely must change—the point is that changing raw data should require a deliberate step.

TipPermissions matter more on shared systems

On your laptop you are usually the only user, so the group and others triplets rarely come into play. On a shared system such as Talapas they are how a lab manages its files: your group there is your PIRG, and the group permissions decide whether labmates can read your results or write into a shared project directory. You will meet this again in Chapter 20.

Making Scripts Executable

The most common reason you will reach for chmod is to run a script you have written. A new text file is not executable, so trying to run it directly fails:

# Create a two-line script
$ echo '#!/bin/bash' > myscript.sh
$ echo 'echo "Hello, World!"' >> myscript.sh

# Try to run it
$ ./myscript.sh
bash: ./myscript.sh: Permission denied

# Add execute permission, then try again
$ chmod +x myscript.sh
$ ./myscript.sh
Hello, World!

The #!/bin/bash line at the top (the “shebang”) tells Unix which program should interpret the script, and the ./ prefix says “the file in the current directory” (Section 5.4.3). You will also see the shebang written as #!/usr/bin/env bash—the form used in the lecture slides and the course’s example scripts. The two are interchangeable for this course: the env form asks Unix to find bash wherever it is installed, which makes a script more portable to systems where bash is not at /bin/bash (some macOS and cluster setups).

Two more things worth knowing from day one:

  • bash myscript.sh works without execute permission. In that command you are running bash (which is already executable) and handing it the file to read, so no x bit is needed. The chmod +x requirement applies only to running the script directly as ./myscript.sh.
  • Downloaded scripts always need chmod +x once. The execute bit is not part of the file’s contents, so it does not survive a download. The four worked example scripts on the course’s Example Files page are a good place to practice: download them, run chmod +x *.sh, then read their comments to see how real scripts are put together.

Shell scripts are the subject of Chapter 10.

Changing Ownership: chown

The chown command changes file ownership:

# Change owner
$ chown alice myfile.txt

# Change owner and group
$ chown alice:researchers myfile.txt

# Recursively change directory ownership
$ chown -R alice:researchers project/

Superuser Access: sudo

Some operations require administrator (root) privileges. The sudo command temporarily elevates your permissions:

# Install software (requires admin rights)
$ sudo apt install cowsay

# Edit a system file
$ sudo nano /etc/hosts
Warning

Use sudo carefully! With great power comes great responsibility. Commands run with sudo can modify or delete system files, potentially breaking your computer. Always double-check commands before running them with sudo.

6.3 Environment Variables

Unix has settings that affect your command-line environment called environment variables. These are distinguished by a $ prefix. Some important ones:

# Your home directory
$ echo $HOME
/home/wcresko

# Your username
$ echo $USER
wcresko

# Your current shell
$ echo $SHELL
/bin/bash

# Where Unix looks for programs
$ echo $PATH
/usr/local/bin:/usr/bin:/bin

# Your current working directory (the same thing pwd prints)
$ echo $PWD
/home/wcresko/projects

# See all environment variables (there are many; page through them)
$ env | less
$ printenv HOME

Environment variables control how your shell behaves and where it looks for programs. The PATH variable is particularly important—it’s a colon-separated list of directories where Unix searches for executable programs.

You can also define your own variables. Assign with NAME=value (no spaces around the =) and refer to the value with a $ prefix:

# Set a variable
$ PROJECT_DIR=~/projects/analysis

# Use it (note the $)
$ cd $PROJECT_DIR
$ ls $PROJECT_DIR/data

A variable set this way is visible only to the current shell. To make it available to the programs you launch from that shell (R, Python, a script), prefix the assignment with export:

$ export PROJECT_DIR=~/projects/analysis

Shell variables are the building blocks of shell scripts, which are covered in Chapter 10.

6.4 Executables and the PATH

In Unix, all programs are files—they’re called executables. When you type ls, grep, or git, Unix runs an executable file. You can find where a program lives using which:

$ which git
/usr/bin/git

$ which python
/usr/bin/python

The PATH environment variable tells Unix where to look for executables. When you type a command, Unix searches through each directory in PATH until it finds a matching program:

$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/home/wcresko/bin

Unix searches these directories in order. If you have your own scripts in ~/bin, you can add that directory to your PATH to make them accessible from anywhere.

6.5 Shells and Configuration Files

The shell is the program that interprets your commands. Bash (Bourne Again Shell) is the most common, but others exist like zsh (default on modern macOS) and fish. You can see which shell you’re using:

$ echo $SHELL
/bin/bash

Your shell can be customized through configuration files that run when you open a terminal (Table 6.3):

Table 6.3: Configuration files for common shells
Shell Configuration Files
Bash ~/.bashrc, ~/.bash_profile
Zsh ~/.zshrc, ~/.zprofile

These files can set environment variables, define aliases (shortcuts for commands), and customize your prompt. For Bash, ~/.bashrc is read by interactive shells and ~/.bash_profile by login shells; on macOS with zsh, put the same lines in ~/.zshrc. Common customizations include:

# Add a directory of your own scripts to the front of your PATH
export PATH="$HOME/bin:$PATH"

# Create an alias (shortcut) for a common command
alias ll='ls -la'
alias rm='rm -i'    # always ask before deleting

# Set default options for programs
export R_LIBS_USER="$HOME/R/library"

After editing a configuration file, apply the changes either by opening a new terminal or by running:

$ source ~/.bashrc      # or: source ~/.zshrc

6.6 Connecting to Remote Computers: ssh and scp

One of the main reasons to learn the shell is that it works the same way on a computer across the room or across the country. The ssh command (secure shell) opens a shell on a remote machine:

$ ssh username@server.university.edu

Once connected, everything in this chapter—pwd, ls, cd, mkdir, and the rest—works exactly as it does locally, except that the files live on the remote system. This is how you will use UO’s Talapas cluster (Chapter 20). Type exit to close the connection.

The scp command (secure copy) copies files between your computer and a remote one. Its syntax mirrors cp, with user@host: prefixed to the remote path:

# Local file -> remote directory
$ scp local_file.txt username@server.edu:~/destination/

# Remote file -> current local directory
$ scp username@server.edu:~/remote_file.txt .

For large or repeated transfers, rsync is more efficient because it copies only what has changed; see Section 20.5.3.

6.7 Commands Worth Learning

Several other commands are worth knowing about as you advance (Table 6.4); grep gets its own treatment in Chapter 9:

Table 6.4: Additional Unix commands worth learning
Command Description
tar Archive files and directories into one file
ssh Connect securely to another computer (Section 6.6)
ln Create symbolic links (shortcuts to files)
nano or vim Edit files in the terminal
awk Column-oriented text processing (covered in Section 9.10)
sed Stream editor for text transformation (covered in Section 9.9)

6.8 File Manipulation from R

You can also perform file operations from within R using functions like file.create(), file.copy(), file.rename(), and file.remove(). See ?files in R for the complete list. The unlink() function removes files and directories.

While not generally recommended, you can run Unix commands from R using the system() function:

system("ls -la")

This can be useful for integrating Unix tools into R scripts, but native R functions are usually preferable for portability.

6.9 Summary

This chapter completed your shell foundations:

  • Wild cards (*, ?, [ ], { }) are expanded by the shell before the command runs—preview with ls before using them with rm
  • ls -l permission strings decode as owner/group/others triplets of rwx
  • chmod changes permissions symbolically (u+x, go-w) or in octal (755, 644)
  • A new script needs chmod +x once before ./script.sh will run; the shebang line says which interpreter runs it
  • Environment variables ($HOME, $PATH) configure your session; the PATH is where the shell looks for programs
  • Shell configuration files (.bashrc, .zshrc) make your customizations permanent
  • ssh and scp extend everything in this book to remote machines like Talapas (Chapter 20)
Tip

For a comprehensive reference of Unix commands with syntax and examples, see Appendix B.

Exercises

TipPractice Exercises

Exercise 1: Wild Cards and Structure

  1. Make five directories called dir_1 through dir_5 with a single mkdir command
  2. Inside each, create file_1.txt, file_2.txt, and file_3.txt using touch
  3. Open dir_1/file_1.txt in nano, type a few words, save, and print it with cat
  4. Use a wild card to list every file_2.txt across all five directories in one command
  5. Delete all files in dir_3 with a single rm command (check with ls first), then remove dir_3 with rmdir

Exercise 2: File Permissions

  1. Create a short script that prints “Hello, World!” and the current date (date)
  2. Try to run it with ./—what error do you get?
  3. Make it executable with chmod +x and run it again
  4. Examine the permissions of several files with ls -l and decode each permission string
  5. Practice changing permissions with both symbolic (chmod u+x) and octal (chmod 755) notation, checking the result with ls -l each time

Exercise 3: Your Environment

  1. Print your $HOME, $USER, and $SHELL variables
  2. Print your $PATH and count how many directories it contains (hint: tr ':' '\n' | wc -l)
  3. Use which to find where ls, bash, and python3 live
  4. Create an alias for ls -lh in your shell configuration file, reload it with source, and test it