# Lecture 1 – Course Overview & The Shell
### Overview
This lecture lays the foundation for the entire course. Before diving into any specific tool, it answers a deceptively simple question: **why do developers need to learn this at all?**
Modern computers ship with beautiful graphical interfaces — but those interfaces only expose a fraction of what your machine can do. The shell is the trapdoor to everything else. It gives you **direct, programmable, composable access** to your system and its tools.
This lecture covers:
* What a shell is and why it exists
* How the shell finds and runs programs
* How to navigate the filesystem from the command line
* How to connect programs together using input/output streams and pipes
* The role of the root user and the `sudo` command
#### Key Takeaways
* The shell is a **programming environment**, not just a command runner.
* The `$PATH` variable controls which programs your shell can find by name.
* Paths are either **absolute** (starting at `/`) or **relative** (starting from where you are now).
* Programs have two standard streams: **stdin** (input) and **stdout** (output) — and you can rewire both.
* The `|` pipe operator **chains programs together**, letting output from one become input to the next.
* Shell operators like `>`, `<`, and `|` are handled **by the shell**, not by the programs themselves — a subtle but critical distinction.
***
### Core Concepts
#### The Shell as a Programming Environment
The shell isn't just a place to type commands. It's a **full programming environment** with variables, conditionals, loops, and functions (covered in Lecture 2).
When you type a command like `echo hello`, the shell:
1. Parses the line (splits by whitespace)
2. Looks up the program name (`echo`) via `$PATH`
3. Executes it, passing `hello` as an argument
This means the shell **interprets your input** before anything is run. Understanding this parsing step is essential for avoiding common bugs around spaces, quotes, and special characters.
***
#### The `$PATH` Variable
When you type a bare command name like `ls` or `python3`, how does the shell know where to find it?
It searches every directory listed in the environment variable `$PATH` — a colon-separated list of directories.
```bash
echo $PATH
# /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
```
The shell scans these directories **left to right** and runs the first match it finds.
You can check exactly which binary will be used for a given command name:
```bash
which python3
# /usr/bin/python3
```
You can also bypass `$PATH` entirely by typing the full path:
```bash
/bin/echo "Hello from the full path"
```
> **Why it matters:** If a command "doesn't exist" or runs the wrong version, `$PATH` is almost always the reason. Knowing how to inspect and modify it is a foundational skill.
***
#### Absolute vs. Relative Paths
Every file and directory on a Linux/macOS system lives inside a single tree rooted at `/`.
| Path Type | Description | Example |
| ------------ | ---------------------------------- | ----------------------- |
| **Absolute** | Starts from the root `/` | `/home/alice/documents` |
| **Relative** | Starts from your current directory | `documents/notes.txt` |
Two special relative path components:
* `.` → the current directory
* `..` → the parent directory
```bash
cd /home/alice # go to /home/alice
cd .. # now in /home
cd ./alice # back to /home/alice (same as cd alice)
../../bin/echo hi # runs /bin/echo from /home/alice
```
***
#### Input/Output Streams and Redirection
Every program in Unix has three default streams:
| Stream | Number | Default |
| ------ | ------ | --------------- |
| stdin | 0 | Keyboard |
| stdout | 1 | Terminal screen |
| stderr | 2 | Terminal screen |
You can **redirect** these streams to and from files:
```bash
echo hello > hello.txt # stdout → file (overwrite)
echo hello >> hello.txt # stdout → file (append)
cat < hello.txt # file → stdin
cat < hello.txt > copy.txt # file → stdin, stdout → file
```
***
#### Pipes: Connecting Programs
The `|` operator wires the **stdout of one program** directly into the **stdin of the next**. This lets you build data-processing pipelines from small, composable tools.
```bash
ls -l / | tail -n1
```
This lists the last entry in the root directory — `ls` produces a list, `tail` keeps only the final line.
```bash
curl --head --silent google.com | grep --ignore-case content-length | cut --delimiter=' ' -f2
```
This fetches HTTP headers, filters to the `content-length` line, then extracts the byte count value.
***
#### File Permissions
When you run `ls -l`, you see entries like:
```
drwxr-xr-x 1 missing users 4096 Jun 15 2019 missing
```
Breaking this down:
| Characters | Meaning |
| ---------- | ------------------------------------- |
| `d` | This is a directory |
| `rwx` | Owner can read, write, execute |
| `r-x` | Group can read and execute (no write) |
| `r-x` | Everyone else can read and execute |
For directories specifically:
* `r` → can list contents (`ls`)
* `w` → can add/remove files inside
* `x` → can enter the directory (`cd`)
***
#### The Root User and `sudo`
The **root** user has unrestricted access to every file and operation on the system. You usually don't log in as root — instead, you use `sudo` ("super-user do") to run a single command with elevated privileges.
```bash
sudo apt install htop # install a package as root
```
> **Important caveat:** `sudo` only elevates the program you're running — **not the shell itself**. This matters when using redirection.
***
### Mental Model
#### How the Shell Interprets a Command
Think of the shell as a translator standing between you and your operating system. When you press Enter, it follows this process:
```
You type: ls -la /home
│
▼
1. Split into tokens: ["ls", "-la", "/home"]
│
▼
2. First token = program name: "ls"
│
▼
3. Look up "ls" in $PATH directories
│
▼
4. Found: /usr/bin/ls
│
▼
5. Execute /usr/bin/ls with args ["-la", "/home"]
│
▼
6. Output goes to your terminal (stdout)
```
The shell is not running `ls` itself — it's **finding and launching** a separate program. The shell is the coordinator; the programs are the workers.
***
#### The Pipe Mental Model
Think of a pipe `|` as a **physical tube** between two programs:
```
Program A writes → [ pipe buffer ] → Program B reads
```
Program A doesn't know about Program B. Program B doesn't know about Program A. The shell wires them together. This is the Unix philosophy: **small tools that do one thing well, composed into powerful pipelines**.
***
#### Why `sudo echo 3 > brightness` Fails
This is one of the most illuminating shell lessons:
```bash
sudo echo 3 > brightness # ← FAILS with "Permission denied"
```
Why? Because `>` is a **shell operator**, processed by the current shell **before** `sudo` runs. The sequence is:
1. Shell tries to open `brightness` for writing — **as your unprivileged user**
2. Shell is denied (the file requires root)
3. `sudo echo 3` never even gets a chance to run
The fix is to use `tee`, which is the actual program that opens the file:
```bash
echo 3 | sudo tee brightness # ← Works: tee runs as root and opens the file
```
Now `sudo` elevates `tee` itself, which is the process actually writing to the file.
***
### Commands and Syntax
#### Command: `date`
```bash
date
```
Prints the current date and time.
```bash
date
# Fri 10 Jan 2020 11:49:31 AM EST
```
***
#### Command: `echo`
```bash
echo [string ...]
```
Prints its arguments to stdout. Useful for displaying values and testing shell behavior.
```bash
echo hello world # prints: hello world
echo $PATH # prints value of PATH variable
echo "My name is $USER" # expands variable inside double quotes
echo '$USER' # single quotes: no variable expansion
```
***
#### Command: `which`
```bash
which program-name
```
Shows the full path of the binary that would be executed for a given command name.
```bash
which python3 # /usr/bin/python3
which ls # /usr/bin/ls
```
Use this to diagnose "wrong version" issues or verify which binary is actually running.
***
#### Command: `pwd`
```bash
pwd
```
**P**rint **W**orking **D**irectory — shows your current location in the filesystem.
```bash
pwd
# /home/alice/projects
```
***
#### Command: `cd`
```bash
cd [path]
```
**C**hange **D**irectory — moves you to a new location.
| Command | Effect |
| ---------------- | ------------------------- |
| `cd /home/alice` | Go to absolute path |
| `cd documents` | Go to relative path |
| `cd ..` | Go up one level |
| `cd ~` | Go to your home directory |
| `cd -` | Go to previous directory |
```bash
cd /tmp
cd ../home
cd ~
```
***
#### Command: `ls`
```bash
ls [flags] [path]
```
Lists the contents of a directory.
| Flag | Effect |
| ----- | ------------------------------------- |
| `-l` | Long format (permissions, size, date) |
| `-a` | Show hidden files (starting with `.`) |
| `-h` | Human-readable file sizes |
| `-la` | Combine long format + hidden files |
```bash
ls # current directory
ls /etc # specific directory
ls -la ~ # home directory, long format, including hidden
```
***
#### Command: `mv`
```bash
mv source destination
```
**Moves** or **renames** files and directories.
```bash
mv old-name.txt new-name.txt # rename
mv report.pdf ~/Documents/ # move to another directory
```
***
#### Command: `cp`
```bash
cp source destination
```
**Copies** files.
```bash
cp config.yaml config.yaml.bak # backup a file
cp -r src/ dst/ # copy a directory recursively
```
***
#### Command: `mkdir`
```bash
mkdir directory-name
```
Creates a new directory.
```bash
mkdir projects
mkdir -p projects/2024/q1 # -p creates intermediate directories too
```
***
#### Command: `man`
```bash
man program-name
```
Opens the **manual page** for a program. Press `q` to exit, arrow keys to scroll.
```bash
man ls
man curl
man bash
```
> **Tip:** If `man` isn't available, try `program --help` for a shorter reference.
***
#### Command: `cat`
```bash
cat [file ...]
```
Con**cat**enates and prints files to stdout.
```bash
cat notes.txt # print a file
cat file1.txt file2.txt # concatenate two files
cat < input.txt > output.txt # copy a file via redirection
```
***
#### Command: `sudo`
```bash
sudo command
```
Runs a command as the root user. You'll be prompted for your password.
```bash
sudo apt update
sudo chmod 600 /etc/secret
echo "value" | sudo tee /sys/class/... # correct way to redirect to privileged files
```
***
#### Command: `tee`
```bash
tee [file]
```
Reads from stdin and writes to **both** stdout and a file simultaneously. Essential when you need root-privileged file writes in a pipeline.
```bash
echo 3 | sudo tee brightness # write to file as root
echo "log entry" | tee -a log.txt # append to file (-a flag)
```
***
### Command Flow Diagrams
#### How a Simple Command Executes
```mermaid
graph LR
U["User types: echo hello"] --> S["Shell parser"]
S --> T["Tokenize: ['echo', 'hello']"]
T --> P["Look up 'echo' in $PATH"]
P --> B["/bin/echo found"]
B --> E["Execute /bin/echo with arg 'hello'"]
E --> O["stdout → Terminal: hello"]
```
***
#### How a Pipe Pipeline Works
```mermaid
graph LR
CMD1["ls -l /"] -->|stdout| PIPE1["|"]
PIPE1 -->|stdin| CMD2["tail -n1"]
CMD2 -->|stdout| TERM["Terminal output"]
```
Each program only sees its own stdin and stdout. The shell builds the pipe connections invisibly.
***
#### How I/O Redirection Works
```mermaid
graph TB
KB["Keyboard"] -->|default stdin| P["Program"]
P -->|default stdout| TERM["Terminal"]
subgraph "With redirection: cmd < in.txt > out.txt"
F1["in.txt"] -->|stdin| P2["Program"]
P2 -->|stdout| F2["out.txt"]
end
```
***
#### Why `sudo echo > file` Fails
```mermaid
sequenceDiagram
participant Shell
participant OS
participant Sudo
participant Echo
Shell->>OS: Open "brightness" for writing (as current user)
OS-->>Shell: ❌ Permission Denied
Note over Shell: Execution stops here.
sudo echo never runs.
Note over Shell,Echo: Correct approach: echo 3 | sudo tee brightness
Shell->>Sudo: Launch tee as root
Sudo->>OS: Open "brightness" for writing (as root)
OS-->>Sudo: ✅ Success
Echo->>Sudo: Pipe "3" to tee's stdin
Sudo->>OS: Write "3" to brightness
```
***
### Command Pipeline Examples
#### Example 1: View the Last Entry in Root Directory
```bash
ls -l / | tail -n1
```
| Step | Command | What it does |
| ---- | ---------- | ----------------------------------- |
| 1 | `ls -l /` | Lists all files in `/` with details |
| 2 | `tail -n1` | Keeps only the last line of output |
***
#### Example 2: Extract HTTP Content-Length
```bash
curl --head --silent google.com | grep --ignore-case content-length | cut --delimiter=' ' -f2
```
| Step | Command | What it does |
| ---- | ----------------------------------- | ---------------------------------------------------- |
| 1 | `curl --head --silent google.com` | Fetches only HTTP headers, silently |
| 2 | `grep --ignore-case content-length` | Keeps only the content-length header line |
| 3 | `cut --delimiter=' ' -f2` | Splits on space, takes the second field (the number) |
```mermaid
graph LR
C["curl --head --silent google.com"] -->|HTTP headers| G["grep content-length"]
G -->|matching line| K["cut -f2"]
K -->|byte count| T["Terminal: 219"]
```
***
#### Example 3: Copy a File Through Redirection
```bash
cat < hello.txt > hello2.txt
```
| Step | What happens |
| -------------- | ------------------------------------------ |
| `< hello.txt` | Shell wires `hello.txt` to `cat`'s stdin |
| `cat` | Reads stdin and writes to stdout |
| `> hello2.txt` | Shell wires `cat`'s stdout to `hello2.txt` |
Result: `hello2.txt` is a copy of `hello.txt`. The `cat` program itself has no idea files are involved.
***
### Real World Workflows
#### Quickly Inspect a Log File's Tail
```bash
tail -n 50 /var/log/syslog
```
Read the last 50 lines of a system log. Add `-f` to follow it live:
```bash
tail -f /var/log/syslog
```
***
#### Find Where a Program Lives
```bash
which git
which python3
which node
```
Useful when you have multiple versions installed or when a tool "isn't found" despite being installed.
***
#### Write Output to a File and Screen Simultaneously
```bash
./long-running-script.sh | tee output.log
```
Output streams to your terminal **and** gets saved to `output.log` at the same time.
***
#### Safely Check Before Running Destructive Commands
```bash
ls /tmp/old_builds/ # review what's there first
rm -rf /tmp/old_builds/ # then remove it
```
Never run `rm -rf` blindly. Use `ls` first.
***
#### Create a Directory Structure in One Shot
```bash
mkdir -p ~/projects/myapp/{src,tests,docs,config}
```
The `-p` flag creates intermediate directories. The `{...}` brace expansion creates multiple subdirectories at once.
***
### Productivity Tricks
#### Navigate Faster
| Trick | Command |
| -------------------------- | ------------------------------------------ |
| Go home | `cd` or `cd ~` |
| Jump to previous directory | `cd -` |
| Tab completion | Press `Tab` to complete file/command names |
| Double-Tab | Shows all possible completions |
***
#### Reuse Previous Commands
| Shortcut | Effect |
| ---------- | --------------------------------------------------------------- |
| `↑` / `↓` | Scroll through command history |
| `Ctrl + R` | **Reverse search** — type part of a past command to find it |
| `!!` | Repeat the last command |
| `sudo !!` | Re-run the last command with sudo (very useful when you forget) |
***
#### Write Multi-Line Commands Cleanly
Use `\` to continue a long command on the next line:
```bash
curl --head \
--silent \
--max-time 5 \
google.com
```
***
#### Read the Manual Effectively
```bash
man ls # full manual
ls --help # quick reference
info ls # GNU info pages (more detailed)
```
Inside `man`, use:
* `/keyword` to search
* `n` to jump to next match
* `q` to quit
***
### Common Mistakes
#### ❌ Forgetting to Quote Paths with Spaces
**Wrong:**
```bash
cd My Documents
# Shell sees: cd My (then) Documents — two arguments!
```
**Correct:**
```bash
cd "My Documents"
cd My\ Documents # backslash-escape the space
```
**Why:** The shell splits on whitespace before passing arguments to programs. A space without quoting = a new argument.
***
#### ❌ Assuming `sudo` Covers Shell Operators
**Wrong:**
```bash
sudo echo "data" > /etc/protected-file
# Error: shell opens the file as your user, not root
```
**Correct:**
```bash
echo "data" | sudo tee /etc/protected-file
```
**Why:** `>` is processed by your shell, not by `sudo` or `echo`. The shell runs as you, so it can't open root-only files for writing.
***
#### ❌ Using `>` When You Mean `>>`
**Wrong (overwrites the file every time):**
```bash
echo "entry 1" > log.txt
echo "entry 2" > log.txt # ← destroys "entry 1"!
```
**Correct (appends):**
```bash
echo "entry 1" >> log.txt
echo "entry 2" >> log.txt
```
***
#### ❌ Running Commands Without Checking `$PATH`
**Wrong:**
```bash
python myScript.py # runs python2 when you expected python3
```
**Correct:**
```bash
which python # check which binary will run
python3 myScript.py # be explicit
```
***
#### ❌ Confusing `.` and `./`
**Context: running a script**
```bash
. myscript.sh # runs in current shell (dot-space = "source")
./myscript.sh # runs as a subprocess
```
These are completely different. `source` (or `.`) runs commands in your current shell session; `./script` spawns a child process.
***
### Exercises
#### Beginner Exercises
1. **Check your shell:** Run `echo $SHELL`. Does it output something like `/bin/bash` or `/usr/bin/zsh`?
2. **Navigate the filesystem:** Use `cd`, `pwd`, and `ls` to explore:
* Your home directory (`~`)
* `/etc` — system configuration files
* `/bin` — common binaries
* `/tmp` — temporary files
3. **Read a manual page:** Run `man ls`. Use `/` to search for the word "human". Press `q` to quit.
4. **Redirect output:** Run:
```bash
echo "Hello, shell" > /tmp/test.txt
cat /tmp/test.txt
```
Then append another line:
```bash
echo "Second line" >> /tmp/test.txt
cat /tmp/test.txt
```
5. **Explore permissions:** Run `ls -la ~`. Identify the permission bits on three different files. What do the `r`, `w`, `x` characters mean for each?
***
#### Intermediate Exercises
6. **Create a directory tree:**
```bash
mkdir -p /tmp/missing
cd /tmp/missing
touch semester
ls -l
```
What permissions does `semester` have? Can you execute it?
7. **Use `which` to trace programs:** Run `which echo`, `which ls`, `which man`. Now try:
```bash
/bin/echo "I used the full path"
```
How does this differ from running `echo` normally?
8. **Build a pipeline:** Without creating any intermediate files, find out how many entries are in `/bin`:
```bash
ls /bin | wc -l
```
Explain each step.
9. **Try the `tee` trick:**
```bash
echo "test output" | tee /tmp/tee-test.txt
cat /tmp/tee-test.txt
```
Notice it printed to both the terminal **and** the file. Why is this useful?
***
#### Advanced Challenge
10. **The semester script exercise (from MIT):**
a. Create the file `/tmp/missing/semester` with this content (write it using `echo` and `>` / `>>`, not a text editor):
```
#!/bin/sh
curl --head --silent https://missing.csail.mit.edu
```
b. Try running `./semester`. What error do you get? Use `ls -l` to understand why.
c. Run it with `sh semester` instead. Why does this work when `./semester` doesn't?
d. Use `man chmod` to understand file permissions. Then make the file executable:
```bash
chmod +x semester
./semester
```
e. Extract just the `last-modified` date from the output and save it to a file:
```bash
./semester | grep last-modified > ~/last-modified.txt
cat ~/last-modified.txt
```
**Bonus:** Why does `#!/bin/sh` at the top of the file matter? Look up "shebang line" if you're curious.
***
### Summary
| Topic | Core Idea |
| ------------------- | --------------------------------------------------------------------- |
| **The Shell** | A programmable text interface to your OS — more powerful than any GUI |
| **$PATH** | Controls where the shell looks for programs by name |
| **Paths** | Absolute paths start at `/`; relative paths start at `.` |
| **Navigation** | `pwd`, `cd`, `ls` are your compass and map |
| **File operations** | `mv`, `cp`, `mkdir`, `cat`, `touch` for basic manipulation |
| **Permissions** | `rwx` for owner/group/others; `ls -l` to inspect |
| **Redirection** | `>` and `<` rewire stdout/stdin to files |
| **Pipes** | `\|` chains program output into program input |
| **sudo** | Elevates a *program* to root — shell operators are still unprivileged |
| **man** | Your built-in reference for every command |
#### Most Important Commands from This Lecture
```
pwd – where am I?
cd – go somewhere
ls – what's here?
echo – print something
cat – read/concatenate files
mv – move/rename
cp – copy
mkdir – make directory
man – read the manual
which – find a binary
sudo – run as root
tee – split output to file + stdout
```
#### What's Next
In **Lecture 2 – Command-line Environment**, you'll learn how to write shell scripts, use control flow (if/for/while), work with shell functions, and use powerful tools like `find`, `grep`, and `xargs` to automate real workflows.
***
*Source:* [*MIT Missing Semester – Course Overview + The Shell*](https://missing.csail.mit.edu/2020/course-shell/) *Licensed under* [*CC BY-NC-SA 4.0*](https://creativecommons.org/licenses/by-nc-sa/4.0)
---
# Agent Instructions: Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter:
```
GET https://shankar-lab.gitbook.io/mylearning/the-missing-semester-of-your-cs-education/lecture-1-course-overview-and-the-shell.md?ask=
```
The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.