diff --git a/intro-unix-command-line/Intro_to_Unix_Part_1.Rmd b/intro-unix-command-line/Intro_to_Unix_Part_1.Rmd new file mode 100644 index 0000000..9a6545c --- /dev/null +++ b/intro-unix-command-line/Intro_to_Unix_Part_1.Rmd @@ -0,0 +1,582 @@ +--- +title: "Introduction to Unix Command-line - Part 1" +author: "Natalie Elphick" +date: "April 17th" +output: + revealjs::revealjs_presentation: + css: style.css +--- + + +```{r, setup, include=FALSE} +knitr::opts_knit$set(root.dir = './unix_workshop_2023') +``` + + +## + +
*Press the ? key for tips on navigating these slides*
+ + +## Introductions + +**Natalie Elphick** +Bioinformatician I + +
+ + +**Yihang Xin (TA)** +Software Engineer II + + + + + + + + +# The Unix Command-line + + +## What is Unix? + +- A family of operating systems that date back to 1970s +- Designed to run on different types of computer hardware +- macOS and Linux are descendants of Unix +- Often used in industry and scientific research + + +## What is the Command-line? + +- A text-based interface for interacting with the operating system +- Execute commands to perform various tasks like: + - Navigating the file system + - Editing files +- Command-line interfaces (CLIs) are how users interacted with computers before graphical interfaces like Windows + +## Advantages of using CLIs + +- Allows for automation and batch processing of data +- Provides more control and flexibility over data manipulation +- Interact with High-Performance Computing clusters (HPCs) like Wynton +- Speed + + +## The Terminal + +- The software that provides access to the CLI +- Open your terminal + + - Mac: press Cmd + Space and type "terminal" + - Linux: press Ctrl+Alt+T + - Windows: open the Ubuntu app + +## Download the workshop materials + +- Copy and paste each of these into your terminal, press enter after each one: + +```{r, engine='bash', eval=FALSE, results='markup', highlight=FALSE, comment=NA, echo = TRUE} +mkdir unix_workshop +``` + + +```{r, engine='bash', eval=FALSE, results='markup', highlight=FALSE, comment=NA, echo = TRUE} +cd unix_workshop +``` + + +```{r, engine='bash', eval=FALSE, results="hide", highlight=FALSE, comment=NA, echo = TRUE} +curl -L -o unix_workshop_2023.tar.gz https://www.dropbox.com/s/smb12au2y82jmvq/unix_workshop_2023.tar.gz?dl=0 +``` + + +```{r, engine='bash', eval=FALSE, results='markup', highlight=FALSE, comment=NA, echo = TRUE} +tar -xzf unix_workshop_2023.tar.gz +``` + + +```{r, engine='bash', eval=FALSE, results='markup', highlight=FALSE, comment=NA, echo = TRUE} +cd unix_workshop_2023 +``` + + + + +## The shell + +- A shell is a specific type of CLI that provides access to the OS as a whole +- There are several different Unix shells + - Bash : The most widely used and the default shell on most Linux systems + - Zsh : an extended version of Bash and is now the default on newer versions of macOS + +## The shell + +- Check which shell you are using by typing: + +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA, echo = TRUE} +echo $0 +``` + +Both bash and zsh should be able to run all of the commands in this workshop. + + +## Commands + +Shell commands are basic instructions used to perform specific tasks. + +
+ +Basic structure of commands: +`command_name -[option(s)] [argument(s)]` + + +Example: +```{r, engine='bash', eval=FALSE, results='markup', highlight=FALSE, comment=NA, echo = TRUE} +ls -lah part_1 +``` + +Here we are providing multiple options to the `ls` command and the directory **part_1** as an argument + +## man: pull up the manual page for a command + + + +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA, echo = FALSE} +echo "man echo" +``` + +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA, echo = FALSE} +man -P cat echo | col -b +``` + +## Manual pages + +Use the arrow keys to navigate the manual and press `q` to close it + +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA, echo = FALSE} +man -P cat man | col -b +``` + +## echo: print a string or value of a variable + +- **Variables** : a named container that holds a value or data +- **Strings** : sequence of characters +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA, echo = TRUE} +message="Hello, World!" +echo $message +``` + +Here, we assign the string "Hello, World!" to the variable `message` and use echo to print its value. + +## history: list previously run commands +- You can also cycle through previously run commands using the up and down arrow keys +- By default bash stores the last 500 commands, zsh stores the last 1000 + +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA, echo = TRUE} +history +``` + +- Use the command `clear` to clear the output from the terminal + + +# Navigating the File System + +## The File System + +![Unix File system](materials/UNIX_filesystem.png) + + +## Paths + + +* Root directory  /  +* Current working directory  .  +* Directory above the current one  ..  +* User home directory  ~  +* Absolute: The path starting from root + +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA, echo = FALSE} + +echo "/data/file1.txt" + +``` + +* Relative: Path from the current directory + +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA, echo = FALSE} + +echo "file1.txt" +echo "./file1.txt" + +``` + + +## +
What is the absolute path to **file2.txt**?
+ +![Unix File system](materials/UNIX_filesystem.png) + +## +
What is the realative path to **file1.txt** if the working directory is **/home/user** ?
+![Unix File system](materials/UNIX_filesystem.png) + +## pwd: print working directory + +```{r, engine='bash', eval=TRUE, results="hide", highlight=FALSE, comment=NA} +pwd +``` + +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA, echo=FALSE} +echo "/Users/your_username/unix_workshop_2023" +``` + +- The default working directory when you log in or open a terminal is your user home directory  ~  + + +## ls: list contents of a directory {.small-bullets} + + +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA} +ls . +``` + + +- `-l ` show more information (file permissions and size) +- `-a ` show all (hidden files) +- `-h ` file sizes in human readable format (e.g., 1K, 2G) + + +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA} +ls -lah part_1 +``` + +## cd: move to a directory + +```{r, engine='bash', eval=FALSE, results='markup', highlight=FALSE, comment=NA} +cd unix_workshop_2023/part_1 +ls -l +``` + +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA,echo=FALSE} +cd part_1 +ls -l +``` + +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA, echo=FALSE} +echo "cd .." +echo "ls -l" +``` + + +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA, echo=FALSE} +ls -l +``` + + + + +# Creating and Altering Files + + +## File Permissions + +![File Permissions](materials/File permissions.png) + +- Permissions for a file or folder are represented by 10 characters +- Each group of 3 represents the permissions for different users: + - Owner of the file/folder + - Group the owns the file + - Others - everyone else + +## File Permissions + +![File Permissions](materials/File permissions.png) + +- There are three types of permissions: + - r - read/view the contents of a file/folder + - w - write + - x - execute the file or access a directory + +## touch: create an empty file +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA} +touch new_file.txt +``` + +- If the file exists, it will update the time stamp + + +## mkdir: make a directory +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA} +mkdir new_directory +``` +- `-p ` make parent directories if they don't exist + +## mv: move a file or folder +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA} +mv new_file.txt new_directory +``` + +- Also used to rename files/folders + +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA} +mv new_directory/new_file.txt new_directory/new_file1.txt +``` + +## cp: copy a file or folder +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA} +cp new_directory/new_file1.txt new_directory/new_file2.txt +``` +* `-r ` to copy a folder (recursive) + + + +## rm: remove a file permanently + +- This command should always be used with care since the files it is used on **cannot be recovered** + +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA} +rm new_directory/new_file2.txt +``` + +## du: check the size of a file or folder + +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA} +du -h */* +``` +-h - Displays the output in human readable format + +## Check-in + +If you are following along with the commands we have run so far, this is the file structure you should have: +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA} +ls ./* +``` + +- `*` is a wildcard so `ls` will list and directories in the current one + +## Text editors + +- Command-line text editors provide lots of key board shortcuts to navigate and alter files +- Some commonly used ones are: + - Vim : Feature rich, steep learning curve + - nano : Simple and user friendly + +## nano {.big-picture} + +```{r, engine='bash', eval=FALSE, results="hide", highlight=FALSE, comment=NA} +nano new_directory/new_file1.txt +``` +![nano](materials/nano.png) + +## Shortcuts for nano {.less-small-bullets} + + +- Ctrl + X :  Exit nano +- Ctrl + O :  Save the file (write Out) +- Ctrl + W :  Search for a string or regular expression +- Ctrl + K :  Cut (remove) the current line or selection +- Ctrl + U :  Uncut (paste) the most recently cut text +- Ctrl + A :  Move the cursor to the start of the current line +- Ctrl + E :  Move the cursor to the end of the current line +- Ctrl + G :  Show the help menu + +# Installing Software + +## Package managers + +- Used to install and manage software + +- macOS + - [homebrew](https://brew.sh/) - not included with the OS and needs to be installed +- WSL/Linux + - [apt-get](https://help.ubuntu.com/community/AptGet/Howto) - included with Ubuntu + +We will not install any software in this workshop but these are how you would +access additional software/commands. + + + +# Downloading Files + + +## curl: download files from the internet + +- We used curl at the begining download the materials +- curl supports multiple protocols but the most commonly used one is HTTPS + +```{r, engine='bash', eval=TRUE, results="hide", highlight=FALSE, comment=NA, echo = TRUE} +curl -o part_2/homo_sapiens.refseq.tsv.gz https://ftp.ensembl.org/pub/current_tsv/homo_sapiens/Homo_sapiens.GRCh38.109.refseq.tsv.gz +``` + +- `-o` gives the output file name and location + +## Other file transfer tools + +- wget : like curl, supports fewer protocols and is not always installed +- scp (secure copy) : used to encrypt and transfer files using SSH protocol + - Commonly used to transfer files on HPCs (Wynton) +Example: + +```{r, engine='bash', eval=FALSE, results="hide", highlight=FALSE, comment=NA, echo = TRUE} +scp /path/to/local/file user@remote.host:/path/to/remote/directory +``` + +# Searching Files and Combining Commands + +## grep: searching files with regular expressions + +- Search the contents of the input file and returns the lines that have a match + +- **Regular Expressions** : sequence of characters that forms a search pattern + +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA, echo = TRUE} +grep "7" part_1/list_numbers.tsv +``` + +## grep options + +- -i : ignore case +- -v : invert match +- -r : recursively searches in all files and subdirectories of a directory +- -c : counts the number of matches found in each file + +## Regular Expressions + +- For a more extensive overview of regular expressions click [here](https://v4.software-carpentry.org/regexp/index.html) +- Some basic ones are: + - " ^ " Matches the beginning of a line + - " . " Matches any single character except newline + - " $ " Matches the end of a line + + +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA, echo = TRUE} +grep "3$" part_1/list_numbers.tsv +``` + +## head/tail: view the first or last n lines of a file + +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA, echo = TRUE} +head -n 1 part_1/list_numbers.tsv +``` + +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA, echo = TRUE} +tail -n 1 part_1/list_numbers.tsv +``` + +- The default n is 10 +- Useful for getting a look at the format of a file + +## cat: print the contents of a file + +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA, echo = TRUE} +cat part_1/list_numbers.tsv +``` + + +## cut: get specific columns from a file + +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA, echo = TRUE} +cut -f 1-3,6 part_1/list_numbers.tsv +``` + +- -f : fields that should be returned +- -d : delimiter - character that the columns are separated by + +By default `cut` expects columns to be separated by tab characters. + + +## Combining Commands + +- Pipes "|" connect one command to another +- The output of the previous command is used as the input for the next one +- Chaining commands allows you to do complex operations on text streams + +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA, echo = TRUE} +grep "3$" part_1/list_numbers.tsv | cut -f 1-3 +``` + +## Output to a file + +- The output of any command can be written to a file with the " > " character + +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA, echo = TRUE} +grep "3$" part_1/list_numbers.tsv | cut -f 1-3 > part_1/subset_list_numbers.tsv +``` + + + +## tr: translate or substitute characters + +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA, echo = TRUE} +cat part_1/list_numbers.tsv | tr "\t" "," > part_1/list_numbers.csv +cat part_1/list_numbers.csv +``` + +## Check-in + +If you followed along with the commands we have run so far, you should have this directory structure: + +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA} +ls ./* +``` + +# End of Part 1 + +## Other helpful commands {.small-bullets} + +- `chmod` : Change the permissions of a file or directory +- `chown` : Change the owner of a file or directory +- `df` : Display information about disk usage and available space +- `ps` : Display information about running processes +- `kill` : Stop a running process +- `less` : View the contents of a file one page at a time +- `date` : prints the date and time +- `curl wttr.in` : check the weather + +## Survey + +- Please take some time to fill out the workshop survey: +https://www.surveymonkey.com/r/DY7K5ZY + + +## Additional learning materials + +- Software carpentry provides a self paced course: + - [The Unix Shell](https://swcarpentry.github.io/shell-novice/) + +- Free online books: + - [The Unix Workbench](https://seankross.com/the-unix-workbench/index.html) + - [The Linux Command Line](http://linuxcommand.org/tlcl.php) + + + + +## Upcoming Data Science Training Program Workshops + +[Linear Mixed Effects Modeling](https://gladstone.org/index.php/events/linear-mixed-effects-modeling-0) +April 24-April 25, 2023 10:00am-12:00pm PDT + +[Machine Learning](https://gladstone.org/index.php/events/machine-learning) +April 28, 2023 10:00am-12:00pm PDT + +[Advanced Cytoscape Automation](https://gladstone.org/index.php/events/advanced-cytoscape-automation-2) +May 2, 2023 1:00-4:00pm PDT + +[Introduction to RNA-Seq Analysis](https://gladstone.org/index.php/events/introduction-rna-seq-analysis-4) +May 15-May 16, 2023 9:00am-12:00pm PDT + + + + + + + + + +```{r, engine='bash', eval=TRUE, results='markup', highlight=FALSE, comment=NA, echo=FALSE} +rm -r new_directory +rm part_2/homo_sapiens.refseq.tsv.gz +rm part_1/subset_list_numbers.tsv +rm part_1/list_numbers.csv +``` \ No newline at end of file diff --git a/intro-unix-command-line/Intro_to_Unix_Part_1.html b/intro-unix-command-line/Intro_to_Unix_Part_1.html new file mode 100644 index 0000000..be25f45 --- /dev/null +++ b/intro-unix-command-line/Intro_to_Unix_Part_1.html @@ -0,0 +1,1500 @@ + + + + + + + Introduction to Unix Command-line - Part 1 + + + + + + + + + + + + + + + + + +
+
+ +
+

Introduction to Unix Command-line - Part 1

+

Natalie Elphick

+

April 17th

+
+ +
+

+
+Press the ? key for tips on navigating these slides +
+
+
+

Introductions

+

Natalie Elphick
+Bioinformatician I

+


+

Yihang Xin (TA)
+Software Engineer II

+
+
+
+

The Unix Command-line

+ +
+
+

What is Unix?

+
    +
  • A family of operating systems that date back to 1970s
    • +
  • Designed to run on different types of computer hardware
    • +
  • macOS and Linux are descendants of Unix
    • +
  • Often used in industry and scientific research
    • +
+
+
+

What is the Command-line?

+
    +
  • A text-based interface for interacting with the operating +system
    • +
  • Execute commands to perform various tasks like: +
      +
    • Navigating the file system
      • +
    • Editing files
      • +
    • +
  • Command-line interfaces (CLIs) are how users interacted with +computers before graphical interfaces like Windows
    • +
+
+
+

Advantages of using CLIs

+
    +
  • Allows for automation and batch processing of data
    • +
  • Provides more control and flexibility over data manipulation
    • +
  • Interact with High-Performance Computing clusters (HPCs) like +Wynton
    • +
  • Speed
    • +
+
+
+

The Terminal

+
    +

  • The software that provides access to the CLI

    • +

  • Open your terminal

    +
      +
    • Mac: press Cmd + Space and type “terminal”
      • +
    • Linux: press Ctrl+Alt+T
      +
      • +
    • Windows: open the Ubuntu app
      • +
    • +
+
+
+

Download the workshop materials

+
    +
  • Copy and paste each of these into your terminal, press enter after +each one:
    • +
+
mkdir unix_workshop
+
cd unix_workshop
+
curl -L -o unix_workshop_2023.tar.gz https://www.dropbox.com/s/smb12au2y82jmvq/unix_workshop_2023.tar.gz?dl=0
+
tar -xzf unix_workshop_2023.tar.gz
+
cd unix_workshop_2023
+
+
+

The shell

+
    +
  • A shell is a specific type of CLI that provides access to the OS as +a whole
    • +
  • There are several different Unix shells +
      +
    • Bash : The most widely used and the default shell on most Linux +systems
      • +
    • Zsh : an extended version of Bash and is now the default on newer +versions of macOS
      • +
    • +
+
+
+

The shell

+
    +
  • Check which shell you are using by typing:
    • +
+
echo $0
+
bash
+

Both bash and zsh should be able to run all of the commands in this +workshop.

+
+
+

Commands

+

Shell commands are basic instructions used to perform specific +tasks.

+


+

Basic structure of commands:
+command_name -[option(s)] [argument(s)]

+

Example:

+
ls -lah part_1
+

Here we are providing multiple options to the ls command +and the directory part_1 as an argument

+
+
+

man: pull up the manual page for a command

+
man echo
+
ECHO(1)             General Commands Manual            ECHO(1)
+
+NAME
+     echo – write arguments to the standard output
+
+SYNOPSIS
+     echo [-n] [string ...]
+
+DESCRIPTION
+     The echo utility writes any specified operands, separated by single blank
+     (‘ ’) characters and followed by a newline (‘\n’) character, to the
+     standard output.
+
+     The following option is available:
+
+     -n    Do not print the trailing newline character.  This may also be
+       achieved by appending ‘\c’ to the end of the string, as is done by
+       iBCS2 compatible systems.  Note that this option as well as the
+       effect of ‘\c’ are implementation-defined in IEEE Std 1003.1-2001
+       (“POSIX.1”) as amended by Cor. 1-2002.  Applications aiming for
+       maximum portability are strongly encouraged to use printf(1) to
+       suppress the newline character.
+
+     Some shells may provide a builtin echo command which is similar or
+     identical to this utility.  Most notably, the builtin echo in sh(1) does
+     not accept the -n option.  Consult the builtin(1) manual page.
+
+EXIT STATUS
+     The echo utility exits 0 on success, and >0 if an error occurs.
+
+SEE ALSO
+     builtin(1), csh(1), printf(1), sh(1)
+
+STANDARDS
+     The echo utility conforms to IEEE Std 1003.1-2001 (“POSIX.1”) as amended
+     by Cor. 1-2002.
+
+macOS 13.2          April 12, 2003              macOS 13.2
+
+
+

Manual pages

+

Use the arrow keys to navigate the manual and press q to +close it

+
MAN(1)              General Commands Manual         MAN(1)
+
+NAME
+     man, apropos, whatis – display online manual documentation pages
+
+SYNOPSIS
+     man [-adho] [-t | -w] [-M manpath] [-P pager] [-S mansect]
+     [-m arch[:machine]] [-p [eprtv]] [mansect] page ...
+
+     man -f [-d] [-M manpath] [-P pager] [-S mansect] keyword ...
+     whatis [-d] [-s mansect] keyword ...
+
+     man -k [-d] [-M manpath] [-P pager] [-S mansect] keyword ...
+     apropos [-d] [-s mansect] keyword ...
+
+DESCRIPTION
+     The man utility finds and displays online manual documentation pages.  If
+     mansect is provided, man restricts the search to the specific section of
+     the manual.
+
+     The sections of the manual are:
+       1.   General Commands Manual
+       2.   System Calls Manual
+       3.   Library Functions Manual
+       4.   Kernel Interfaces Manual
+       5.   File Formats Manual
+       6.   Games Manual
+       7.   Miscellaneous Information Manual
+       8.   System Manager's Manual
+       9.   Kernel Developer's Manual
+
+     Options that man understands:
+
+     -M manpath
+         Forces a specific colon separated manual path instead of the
+         default search path.  See manpath(1).  Overrides the MANPATH
+         environment variable.
+
+     -P pager
+         Use specified pager.  Defaults to “less -sR” if color support is
+         enabled, or “less -s”.  Overrides the MANPAGER environment
+         variable, which in turn overrides the PAGER environment variable.
+
+     -S mansect
+         Restricts manual sections searched to the specified colon
+         delimited list.  Defaults to “1:8:2:3:3lua:n:4:5:6:7:9:l”.
+         Overrides the MANSECT environment variable.
+
+     -a      Display all manual pages instead of just the first found for each
+         page argument.
+
+     -d      Print extra debugging information.  Repeat for increased
+         verbosity.  Does not display the manual page.
+
+     -f      Emulate whatis(1).  Note that only a subset of options will have
+         any effect when man is invoked in this mode.  See the below
+         description of whatis options for details.
+
+     -h      Display short help message and exit.
+
+     -k      Emulate apropos(1).  Note that only a subset of options will have
+         any effect when man is invoked in this mode.  See the below
+         description of apropos options for details.
+
+     -m arch[:machine]
+         Override the default architecture and machine settings allowing
+         lookup of other platform specific manual pages.  This option is
+         accepted, but not implemented, on macOS.
+
+     -o      Force use of non-localized manual pages.  See IMPLEMENTATION
+         NOTES for how locale specific searches work.  Overrides the
+         LC_ALL, LC_CTYPE, and LANG environment variables.
+
+     -p [eprtv]
+         Use the list of given preprocessors before running nroff(1) or
+         troff(1).  Valid preprocessors arguments:
+
+         e       eqn(1)
+         p       pic(1)
+         r       refer(1)
+         t       tbl(1)
+         v       vgrind(1)
+
+         Overrides the MANROFFSEQ environment variable.
+
+     -t      Send manual page source through troff(1) allowing transformation
+         of the manual pages to other formats.
+
+     -w      Display the location of the manual page instead of the contents
+         of the manual page.
+
+     Options that apropos and whatis understand:
+
+     -d      Same as the -d option for man.
+
+     -s      Same as the -S option for man.
+
+     When man is operated in apropos or whatis emulation mode, only a subset
+     of its options will be honored.  Specifically, -d, -M, -P, and -S have
+     equivalent functionality in the apropos and whatis implementation
+     provided.  The MANPATH, MANSECT, and MANPAGER environment variables will
+     similarly be honored.
+
+IMPLEMENTATION NOTES
+   Locale Specific Searches
+     The man utility supports manual pages in different locales.  The search
+     behavior is dictated by the first of three environment variables with a
+     nonempty string: LC_ALL, LC_CTYPE, or LANG.  If set, man will search for
+     locale specific manual pages using the following logic:
+
+       lang_country.charset
+       lang.charset
+       en.charset
+
+     For example, if LC_ALL is set to “ja_JP.eucJP”, man will search the
+     following paths when considering section 1 manual pages in
+     /usr/share/man:
+
+       /usr/share/man/ja_JP.eucJP/man1
+       /usr/share/man/ja.eucJP/man1
+       /usr/share/man/en.eucJP/man1
+       /usr/share/man/man1
+
+   Displaying Specific Manual Files
+     The man utility also supports displaying a specific manual page if passed
+     a path to the file as long as it contains a ‘/’ character.
+
+ENVIRONMENT
+     The following environment variables affect the execution of man:
+
+     LC_ALL, LC_CTYPE, LANG
+             Used to find locale specific manual pages.  Valid values
+             can be found by running the locale(1) command.  See
+             IMPLEMENTATION NOTES for details.  Influenced by the -o
+             option.
+
+     MACHINE_ARCH, MACHINE
+             Used to find platform specific manual pages.  If unset,
+             the output of “sysctl hw.machine_arch” and “sysctl
+             hw.machine” is used respectively.  See IMPLEMENTATION
+             NOTES for details.  Corresponds to the -m option.
+
+     MANPATH         The standard search path used by man(1) may be changed by
+             specifying a path in the MANPATH environment variable.
+             Invalid paths, or paths without manual databases, are
+             ignored.  Overridden by -M.  If MANPATH begins with a
+             colon, it is appended to the default list; if it ends
+             with a colon, it is prepended to the default list; or if
+             it contains two adjacent colons, the standard search path
+             is inserted between the colons.  If none of these
+             conditions are met, it overrides the standard search
+             path.
+
+     MANROFFSEQ      Used to determine the preprocessors for the manual source
+             before running nroff(1) or troff(1).  If unset, defaults
+             to tbl(1).  Corresponds to the -p option.
+
+     MANSECT         Restricts manual sections searched to the specified colon
+             delimited list.  Corresponds to the -S option.
+
+     MANWIDTH        If set to a numeric value, used as the width manpages
+             should be displayed.  Otherwise, if set to a special
+             value “tty”, and output is to a terminal, the pages may
+             be displayed over the whole width of the screen.
+
+     MANCOLOR        If set, enables color support.
+
+     MANPAGER        Program used to display files.
+
+             If unset, and color support is enabled, “less -sR” is
+             used.
+
+             If unset, and color support is disabled, then PAGER is
+             used.  If that has no value either, “less -s” is used.
+
+FILES
+     /etc/man.conf
+         System configuration file.
+     /usr/local/etc/man.d/*.conf
+         Local configuration files.
+
+EXIT STATUS
+     The man utility exits 0 on success, and >0 if an error occurs.
+
+EXAMPLES
+     Show the manual page for stat(2):
+
+       $ man 2 stat
+
+     Show all manual pages for ‘stat’.
+
+       $ man -a stat
+
+     List manual pages which match the regular expression either in the title
+     or in the body:
+
+       $ man -k '\<copy\>.*archive'
+
+     Show the manual page for ls(1) and use cat(1) as pager:
+
+       $ man -P cat ls
+
+     Show the location of the ls(1) manual page:
+
+       $ man -w ls
+
+SEE ALSO
+     apropos(1), intro(1), mandoc(1), manpath(1), whatis(1), intro(2),
+     intro(3), intro(3lua), intro(4), intro(5), man.conf(5), intro(6),
+     intro(7), mdoc(7), intro(8), intro(9)
+
+macOS 13.2          January 9, 2021             macOS 13.2
+
+
+

echo: print a string or value of a variable

+
    +
  • Variables : a named container that holds a value or +data
    • +
  • Strings : sequence of characters
    • +
+
message="Hello, World!"
+echo $message
+
Hello, World!
+

Here, we assign the string “Hello, World!” to the variable +message and use echo to print its value.

+
+
+

history: list previously run commands

+
    +
  • You can also cycle through previously run commands using the up and +down arrow keys
    • +
  • By default bash stores the last 500 commands, zsh stores the last +1000
    • +
+
history
+
    +
  • Use the command clear to clear the output from the +terminal
    • +
+
+
+ +
+

The File System

+

Unix File system

+
+
+

Paths

+
    +
  • Root directory + / 
    • +
  • Current working directory  . 
    +
    • +
  • Directory above the current one  .. 
    • +
  • User home directory  ~ 
    • +
  • Absolute: The path starting from root
    • +
+
/data/file1.txt
+
    +
  • Relative: Path from the current directory
    • +
+
file1.txt
+./file1.txt
+
+
+

+
+What is the absolute path to file2.txt? +
+

Unix File system

+
+
+

+
+What is the realative path to file1.txt if the working +directory is /home/user ? +
+

Unix File system

+
+
+

pwd: print working directory

+
pwd
+
/Users/your_username/unix_workshop_2023
+
    +
  • The default working directory when you log in or open a terminal is +your user home directory  ~ 
    • +
+
+
+

ls: list contents of a directory

+
ls .
+
part_1
+part_2
+
    +
  • -l show more information (file permissions and +size)
    • +
  • -a show all (hidden files)
    • +
  • -h file sizes in human readable format (e.g., 1K, +2G)
    • +
+
ls -lah part_1
+
total 8
+drwx---rw-@ 4 nelphick  staff   128B Apr 14 11:25 .
+drwxr-xr-x@ 5 nelphick  staff   160B Apr 14 11:25 ..
+-rw-r--rw-@ 1 nelphick  staff     0B Apr 11 16:29 .hidden_file.txt
+-rw-r--rw-@ 1 nelphick  staff    60B Apr 12 15:40 list_numbers.tsv
+
+
+

cd: move to a directory

+
cd unix_workshop_2023/part_1
+ls -l
+
total 8
+-rw-r--rw-@ 1 nelphick  staff  60 Apr 12 15:40 list_numbers.tsv
+
cd ..
+ls -l
+
total 0
+drwx---rw-@ 4 nelphick  staff  128 Apr 14 11:25 part_1
+drwxr-xr-x@ 2 nelphick  staff   64 Apr 14 11:25 part_2
+
+
+
+

Creating and Altering Files

+ +
+
+

File Permissions

+

File Permissions

+
    +
  • Permissions for a file or folder are represented by 10 +characters
    • +
  • Each group of 3 represents the permissions for different users: +
      +
    • Owner of the file/folder
      • +
    • Group the owns the file
      • +
    • Others - everyone else
      • +
    • +
+
+
+

File Permissions

+

File Permissions

+
    +
  • There are three types of permissions: +
      +
    • r - read/view the contents of a file/folder
      • +
    • w - write
      • +
    • x - execute the file or access a directory
      • +
    • +
+
+
+

touch: create an empty file

+
touch new_file.txt
+
    +
  • If the file exists, it will update the time stamp
    • +
+
+
+

mkdir: make a directory

+
mkdir new_directory
+
    +
  • -p make parent directories if they don’t exist
    • +
+
+
+

mv: move a file or folder

+
mv new_file.txt new_directory
+
    +
  • Also used to rename files/folders
    • +
+
mv new_directory/new_file.txt new_directory/new_file1.txt
+
+
+

cp: copy a file or folder

+
cp new_directory/new_file1.txt new_directory/new_file2.txt
+
    +
  • -r to copy a folder (recursive)
    • +
+
+
+

rm: remove a file permanently

+
    +
  • This command should always be used with care since the files it is +used on cannot be recovered
    • +
+
rm new_directory/new_file2.txt
+
+
+

du: check the size of a file or folder

+
du -h */*
+
  0B    new_directory/new_file1.txt
+4.0K    part_1/list_numbers.tsv
+

-h - Displays the output in human readable format

+
+
+

Check-in

+

If you are following along with the commands we have run so far, this +is the file structure you should have:

+
ls ./*
+
./new_directory:
+new_file1.txt
+
+./part_1:
+list_numbers.tsv
+
+./part_2:
+
    +
  • * is a wildcard so ls will list and +directories in the current one
    • +
+
+
+

Text editors

+
    +
  • Command-line text editors provide lots of key board shortcuts to +navigate and alter files
    • +
  • Some commonly used ones are: +
      +
    • Vim : Feature rich, steep learning curve
      • +
    • nano : Simple and user friendly
      • +
    • +
+
+
+

nano

+
nano new_directory/new_file1.txt
+

nano

+
+
+

Shortcuts for nano

+
    +
  • Ctrl + X :  Exit nano
    • +
  • Ctrl + O :  Save the file (write Out)
    • +
  • Ctrl + W :  Search for a string or regular expression
    • +
  • Ctrl + K :  Cut (remove) the current line or selection
    • +
  • Ctrl + U :  Uncut (paste) the most recently cut text
    • +
  • Ctrl + A :  Move the cursor to the start of the current line
    • +
  • Ctrl + E :  Move the cursor to the end of the current line
    • +
  • Ctrl + G :  Show the help menu
    • +
+
+
+
+

Installing Software

+ +
+
+

Package managers

+
    +

  • Used to install and manage software

    • +

  • macOS

    +
      +
    • homebrew - not included with the OS +and needs to be installed
      • +
    • +

  • WSL/Linux

    +
      +
    • apt-get +- included with Ubuntu
      • +
    • +
+

We will not install any software in this workshop but these are how +you would access additional software/commands.

+
+
+
+

Downloading Files

+ +
+
+

curl: download files from the internet

+
    +
  • We used curl at the begining download the materials
    • +
  • curl supports multiple protocols but the most commonly used one is +HTTPS
    • +
+
curl -o part_2/homo_sapiens.refseq.tsv.gz https://ftp.ensembl.org/pub/current_tsv/homo_sapiens/Homo_sapiens.GRCh38.109.refseq.tsv.gz
+
    +
  • -o gives the output file name and location
    • +
+
+
+

Other file transfer tools

+
    +
  • wget : like curl, supports fewer protocols and is not always +installed
    • +
  • scp (secure copy) : used to encrypt and transfer files using SSH +protocol +
      +
    • Commonly used to transfer files on HPCs (Wynton) Example:
      • +
    • +
+
scp /path/to/local/file user@remote.host:/path/to/remote/directory
+
+
+
+

Searching Files and Combining Commands

+ +
+
+

grep: searching files with regular expressions

+
    +

  • Search the contents of the input file and returns the lines that +have a match

    • +

  • Regular Expressions : sequence of characters +that forms a search pattern

    • +
+
grep "7" part_1/list_numbers.tsv
+
7   8   52  13  6   42  79  1
+13  4   9   82  67  71  93  3
+
+
+

grep options

+
    +
  • -i : ignore case
    • +
  • -v : invert match
    • +
  • -r : recursively searches in all files and subdirectories of a +directory
    • +
  • -c : counts the number of matches found in each file
    • +
+
+
+

Regular Expressions

+
    +
  • For a more extensive overview of regular expressions click here
    • +
  • Some basic ones are: +
      +
    • ” ^ ” Matches the beginning of a line
      • +
    • ” . ” Matches any single character except newline
      • +
    • ” $ ” Matches the end of a line
      • +
    • +
+
grep "3$" part_1/list_numbers.tsv
+
1   3   6   10  11  22  0   3
+13  4   9   82  67  71  93  3
+
+
+

head/tail: view the first or last n lines of a file

+
head -n 1 part_1/list_numbers.tsv
+
1   3   6   10  11  22  0   3
+
tail -n 1 part_1/list_numbers.tsv
+
13  4   9   82  67  71  93  3
+
    +
  • The default n is 10
    • +
  • Useful for getting a look at the format of a file
    • +
+
+
+

cat: print the contents of a file

+
cat part_1/list_numbers.tsv
+
1   3   6   10  11  22  0   3
+7   8   52  13  6   42  79  1
+13  4   9   82  67  71  93  3
+
+
+

cut: get specific columns from a file

+
cut -f 1-3,6 part_1/list_numbers.tsv
+
1   3   6   22
+7   8   52  42
+13  4   9   71
+
    +
  • -f : fields that should be returned
    • +
  • -d : delimiter - character that the columns are separated by
    • +
+

By default cut expects columns to be separated by tab +characters.

+
+
+

Combining Commands

+
    +
  • Pipes “|” connect one command to another
    • +
  • The output of the previous command is used as the input for the next +one
    • +
  • Chaining commands allows you to do complex operations on text +streams
    • +
+
grep "3$" part_1/list_numbers.tsv | cut -f 1-3
+
1   3   6
+13  4   9
+
+
+

Output to a file

+
    +
  • The output of any command can be written to a file with the ” > ” +character
    • +
+
grep "3$" part_1/list_numbers.tsv | cut -f 1-3 > part_1/subset_list_numbers.tsv
+
+
+

tr: translate or substitute characters

+
cat part_1/list_numbers.tsv | tr "\t" "," > part_1/list_numbers.csv
+cat part_1/list_numbers.csv
+
1,3,6,10,11,22,0,3
+7,8,52,13,6,42,79,1
+13,4,9,82,67,71,93,3
+
+
+

Check-in

+

If you followed along with the commands we have run so far, you +should have this directory structure:

+
ls ./*
+
./new_directory:
+new_file1.txt
+
+./part_1:
+list_numbers.csv
+list_numbers.tsv
+subset_list_numbers.tsv
+
+./part_2:
+homo_sapiens.refseq.tsv.gz
+
+
+
+

End of Part 1

+ +
+
+

Other helpful commands

+
    +
  • chmod : Change the permissions of a file or +directory
    • +
  • chown : Change the owner of a file or directory
    • +
  • df : Display information about disk usage and available +space
    • +
  • ps : Display information about running processes
    • +
  • kill : Stop a running process
    • +
  • less : View the contents of a file one page at a +time
    • +
  • date : prints the date and time
    • +
  • curl wttr.in : check the weather
    • +
+
+
+

Survey

+ +
+
+

Additional learning materials

+ +
+
+

Upcoming Data Science Training Program Workshops

+

Linear +Mixed Effects Modeling
+April 24-April 25, 2023 10:00am-12:00pm PDT

+

Machine +Learning
+April 28, 2023 10:00am-12:00pm PDT

+

Advanced +Cytoscape Automation
+May 2, 2023 1:00-4:00pm PDT

+

Introduction +to RNA-Seq Analysis
+May 15-May 16, 2023 9:00am-12:00pm PDT

+
+
+
+ + + + + + + + + + + + + + diff --git a/intro-unix-command-line/materials/File permissions.png b/intro-unix-command-line/materials/File permissions.png new file mode 100644 index 0000000..628aca5 Binary files /dev/null and b/intro-unix-command-line/materials/File permissions.png differ diff --git a/intro-unix-command-line/materials/UNIX_filesystem.png b/intro-unix-command-line/materials/UNIX_filesystem.png new file mode 100644 index 0000000..1968373 Binary files /dev/null and b/intro-unix-command-line/materials/UNIX_filesystem.png differ diff --git a/intro-unix-command-line/materials/nano.png b/intro-unix-command-line/materials/nano.png new file mode 100644 index 0000000..c3d3159 Binary files /dev/null and b/intro-unix-command-line/materials/nano.png differ diff --git a/intro-unix-command-line/style.css b/intro-unix-command-line/style.css new file mode 100644 index 0000000..b5fbda4 --- /dev/null +++ b/intro-unix-command-line/style.css @@ -0,0 +1,126 @@ + +/* Set font size and alignment for the message */ +.bottom-message { + font-size: 0.8em !important; + font-style: italic !important; + text-align: center; + position: relative; + bottom: 0 !important; + left: 0; + right: 0; +} + +/* Code chunks background */ +.reveal pre code { + background-color: #2e2e2e; +} + +/* Left-align all code outputs */ +.reveal pre code { + text-align: left !important; +} + +/* Code chunks text color */ +.reveal pre code { + color: white; +} + +/* Code output background */ +.reveal pre code.output { + background-color: black; +} + +/* Code output text color */ +.reveal pre code.output { + color: white; + } + + +/* Add horizontal scrolling to all code outputs */ +.reveal pre code.output { + white-space: pre !important; + overflow-x: auto !important; +} +/* Change the font family used for code blocks */ +pre, code, kbd, samp { + font-family: "Courier New", Courier, monospace; + } + + +/* Add horizontal scrolling to all code chunks */ +.reveal pre code { + white-space: pre !important; + overflow-x: auto !important; +} +/* Bold slide titles and change color */ +.reveal h2 { + font-weight: bold !important; + color: #9c0366; +} +/* Bold slide titles and change color */ +.reveal h1 { + font-weight: bold !important; + color: #9c0366; +} +.reveal .slides>section:first-child h2 { + color: #333; + font-weight: normal !important; + } +/* Custom slide title */ +.my-title-slide h1 { + font-weight: bold; + color: #9c0366; + } +.my-title-slide h2 { +color: #333; +font-weight: normal !important; +} + + + +.reveal .slides>section:first-child h1 { + font-weight: bold !important; + color: #9c0366; +} + +/* Increase the spacing between list items */ +.reveal p { + text-align: left; + margin-left: 20px !important; + } +.reveal ul { + display: block; + margin-left: 75px !important; + margin-right: 50px !important; +} +.reveal ol { + display: block; + margin-bottom: 20px !important; +} + + +/* Decrease size of image, remove border, shadow and center align*/ +.reveal img { + max-width: 60%; + border: none !important; + box-shadow: none !important; + display: block !important; + margin: 0 auto !important; +} + +small { + font-size: 70%; +} + +.small-bullets ul { + font-size: 70%; +} + +.less-small-bullets ul { + font-size: 80%; +} + +.big-picture img{ + max-width: 70%; + border: 1px solid black !important; +} \ No newline at end of file diff --git a/intro-unix-command-line/unix_workshop_2023/part_1/.hidden_file.txt b/intro-unix-command-line/unix_workshop_2023/part_1/.hidden_file.txt new file mode 100644 index 0000000..e69de29 diff --git a/intro-unix-command-line/unix_workshop_2023/part_1/list_numbers.tsv b/intro-unix-command-line/unix_workshop_2023/part_1/list_numbers.tsv new file mode 100644 index 0000000..5a13953 --- /dev/null +++ b/intro-unix-command-line/unix_workshop_2023/part_1/list_numbers.tsv @@ -0,0 +1,3 @@ +1 3 6 10 11 22 0 3 +7 8 52 13 6 42 79 1 +13 4 9 82 67 71 93 3