Command-line tools for compression

• Compression reduces the size of a file
• gzip : compresses a file and replaces it with a compressed version (.gz)
• tar : create and manipulate archive files

Archive: a single file that contains one or more files and/or folders that have been compressed

gzip/gunzip: compress/uncompress a file

gunzip part_2/homo_sapiens.refseq.tsv.gz
du -h part_2/homo_sapiens.refseq.tsv

 26M    part_2/homo_sapiens.refseq.tsv

• The uncompressed file is 27 megabytes

gzip part_2/homo_sapiens.refseq.tsv
du -h part_2/homo_sapiens.refseq.tsv.gz

2.7M    part_2/homo_sapiens.refseq.tsv.gz

• Compressing it makes it a 10th of the size

Note

• The magnitude of the compression depends on type of data
• The units for file sizes are not the same across all systems
  • Some systems define a kilobyte as 1000 bytes, while others define it as 1024 bytes

tar: compressing folders into archives

• Does not provide compression on its own, it uses gzip to create compressed archive files

tar -czf part_1.tar.gz part_1
ls -l

total 8
drwx---rw-@ 4 nelphick  staff  128 Apr 16 21:10 part_1
-rw-r--r--@ 1 nelphick  staff  814 Apr 16 21:11 part_1.tar.gz
drwxr-xr-x@ 3 nelphick  staff   96 Apr 16 21:11 part_2

• -c: create a new archive
• -f: specify the name of the archive file
• -z: compress the archive with gzip

Unarchiving

• We did this in part 1 to unarchive the workshop folders

tar -xzf part_1.tar.gz

• -x: extract an archive
• -z: uncompress the archive with gzip
• -f: specify the name of the archive file

gunzip -c: cat compressed files

• To avoid uncompressing a large file just to read its contents, we can use gunzip -c
• This will output the the file to the terminal

gunzip -c part_2/homo_sapiens.refseq.tsv.gz | head

gene_stable_id  transcript_stable_id    protein_stable_id   xref    db_name info_type   source_identity xref_identity   linkage_type
ENSG00000160072 ENST00000673477 ENSP00000500094 NP_001304167    RefSeq_peptide  INFERRED_PAIR   -   -   -
ENSG00000160072 ENST00000673477 ENSP00000500094 NP_114127   RefSeq_peptide  DIRECT  100 100 -
ENSG00000160072 ENST00000673477 ENSP00000500094 NM_001317238    RefSeq_mRNA DIRECT  90  82  -
ENSG00000160072 ENST00000673477 ENSP00000500094 NM_031921   RefSeq_mRNA DIRECT  100 100 -
ENSG00000160072 ENST00000673477 ENSP00000500094 XM_005244806    RefSeq_mRNA_predicted   DIRECT  45  94  -
ENSG00000160072 ENST00000673477 ENSP00000500094 XM_011542241    RefSeq_mRNA_predicted   DIRECT  35  87  -
ENSG00000160072 ENST00000673477 ENSP00000500094 XM_011542244    RefSeq_mRNA_predicted   DIRECT  90  87  -
ENSG00000160072 ENST00000673477 ENSP00000500094 XM_047431593    RefSeq_mRNA_predicted   SEQUENCE_MATCH  90  96  -
ENSG00000160072 ENST00000673477 ENSP00000500094 XR_001737468    RefSeq_ncRNA_predicted  DIRECT  -   -   -

What are system variables?

• Special variables that contain information about the system’s configuration and state
• Used by the OS and programs to change their behavior based on the system’s state

Example:

echo $HOME

/Users/nelphick

Common System Variables

• $PWD : The working directory
• $HOME : The current user’s home directory
• $PS1 : the shell prompt string
• $TEMP : location of temporary files

PATH: locations of executable files

• When you enter a command, the OS searches the directories in the $PATH to find its associated executable file

echo $PATH

/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin:/usr/local/mysql/bin

• The OS will check these directories in the order they appear and use the first executable it finds

export: set system variables

• Useful for setting variables you want to be used across programs
• You can add new software to your $PATH like this:

export PATH="/path/to/new/software:$PATH"

• This will modify the $PATH for the current terminal session

Modifying the PATH for all future terminal sessions

• Add the export line to your ~/.bashrc or ~/.zshrc
• Proceed with caution
  • Make backups of these and read this guide
  • Changing the $PATH incorrectly can break system functionality

which: locate the executable associated with a command

• This command shows the location of the executable that the OS finds

which ls

/bin/ls

• Useful to check if there are multiple versions of a software installed

What is a script?

• Scripts are executable files for reusing code
• By convention scripts end in .sh
• This first line of the script is called the shebang

nano part_2/example_script.sh

#!/bin/bash

• The text that follows #! tells the OS where the interpreter is

which bash

/bin/bash

chmod: making a script executable

• By default, files are not executable

ls -l part_2/example_script.sh

-rw-r--r--@ 1 nelphick  staff  287 Apr 16 21:11 part_2/example_script.sh

• We can set the execute bit like this

chmod u+x part_2/example_script.sh
ls -l part_2/example_script.sh

-rwxr--r--@ 1 nelphick  staff  287 Apr 16 21:11 part_2/example_script.sh

Example

#!/bin/bash

# This is a comment. Comments are ignored by the shell.

# $1 is the first argument passed to the script
echo "Counting the genes in $1"

# count the unique genes in the file
u_genes=$(gunzip -c $1 | cut -f 1 | sort -u | wc -l)

echo "There are $u_genes unique genes in $1"

Let’s run it

./part_2/example_script.sh part_2/homo_sapiens.refseq.tsv.gz

Counting the genes in part_2/homo_sapiens.refseq.tsv.gz
There are    32538 unique genes in part_2/homo_sapiens.refseq.tsv.gz

Loops

• Useful for iterating over lines of a file or lists

for i in {1..3}
do

echo $i

done

1
2
3

While loops

count=0

while [ $count -lt 5 ]        # loop while count is less than 5
do
    echo $count
    count=$((count+1))
done

0
1
2
3
4

If statements

x=5

if [ $x -gt 10 ]                      # check if x is greater than 10
then
    echo "x is greater than 10"
else
    echo "x is not greater than 10"
fi                                    # end if statement

x is not greater than 10

sed : stream editor

• Parses and transforms text, using a compact programming language
• It reads and modifies text line by line from a file or input stream
• Supports regular expressions
• Useful for replacing text

Example:

sed 's/search_string/replace_string/g' input.txt > output.txt

ssh : secure shell - conect to remote server

• Logging in to a remote server
• Remote desktop for the terminal

ssh username@remote

• The username would be your user on the remote server and remote is the hostname or IP address of the remote server or computer

scp : secure copy

• Copy files from a remote server or computer

scp [options] [source] [destination]

• Copy from local to remote

scp /path/to/local/file.txt username@remote:/path/to/remote/directory/

• Copy from remote to local

scp username@remote:/path/to/file.txt /path/to/local/directory/

• -r : copy a whole folder

awk : processing structured data

• A small programming language that is designed to work with structured data
• Has more complicated syntax but is faster at processing large files
• Designed to read a file or input stream line by line
• Operates on records (lines) and fields (columns)

Basic command:

awk options 'pattern {action}' input_file

Example : Sum the first 2 columns of a file

awk -F '\t' '{print $1+$2}' part_1/list_numbers.tsv

4
15
17

• -F : provides the field separator
• $1,$2 : the first and second fields

Example : Find the average of a column

• For this example we only want the average if the 5th column equals “RefSeq_mRNA”

gunzip -c part_2/homo_sapiens.refseq.tsv.gz | \
awk -F '\t' '$5 == "RefSeq_mRNA" {sum += $7; count++} \
END {print sum / count}'

65.4642

Resources for learning AWK and sed

• The GNU AWK manual
• AWK Tutorial by Bruce Barnett
• Sed Tutorial by Bruce Barnett

Survey

• Please take some time to fill out the workshop survey: https://www.surveymonkey.com/r/DY7K5ZY

Additional learning materials

• Software carpentry provides a self paced course:
  • The Unix Shell
• Free online books:
  • The Unix Workbench
  • The Linux Command Line

Upcoming Data Science Training Program Workshops

Linear Mixed Effects Modeling
April 24-April 25, 2023 10:00am-12:00pm PDT

Machine Learning
April 28, 2023 10:00am-12:00pm PDT

Advanced Cytoscape Automation
May 2, 2023 1:00-4:00pm PDT

Introduction to RNA-Seq Analysis
May 15-May 16, 2023 9:00am-12:00pm PDT