diff --git a/docs/Working_on_Wynton_Part_1.html b/docs/Working_on_Wynton_Part_1.html new file mode 100644 index 0000000..2f805ff --- /dev/null +++ b/docs/Working_on_Wynton_Part_1.html @@ -0,0 +1,1270 @@ + + + + + + + Working on Wynton - Part 1 + + + + + + + + + + + + + + + + + +
+
+ +
+

Working on Wynton - Part 1

+

Natalie Elphick

+

April 15th, 2024

+
+ +
+

+
+Press the ? key for tips on navigating these slides +
+
+
+

Introductions

+

Natalie Elphick
+Bioinformatician I

+

Alex Pico
+Bioinformatics Core Director

+
+
+

Target Audience

+
    +
  • Prior experience with UNIX command-line
    • +
+
+
+

Part 1:

+
    +
  1. What is an HPC cluster?
    2. +
  2. Node Types and Loging in
    3. +
  3. Storage
    4. +
  4. Data Transfer
    5. +
  5. Installing Software
    6. +
  6. Containers
    7. +
+
+
+
+

What is Wynton HPC?

+ +
+
+

High-performance Computing Cluster

+
    +
  • A collection of specialized computers (nodes) connected together on +a fast local network
    • +
+

HPC Diagram

+
+
+

Wynton

+
    +
  • A HPC Linux environment available to all UCSF researchers for +free
    +
    • +
  • Uses the Rocky 8 linux OS
    • +
  • Includes several hundred compute nodes and a large shared storage +system (Cluster +specifications)
    +
    • +
  • Funded and administered cooperatively by UCSF campus IT and key +research groups
    • +
+

https://wynton.ucsf.edu

+
+
+
+

Node Types and Logging in

+ +
+
+

Node Types

+
    +
  • Login: Submit and query jobs. SSH to development +nodes. File management.
    • +
  • Development: Compile and install software. Test job +scripts. Submit and query jobs. Version control. File management.
    • +
  • Compute: Running short and long-running job +scripts.
    • +
  • Transfer: Fast in- & outbound file transfers. +File management.
    • +
+
+
+

The Login Nodes

+
    +
  • Only capable of basic tasks (file management, submitting and +checking on jobs)
    • +
  • Lacks access to pre-installed software tools that the development +nodes have
    • +
  • The primary method to log in is to use an SSH client +application
    • +
  • The Wynton HPC is up to date with information on logging in: Access +Cluster
    • +
+

Names:

+

log1, log2 and plog (for PHI users)

+
+
+

Login

+
    +
  • Make sure you are on the UCSF or Gladstone WiFi networks (or the +respective VPN)
    • +
  • ssh +[your-username]@[node-name].wynton.ucsf.edu
    • +
+
{local}$ ssh alice@log1.wynton.ucsf.edu
+alice@log1.wynton.ucsf.edu's password: 
+[alice@log1 ~]$
+
    +
  • There will not be any visual feedback when typing your password
    • +
+
+
+

The Development Nodes

+
    +
  • Has a set of core +software installed +
      +
    • e.g. git, vim, nano, make and python
      • +
    • +
  • Also has access to software +repositories some which are maintained by other users or research +groups +
      +
    • e.g. matlab, R and openjdk
      • +
    • +
  • Cannot be logged in to directly, only from a login node
    • +
+
ssh dev1
+

Names:

+

dev[1-3], gpudev1, pdev1 (PHI) and pgpudev1 (PHI)

+
+
+

Data Transfer Nodes

+
    +
  • Can be logged in to directly
    • +
  • Fast network speed
    • +
  • Limited software
    • +
  • Use for transferring files to and from Wynton
    • +
+

Example:

+
{local}$ scp local_file.tsv alice@dt1.wynton.ucsf.edu:~/
+

Names:

+

dt1 and dt2

+
+
+

Compute Nodes

+
    +
  • Can not be logged in to directly
    • +
  • Used to run non-interactive compute job scripts
    • +
  • The software to run the job script is provided using a +container
    • +
+

Compute Jobs

+
+
+
+

Storage

+ +
+
+

The File System

+
    +
  • A file system how information is stored and retrieved on a computer +
      +
    • Consists of files and directories
      • +
    • +
  • A local file system is function of the operating system and only +accessible from a single computer
    • +
  • A shared file system is accessible from multiple computers
    • +
+
+
+

BeeGFS

+
    +
  • Wynton uses a parallel shared file system called BeeGFS +
      +
    • The files are stored as “chunks” spread across many different +servers
      • +
    • +
  • BeeGFS has multiple services that work together to manage the file +system +
      +
    • Storage (stores the chunks)
      • +
    • Metadata (tracks the chunks and information about their file)
      • +
    • Management (tracks all of the services)
      • +
    • Client (provides linux access to the file system)
      • +
    • +
+
+
+

BeeGFS - Advantages

+
    +
  • High throughput
    • +
  • Redundancy can be built in by mirroring services
    • +
  • Adding new storage is fast and does not require downtime
    • +
+
+
+

BeeGFS - Caveats

+
    +
  • For any client node, performance is limited by the network bandwidth +of that node
    • +
  • Network latency becomes extremely important for all metadata +requests
    • +
  • Certain input/output patterns can be problematic
    • +
+
+
+

BeeGFS - I/O patterns

+
    +
  • Anything that requires lots of metadata operations can feel slow +
      +
    • e.g: lots of writes to the same directory and lots of file lookups +and directory searches (conda)
      • +
    • +
  • Keep the number of reads and writes to a single directory to a +reasonable number
    • +
  • If using conda, putting the conda application inside a Apptainer +(formerly singularity) container will result in better performance
    • +
+
+
+

BeeGFS - Tips

+
    +
  • Prefer fewer, large files over many small ones
    • +
  • Distribute reading and writing over several directories +
      +
    • Including compute job output and error files
      • +
    • +
  • Use local scratch (/scratch) when possible
    • +
  • Don’t include anything in /wynton in your default +LD_LIBRARY_PATH
    • +
+
+
+

Storage

+
    +
  • Wynton storage is not backed up
    • +
  • /wynton/[group_name]/[user] +
      +
    • User home directory - limited to 500 GiB
      • +
    • +
  • /wynton/[group_name] +
      +
    • User group directory - disk quota varies by group
      • +
    • Use this directory for any analysis you want to share with your +lab
      • +
    • +
  • More +information on disk quotas
    • +
+

To check your group disk quota run:

+
beegfs-ctl --getquota --storagepoolid=12 --gid "$(id --group)"
+
+
+

Scratch - Temporary Storage

+
    +
  • Local /scratch - 0.1-1.8 TiB/node storage unique to +each compute node +
      +
    • Can only be accessed from the specific compute node
      • +
    • +
  • /wynton/scratch and +/wynton/protected/scratch (for PHI users) +
      +
    • 703 TiB storage accessible from everywhere
      • +
    • +
  • No quotas
    • +
+


+

Files not used for 2 weeks are automatically +deleted

+
+
+

Gladstone HIVE

+
    +
  • Gladstone’s HIVE storage server is mounted directly to Wynton under +/gladstone +
      +
    • Only certain HIVE folders are accessible directly on Wynton
      • +
    • Files under /gladstone are backed up
      • +
    • +
  • Naming: /gladstone/[lab] +
      +
    • Directories that are shared between multiple labs can be set up by +contacting Gladstone IT
      +
      • +
    • +
  • For more information visit the IT +knowledge base page
    • +
+
+
+

Storage Advice

+
    +
  • Always back up anything you store under +/wynton
    • +
  • Use /gladstone if you have access to it for all of +your work since it is automatically backed up
    • +
  • Use the scratch directories to store temporary files so they do not +count against your group or user quotas
    • +
+
+
+
+

Data Transfer

+ +
+
+

Secure Copy - scp

+
    +
  • Local file to Wynton
    • +
+
{local}$ scp /path/to/local_file.tsv alice@dt1.wynton.ucsf.edu:/destination/path
+
    +
  • Copy a directory to a folder on Wynton
    • +
+
{local}$ scp -r local_folder/ alice@dt1.wynton.ucsf.edu:/destination/path
+
    +
  • Copy a single file to Wynton from your local machine
    • +
+
{local}$ scp alice@dt1.wynton.ucsf.edu:/path/to/local_file.tsv /destination/path
+
+
+

GUI SFTP Clients

+
    +
  • These let you transfer files to and from Wynton using a GUI
    • +
  • 2 +factor authentication may be required
    • +
  • Cyberduck +
      +
    • Navigate to Preferences -> Transfers -> General
      • +
    • change the Transfer Files setting “Use browser connection” instead +of “Open Multiple connections”
      • +
    • +
  • FileZilla +
      +
    • In the General tab, select ‘SFTP’ as the Protocol instead of +‘FTP’
      • +
    • For Logon Type, select ‘Interactive’ instead of ‘Ask for +Password’
      • +
    • Under the Transfer Settings tab, you might need to click the ‘Limit +number of simultaneous connections’ and make sure the ‘Maximum number of +connections’ is set to 1
      • +
    • +
+
+
+

Globus

+
    +
  • Globus is a +service for moving, syncing, and sharing large amounts of data
    • +
  • Wynton Accounts are not required to transfer data with Globus
    • +
  • Useful for transferring data between institutions
    • +
+
+
+

Rclone

+
    +
  • Rclone is a command-line program to manage files on remote +storage
    • +
  • Can be used to transfer data from Wynton directly to DropBox or other storage systems +(AWS, Azure, Google Drive etc.) +
      +
    • Do this from a data transfer node using screen/tmux
      • +
    • +
  • Do not use rclone for transfers to Box, follow the Wynton to +UCSF Box instructions
    • +
+
+
+

Poll 1

+

Which of these can you not log in to from your +computer?

+
    +
  1. Login Nodes
    2. +
  2. Development Nodes
    3. +
  3. Data transfer Nodes
    4. +
  4. Compute Nodes
    5. +
+
+
+

Poll 2

+

The /wynton directory is backed up on a nightly +basis so do not need to back up the data you store here.

+
    +
  1. True
    2. +
  2. False
    3. +
+
+
+
+

Installing Software

+ +
+
+

Basics

+
    +
  • Check if the tool is already available in a module
    • +
  • Ensure the software you are trying to install is compatible with +Rocky 8 linux (use a container if not)
    • +
  • Always install software in a development node
    • +
  • Download a precompiled binary or install +from source
    • +
+
+
+

Install Samtools from Source

+
    +
  1. Download and extract source code
    2. +
+
[alice@dev1 ~]$ mkdir -p "/scratch/$USER"
+[alice@dev1 ~]$ cd "/scratch/$USER"
+[alice@dev1 alice]$ wget https://github.com/samtools/samtools/releases/download/1.19.2/samtools-1.19.2.tar.bz2
+[alice@dev1 alice]$ tar -x -f samtools-1.19.2.tar.bz2
+
    +
  1. Create install location and configure
    2. +
+
[alice@dev1 ~]$ mkdir -p $HOME/software/samtools-1.14
+[alice@dev1 ~]$ ./configure --prefix=$HOME/software/samtools-1.14
+
    +
  1. Build and install
    2. +
+
[alice@dev1 ~]$ make
+[alice@dev1 ~]$ make install
+
+
+

Install Nextflow

+
    +
  • Scientific workflow system with a community maintained set of core bioinformatics analysis pipelines +
      +
    • We will cover an example RNA-seq pipeline in part 2
      +
      • +
    • +
  • These can be configured to use the Wynton compute job submission +system
    • +
+
[alice@dev1 ~]$ cd ~/software
+[alice@dev1 ~]$ curl -s "https://get.sdkman.io" | bash
+[alice@dev1 ~]$ exit
+[alice@log1 ~]$ ssh dev1
+[alice@dev1 ~]$ sdk install java 17.0.6-tem
+[alice@dev1 ~]$ wget -qO- https://get.nextflow.io | bash
+[alice@dev1 ~]$ nextflow -v
+
+
+
+

Containers

+ +
+
+

Motivation

+
    +
  • Compute heavy jobs (high RAM, multiple cores) should be run on +compute nodes
    • +
  • Containers allow us to make additional software available to the +compute nodes +
      +
    • Also allows the use of software that might be hard to install on +Rocky 8 Linux
      • +
    • Improves reproducibility
      • +
    • +
+

Compute Jobs

+
+
+

Definitions

+
    +
  • Virtualization: When software mimics the functions +of physical hardware to run virtual machines
    • +
  • Containers: Implements virtualization using an +image as its base
    • +
  • Images: An ordered collection of root filesystem +changes and the corresponding execution parameters for use within a +container runtime
    • +
+
+
+

Apptainer

+
    +

  • Wynton supports Apptainer +(formerly singularity) containers

    • +

  • Docker is a commonly used +image creation software, these can be turned into apptainer image files +(.sif) easily

    • +

  • apptainer run

    +
      +
    • Run predefined script within container
      • +
    • +

  • apptainer exec

    +
      +
    • Execute any command within container
      • +
    • +

  • apptainer shell

    +
      +
    • Run bash shell within container
      • +
    • +
+
+
+

Example Container - Hello World

+
    +
  • Run this command to convert the public Docker image to a apptainer +image file
    • +
+
[alice@dev1 ~]$ apptainer pull docker://natalie23gill/hello-world:1.0
+
    +
  • Execute the “hi” command in the container
    • +
+
[alice@dev1 ~]$ apptainer exec hello-world_1.0.sif hi
+
    __  __     ____         _       __           __    __   __
+   / / / /__  / / /___     | |     / /___  _____/ /___/ /  / /
+  / /_/ / _ \/ / / __ \    | | /| / / __ \/ ___/ / __  /  / / 
+ / __  /  __/ / / /_/ /    | |/ |/ / /_/ / /  / / /_/ /  /_/  
+/_/ /_/\___/_/_/\____/     |__/|__/\____/_/  /_/\__,_/  (_)
+
+
+

Example Container

+
    +
  • This container has figlet installed which creates +ASCII art from text input
    • +
  • Try running this command to create your own using exec
    • +
+
[alice@dev1 ~]$ apptainer exec hello-world_1.0.sif figlet your_text
+
+
+

Docker

+
    +
  • Docker uses Dockerfiles to specify image creation
    • +
  • Preferred by the Gladstone Bioinformatics Core to create new +images
    • +
  • In part 2, we will go over how to build custom container images from +DockerFiles +
    • +
  • To see the Dockerfile used to create the hello-world image, +run:
    • +
+
[alice@dev1 ~]$ apptainer exec hello-world_1.0.sif cat /Dockerfile
+
+
+
+

End of Part 1

+ +
+
+

Thank You!

+ +
+
+

Upcoming Data Science Training Program Workshops

+

Introduction +to Linear Mixed Effects Models
+April 25-April 26, 2024 1-3pm PDT

+

Single +Cell RNA-Seq Data Analysis
+April 29-April 30, 2024 9am-4pm PDT

+

Single +Cell ATAC-Seq Data Analysis Part 1
+May 6-May 7, 2024 1-4pm PDT

+

Complete +Schedule

+
+
+
+ + + + + + + + + + + + + + diff --git a/docs/Working_on_Wynton_Part_2.html b/docs/Working_on_Wynton_Part_2.html new file mode 100644 index 0000000..cdf57b3 --- /dev/null +++ b/docs/Working_on_Wynton_Part_2.html @@ -0,0 +1,1316 @@ + + + + + + + Working on Wynton - Part 2 + + + + + + + + + + + + + + + + + +
+
+ +
+

Working on Wynton - Part 2

+

Natalie Elphick

+

April 16th, 2024

+
+ +
+

+
+Press the ? key for tips on navigating these slides +
+
+
+

Introductions

+

Natalie Elphick
+Bioinformatician I

+

Alex Pico
+Bioinformatics Core Director

+
+
+

Target Audience

+
    +
  • Prior experience with UNIX command-line
    • +
+
+
+

Part 2:

+
    +
  1. Custom Containers
    2. +
  2. Submitting Compute Jobs
    3. +
  3. Array Jobs
    4. +
  4. GPU Jobs
    5. +
  5. Running Pipelines
    6. +
  6. Jupyter Notebooks
    7. +
  7. RStudio Server
    8. +
  8. How to get help
    9. +
+
+
+
+

Custom Containers

+ +
+
+

Motivation

+
    +
  • Compute heavy jobs (high RAM, multiple cores) should be run on +compute nodes
    • +
  • Containers allow us to make additional software available to the +compute nodes +
      +
    • Also allows the use of software that might be hard to install on +Rocky 8 Linux
      • +
    • Improves reproducibility
      • +
    • +
+

Compute Jobs

+
+
+

Dockerfile Basics

+
    +
  • Dockerfiles contain instructions to build an image in +layers
    • +
  • Layers are added using Dockerfile instruction syntax
    • +
  • Images are built by navigating to the directory that contains the +Dockerfile and running:
    • +
+
docker build .
+
+
+

Dockerfile Instructions

+
    +
  • First instruction is always FROM which specifies +the base image +
      +
    • Base images are a starting point with some basics already installed +like the OS and build tools, find them on DockerHub
      • +
    • +
  • RUN : Use before running any shell commands
    • +
  • SHELL : Set the shell
    • +
  • USER : Set the user (within the image)
    • +
  • CMD : Set the default instruction to be run by the +image
    • +
  • COPY : COPY files into the image
    • +
+

See the Dockerfile +documentation for a full list of instructions

+
+
+

Example Dockerfile

+
    +
  • Click here +to download the example Dockerfile
    • +
  • Open in your preffered text editor
    • +
+
# Bioconductor base image gives us access to a lot of bioinformatics tools and R packages.
+FROM bioconductor/bioconductor_docker:RELEASE_3_17
+
+# Shell options, we want to exit if any command fails
+SHELL ["/bin/bash", "-o", "pipefail", "-c"]
+
+# Root permissions are required to install packages
+USER root
+
+
+# Install any UNIX packages you need
+# First we update the package list and then install GNU make
+# We clean up after ourselves to reduce the image size
+RUN apt-get update && apt-get upgrade -y \
+    && apt-get install -y --no-install-recommends make \
+    && apt-get clean \
+    && rm -rf /var/lib/apt/lists/*
+
+# Install Seurat and harmony
+RUN Rscript -e 'install.packages(c("Seurat","harmony"))'
+# Check if installs worked
+RUN Rscript -e 'lapply(c("Seurat","harmony"), library, character.only = TRUE)'
+
+
+# Run container as non-root to avoid permission issues
+RUN groupadd -g 10001 notroot && \
+   useradd -u 10000 -g notroot notroot 
+
+# Switch to the non-root user
+USER notroot:notroot
+
+# Default command to run when the container starts
+CMD ["/bin/bash"]
+
+# Copy dockerfile into the image (optional, but can be useful for reproducibility)
+COPY Dockerfile /Dockerfile
+
+
+

Building Example Image

+
    +
  • Do not run this during the workshop +
      +
    • It requires a lot of RAM
      • +
    • +
  • On macOS, make sure you have the Docker Desktop App running
    • +
  • We can provide an additional argument to the build +command, -t, to set the name of the docker image +
      +
    • We can add version tags after the name using “:”
      • +
    • +
+
docker build -t docker_hub_user/seurat-harmony:1.0 .
+
+
+

Pushing Images to DockerHub

+
    +
  • Make sure you are signed in to your DockerHub account locally +(Docker Desktop for macOS)
    • +
  • The image name must start with your user name
    • +
+
docker push docker_hub_user/seurat-harmony:1.0
+
    +
  • These can then be “pulled” on to Wynton as apptainer image files +(image must be public)
    • +
+
[alice@dev1 ~]$ apptainer pull docker://docker_hub_user/seurat-harmony:1.0
+
+
+

Notes on Building Custom Images

+
    +
  • Time consuming process and uses a lot of RAM on your local +machine
    • +
  • A good base image can save you a lot of time
    • +
  • You must run apt-get update and +apt-get install in the same command +
      +
    • Otherwise you will encounter caching issues
      • +
    • +
  • Remember to use apt-get install -y +
      +
    • You will have no control over the process while it’s building
      • +
    • +
+
+
+
+

Compute Jobs

+ +
+
+

Submission Script - Basics

+ +
#!/bin/bash           # the shell language when run outside of the job scheduler
+#                     # lines starting with #$ is an instruction to the job scheduler
+#$ -S /bin/bash       # the shell language when run via the job scheduler [IMPORTANT]
+#$ -cwd               # job should run in the current working directory
+#$ -j y               # STDERR and STDOUT should be joined
+#$ -l mem_free=1G     # job requires up to 1 GiB of RAM per slot (core)
+#$ -l scratch=2G      # job requires up to 2 GiB of local /scratch space
+#$ -l h_rt=1:00:00   # job requires up to 1 hour of runtime 
+#$ -r y               # if job crashes, it should be restarted
+
+date
+hostname
+
+## End-of-job summary, if running as a job
+[[ -n "$JOB_ID" ]] && qstat -j "$JOB_ID"  # This is useful for debugging and usage purposes,
+                                          # e.g. "did my job exceed its memory request?"
+
+
+

Submission Script - Apptainer

+
    +
  • Download +this example job submission script that uses a container
    • +
  • Paths that the container needs read/write access to need to be +mounted with APPTAINER_BINDPATH
    • +
+
#!/bin/bash
+#$ -S /bin/bash      # the shell language when run via the job scheduler
+#$ -cwd               # job should run in the current working directory
+#$ -j y               # STDERR and STDOUT should be joined
+#$ -l mem_free=1G     # job requires up to 1 GiB of RAM per slot
+#$ -l scratch=2G      # job requires up to 2 GiB of local /scratch space
+#$ -l h_rt=1:00:00    # job requires up to 1 hour of runtime
+
+
+# Mount the current directory to the container
+# Any directroy that needs to be accessed by the container should be mounted
+directory=$(pwd)
+export APPTAINER_BINDPATH="$directory"
+
+h=$(hostname)
+
+singularity run hello-world_1.0.sif figlet $h > $directory/hello.txt
+
+[[ -n "$JOB_ID" ]] && qstat -j "$JOB_ID"
+
+
+

Parallel Processing Jobs

+
    +

  • By default jobs run on a single core

    • +

  • Multicore jobs must run in a SGE parallel environment (PE) and +tell SGE how many cores the job will use

    • +

  • Do not use more cores than requested

    • +

  • There are four parallel environments on Wynton:

    +
      +
    • smp: for single-host parallel jobs using Symmetric +multiprocessing (SMP)
      • +
    • mpi: for multiple-host parallel jobs based on MPI +parallelization
      • +
    • mpi_onehost: for single-host parallel jobs based on +MPI parallelization
      • +
    • mpi-8: for multi-threaded multi-host jobs based on +MPI parallelization
      • +
    • +
+
+
+

Example Parallel Job

+
    +
  • The simplest parallel environment on Wynton is smp, +a single node with n cores
    • +
  • Download +this example smp job submission script
    • +
+
#!/bin/bash
+#$ -S /bin/bash 
+#$ -cwd
+#$ -j y
+#$ -pe smp 4                    # 4 cores on a single node
+#$ -l mem_free=2G               # 2 GiB of RAM per slot (core), so 8 GiB total
+#$ -l scratch=5G                # 5 GiB of local /scratch space
+#S -l h_rt=08:00:00
+
+
+# Code that requires 4 cores
+# **Specify the number of cores as ${NSLOTS}**
+
+
+
+[[ -n "$JOB_ID" ]] && qstat -j "$JOB_ID"
+
+
+

Array Jobs

+
    +
  • This is a good option if the script you want to run operates on +discrete sets of data +
      +
    • e.g. sample or chromosome
      • +
    • +
  • Download +this example array job submission script
    • +
+
#!/bin/bash           
+#$ -S /bin/bash       
+#$ -cwd               
+#$ -j y               
+#$ -l mem_free=1G     
+#$ -l scratch=2G     
+#$ -l h_rt=1:00:00   
+#$ -t 1-5          # Number of tasks to run in the array (each is a job with the same resource requirements above)
+
+params=(sample1 sample2 sample3 sample4 sample5)
+
+# The task ID is stored in the variable SGE_TASK_ID
+# This variable is used to index the array of parameters
+# The task ID is 1-indexed
+param=${params[$SGE_TASK_ID - 1]}
+
+echo "Running task $SGE_TASK_ID with parameter $param"
+
+# Code for each task
+
+[[ -n "$JOB_ID" ]] && qstat -j "$JOB_ID"
+
+
+

GPU Jobs

+
    +
  • To run a GPU job, +specify -q gpu.q (queue) as a GPU queue +
      +
    • Other GPU queues may be available to you depending on your lab
      • +
    • +
  • It is important to specify the GPU using the +SGE_GPU variable so that your job uses its assigned GPU +
      +
    • For CUDA based tools, add export +CUDA_VISIBLE_DEVICES=$SGE_GPU to your submission script
      • +
    • +
  • GPU jobs must include a runtime request or they will be removed from +the queue
    • +
+
+
+

Submitting and Querying jobs

+
    +
  • Use qsub to submit jobs
    • +
+
[alice@dev1 ~]$ qsub job1.sh
+Your job 714888 ("job1.sh") has been submitted
+
    +
  • Use qstat to check the status of your jobs
    • +
+
[alice@dev1 ~]$ qstat
+job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
+-----------------------------------------------------------------------------------------------------------------
+ 714888 0.06532 job1 alice     r     03/25/2024 19:54:18 member.q@msg-hmio1                 1        
+ 714889 0.06532 job2 alice     r     03/25/2024 19:54:19 member.q@msg-hmio1                 1
+

Read the querying +jobs Wynton documentation for more information.

+
+
+

Estimating Job Resources

+
    +
  • Try to estimate the amount of RAM needed using a small test +dataset
    • +
  • Request a little more RAM than you need to avoid having your job +cancelled
    • +
  • Check on jobs you are running for the first time with qstat +-j to make sure they are not going over
    • +
+
+
+

Poll 3

+

Anything that you can run on a compute node can be run on a +development node.

+
    +
  1. True
    2. +
  2. False
    3. +
+
+
+
+

Running Pipelines

+ +
+
+

Nextflow RNA-seq

+ +

RNA-seq

+
+
+

Example - RNA-seq Pipeline

+

Do not run this during the workshop as it will fill up the +Wynton SGE queue

+
    +
  • Download the testing +script +
      +
    • Runs a minimal test on the RNA-seq pipeline
      • +
    • +
  • Download the config +file +
      +
    • Configures nextflow to use the SGE job scheduler and sets limits on +compute job resources for each process
      • +
    • +
  • Put these in the same directory (do not use your user home directory +for this) and run the script in a screen/tmux session
    • +
  • When not running the test, the -profile should be +apptainer
    • +
+
+
+
+

Jupyter Notebooks

+ +
+
+

Installing Jupyter Notebooks

+
    +
  • The preferred way to install and use Jupyter +notebooks on Wynton is though pip, not conda
    • +
+
python3 -m pip install --user notebook
+
    +
  • Jupyter notebooks can only be run on development nodes
    • +
  • See the Wynton python +documentation for more info on managing python environments on +Wynton
    • +
+
+
+

Running Jupyter Notebooks - Step 1

+
    +
  • You cannot connect from outside Wynton HPC directly to a development +node +
      +
    • Instead we need to use SSH port forwarding to establish the +connection with a local web browser
      • +
    • +
  • Find an available TCP port:
    • +
+
[alice@dev1 ~]$ module load CBI port4me
+[alice@dev1 ~]$ port4me --tool=jupyter
+47467
+

Note the port number returned by port4me, you will need this +later.

+
+
+

Running Jupyter Notebooks - Step 2

+
    +
  • Launch Jupyter notebook using the port numer from step 1
    • +
+
[alice@dev1]$ jupyter notebook --no-browser --port 47467
+[I 2024-03-20 14:48:45.693 ServerApp] jupyter_lsp | extension was successfully linked.
+[I 2024-03-20 14:48:45.698 ServerApp] jupyter_server_terminals | extension was successfully linked.
+[I 2024-03-20 14:48:45.703 ServerApp] jupyterlab | extension was successfully linked.
+[I 2024-03-20 14:48:45.708 ServerApp] notebook | extension was successfully linked.
+[I 2024-03-20 14:48:46.577 ServerApp] notebook_shim | extension was successfully linked.
+[I 2024-03-20 14:48:46.666 ServerApp] notebook_shim | extension was successfully loaded.
+[I 2024-03-20 14:48:46.668 ServerApp] jupyter_lsp | extension was successfully loaded.
+[I 2024-03-20 14:48:46.669 ServerApp] jupyter_server_terminals | extension was successfully loaded.
+[I 2024-03-20 14:48:46.675 LabApp] JupyterLab extension loaded from /wynton/home/boblab/alice/.local/lib/python3.11/site-packages/jupyterlab
+[I 2024-03-20 14:48:46.675 LabApp] JupyterLab application directory is /wynton/home/boblab/alice/.local/share/jupyter/lab
+[I 2024-03-20 14:48:46.677 LabApp] Extension Manager is pypi.
+[I 2024-03-20 14:48:46.707 ServerApp] jupyterlab | extension was successfully loaded.
+[I 2024-03-20 14:48:46.711 ServerApp] notebook | extension was successfully loaded.
+[I 2024-03-20 14:48:46.712 ServerApp] Serving notebooks from local directory: /wynton/home/boblab/alice
+[I 2024-03-20 14:48:46.712 ServerApp] Jupyter Server 2.13.0 is running at:
+[I 2024-03-20 14:48:46.712 ServerApp] http://localhost:44214/tree?token=8e37f8d62fca6a1c9b2da429f27df5ebcec706a808c3a8f2
+[I 2024-03-20 14:48:46.712 ServerApp]     http://127.0.0.1:44214/tree?token=8e37f8d62fca6a1c9b2da429f27df5ebcec706a808c3a8f2
+[I 2024-03-20 14:48:46.712 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
+[C 2024-03-20 14:48:46.725 ServerApp]
+
+    To access the server, open this file in a browser:
+        file:///wynton/home/boblab/alice/.local/share/jupyter/runtime/jpserver-2853162-open.html
+    Or copy and paste one of these URLs:
+        http://localhost:44214/tree?token=8e37f8d62fca6a1c9b2da429f27df5ebcec706a808c3a8f2
+        http://127.0.0.1:44214/tree?token=8e37f8d62fca6a1c9b2da429f27df5ebcec706a808c3a8f2
+
+
+

Running Jupyter Notebooks - Step 3

+
    +
  • Set up SSH port forwarding on your local machine in a separate +terminal, leave both terminals open
    • +
+
{local}$ ssh -J alice@log1.wynton.ucsf.edu -L 47467:localhost:47467 alice@dev1
+...
+[alice@dev1 ~]$
+

The notebook should now be available at the URL from step 2

+
+
+
+

RStudio Server

+ +
+
+

RStudio Server

+
    +
  • RStudio +server is already available in the CBI module
    • +
  • This allows you to set up a personal RStudio instance that only you +can access
    • +
  • Requires two separate SSH connections to the cluster:
    + +
      +
    • One to launch RStudio Server
      • +
    • One to connect to it
      • +
    • +
+
+
+

RStudio Server - Step 1

+
    +
  • Launch your own RStudio Server instance
    • +
+
[alice@dev1 ~]$ module load CBI rstudio-server-controller
+[alice@dev1 ~]$ rsc start
+alice, your personal RStudio Server 2023.09.1-494 running R 4.3.2 is available on:
+
+  <http://127.0.0.1:20612>
+
+Importantly, if you are running from a remote machine without direct access
+to dev1, you need to set up SSH port forwarding first, which you can do by
+running:
+
+  ssh -L 20612:dev1:20612 alice@log1.wynton.ucsf.edu
+
+in a second terminal from your local computer.
+
+Any R session started times out after being idle for 120 minutes.
+WARNING: You now have 10 minutes, until 2023-11-15 17:06:50-08:00, to
+connect and log in to the RStudio Server before everything times out.
+Your one-time random password for RStudio Server is: y+IWo7rfl7Z7MRCPI3Z4
+

Note the password and URL, they will be needed to log in to the +server instance.

+
+
+

RStudio Server - Step 2

+
    +
  • Connect to your personal RStudio Server instance from your local +machine in a separate terminal
    • +
+
{local}$ ssh -L 20612:dev1:20612 alice@log1.wynton.ucsf.edu
+alice1@log1.wynton.ucsf.edu:s password: XXXXXXXXXXXXXXXXXXX
+[alice@log1 ~]$
+
+
+

RStudio Server - Step 3

+
    +
  • Open RStudio Server in your local web browser
    • +
  • Open the link from step 1
    • +
  • Enter your Wynton user name
    • +
  • Enter the password from step 1
    • +
+
+
+
+

How to Get Help

+ +
+
+

Wynton Questions

+
    +
  • Follow the Wynton question +checklist
    • +
  • Email +
    • +
  • Slack +
      +
    • ucsf-wynton
      • +
    • Sign-up using a UCSF email address
      • +
    • Email support if that does not work
      • +
    • +
  • Zoom office hours every Tuesday at 11-12pm +
      +
    • Zoom URL in the message-of-the-day (MOTD) that you get when you log +into Wynton
      • +
    • +
+
+
+

Bioinformatics Questions

+ +
+
+
+

End of Part 2

+ +
+
+

Thank You!

+ +
+
+

Upcoming Data Science Training Program Workshops

+

Introduction +to Linear Mixed Effects Models
+April 25-April 26, 2024 1-3pm PDT

+

Single +Cell RNA-Seq Data Analysis
+April 29-April 30, 2024 9am-4pm PDT

+

Single +Cell ATAC-Seq Data Analysis Part 1
+May 6-May 7, 2024 1-4pm PDT

+

Complete +Schedule

+
+
+
+ + + + + + + + + + + + + + diff --git a/working-on-wynton-hpc/.Rprofile b/working-on-wynton-hpc/.Rprofile new file mode 100644 index 0000000..81b960f --- /dev/null +++ b/working-on-wynton-hpc/.Rprofile @@ -0,0 +1 @@ +source("renv/activate.R") diff --git a/working-on-wynton-hpc/README.md b/working-on-wynton-hpc/README.md index 83f1535..1098c53 100644 --- a/working-on-wynton-hpc/README.md +++ b/working-on-wynton-hpc/README.md @@ -2,4 +2,14 @@ [Link to wiki](https://github.com/gladstone-institutes/Bioinformatics-Workshops/wiki/Working-on-Wynton-HPC) -### Description of files +## Structure + +- `working-on-wynton-hpc.Rproj` : Used to manage the `renv` for the workshop slides + - To install the project env, open this file and run `renv::restore()` in the console + +- `./slide_materials` : Contains any images/assets needed for the slides. +- `./workshop_materials` : Contains any materials needed by the registrants (switch to using DropBox if the size of this becomes too large) +- `./renv` : Used by `renv` to manage project library, do not modfiy these files directly +- `./Working_on_Wynton_Part_1.Rmd` : revealjs based slides for part 1 +- `./Working_on_Wynton_Part_2.Rmd` : revealjs based slides for part 2 +- `./style.css` : CSS style sheet for both sets of revealjs slides diff --git a/working-on-wynton-hpc/Working_on_Wynton_Part_1.Rmd b/working-on-wynton-hpc/Working_on_Wynton_Part_1.Rmd new file mode 100644 index 0000000..88b95f1 --- /dev/null +++ b/working-on-wynton-hpc/Working_on_Wynton_Part_1.Rmd @@ -0,0 +1,479 @@ +--- +title: "Working on Wynton - Part 1" +author: "Natalie Elphick" +date: "April 15th, 2024" +knit: (function(input, ...) { + rmarkdown::render( + input, + output_dir = "../docs" + ) + }) +output: + revealjs::revealjs_presentation: + theme: simple + highlight: default + css: style.css +--- + +```{r, setup, include=FALSE} + +``` + +## + +
*Press the ? key for tips on navigating these slides*
+ +## Introductions + +**Natalie Elphick** +Bioinformatician I + +**Alex Pico** +Bioinformatics Core Director + + +## Target Audience +- Prior experience with UNIX command-line + + + +## Part 1: + +1. What is an HPC cluster? +2. Node Types and Loging in +3. Storage +4. Data Transfer +5. Installing Software +6. Containers + + + +# What is Wynton HPC? + +## High-performance Computing Cluster + +- A collection of specialized computers (nodes) connected together on a fast local network + +![HPC Diagram](slide_materials/HPC_diagram.png) + + + +## Wynton {.small-bullets} + +- A HPC Linux environment available to all UCSF researchers for free +- Uses the Rocky 8 linux OS +- Includes several hundred compute nodes and a large shared storage system ([Cluster specifications](https://wynton.ucsf.edu/hpc/about/specs.html)) +- Funded and administered cooperatively by UCSF campus IT and key research groups + +[https://wynton.ucsf.edu](https://wynton.ucsf.edu) + + + +# Node Types and Logging in + +## Node Types {.small-bullets} + +- **Login:** Submit and query jobs. SSH to development nodes. File management. +- **Development:** Compile and install software. Test job scripts. Submit and query jobs. Version control. File management. +- **Compute:** Running short and long-running job scripts. +- **Transfer:** Fast in- & outbound file transfers. File management. + +## The Login Nodes {.small-bullets} + +- Only capable of basic tasks (file management, submitting and checking on jobs) +- Lacks access to pre-installed software tools that the development nodes have +- The primary method to log in is to use an SSH client application +- The Wynton HPC is up to date with information on logging in: [Access Cluster](https://wynton.ucsf.edu/hpc/get-started/access-cluster.html) + +Names: + + +log1, log2 and plog (for PHI users) + +## Login {.small-bullets} + +- Make sure you are on the UCSF or Gladstone WiFi networks (or the respective VPN) +- **ssh [your-username]@[node-name].wynton.ucsf.edu** + +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +echo "{local}$ ssh alice@log1.wynton.ucsf.edu +alice@log1.wynton.ucsf.edu's password: +[alice@log1 ~]$" +``` + +- There will not be any visual feedback when typing your password + + +## The Development Nodes {.small-bullets} + +- Has a set of [core software](https://wynton.ucsf.edu/hpc/software/core-software.html) installed + - e.g. git, vim, nano, make and python +- Also has access to [software repositories](https://wynton.ucsf.edu/hpc/software/software-repositories.htmll) some which are maintained by other users or research groups + - e.g. matlab, R and openjdk +- Cannot be logged in to directly, only from a login node +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +echo "ssh dev1" +``` + +Names: + +dev[1-3], gpudev1, pdev1 (PHI) and pgpudev1 (PHI) + +## Data Transfer Nodes {.small-bullets} + +- Can be logged in to directly +- Fast network speed +- Limited software +- Use for transferring files to and from Wynton + +Example: + +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +echo "{local}$ scp local_file.tsv alice@dt1.wynton.ucsf.edu:~/" +``` + +Names: + +dt1 and dt2 + + + +## Compute Nodes {.small-bullets .big-picture} + +- Can **not** be logged in to directly +- Used to run non-interactive compute job scripts +- The software to run the job script is provided using a container + + +![Compute Jobs](slide_materials/compute_job_workflow.svg) + + + +# Storage + + +## The File System {.small-bullets} + +- A file system how information is stored and retrieved on a computer + - Consists of files and directories +- A local file system is function of the operating system and only accessible from a single computer +- A shared file system is accessible from multiple computers + + + + + + +## BeeGFS {.small-bullets} + +- Wynton uses a *parallel* shared file system called BeeGFS + - The files are stored as "chunks" spread across many different servers +- BeeGFS has multiple services that work together to manage the file system + - Storage (stores the chunks) + - Metadata (tracks the chunks and information about their file) + - Management (tracks all of the services) + - Client (provides linux access to the file system) + + +## BeeGFS - Advantages +- High throughput +- Redundancy can be built in by mirroring services +- Adding new storage is fast and does not require downtime + +## BeeGFS - Caveats +- For any client node, performance is limited by the network bandwidth of that node +- Network latency becomes extremely important for all metadata requests +- Certain input/output patterns can be problematic + +## BeeGFS - I/O patterns {.small-bullets} +- Anything that requires lots of metadata operations can feel slow + - e.g: lots of writes to the same directory and lots of file lookups and directory searches (**conda**) +- Keep the number of reads and writes to a single directory to a reasonable number +- If using conda, putting the conda application inside a Apptainer (formerly singularity) container will result in better performance + +## BeeGFS - Tips + +- Prefer fewer, large files over many small ones +- Distribute reading and writing over several directories + - Including compute job output and error files +- Use local scratch (/scratch) when possible +- Don't include anything in **/wynton** in your default LD_LIBRARY_PATH + + + + +## Storage {.small-bullets} + +- **Wynton storage is not backed up** +- /wynton/**[group_name]**/**[user]** + - User home directory - limited to 500 GiB +- /wynton/**[group_name]** + - User group directory - disk quota varies by group + - Use this directory for any analysis you want to share with your lab +- [More information on disk quotas](https://wynton.ucsf.edu/hpc/howto/storage-size.html#file-sizes-and-disk-quotas) + +To check your group disk quota run: + +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +echo 'beegfs-ctl --getquota --storagepoolid=12 --gid "$(id --group)"' +``` + + + +## Scratch - Temporary Storage +- Local **/scratch** - 0.1-1.8 TiB/node storage unique to each compute node + - Can only be accessed from the specific compute node +- **/wynton/scratch** and **/wynton/protected/scratch** (for PHI users) + - 703 TiB storage accessible from everywhere +- No quotas + +

+ +**Files not used for 2 weeks are automatically deleted** + + + +## Gladstone HIVE +- Gladstone's HIVE storage server is mounted directly to Wynton under **/gladstone** + - Only certain HIVE folders are accessible directly on Wynton + - Files under **/gladstone** are backed up +- Naming: **/gladstone/[lab]** + - Directories that are shared between multiple labs can be set up by contacting Gladstone IT +- For more information visit the [IT knowledge base page](https://help.gladstone.org/support/solutions/articles/14000033963) + + +## Storage Advice + +- Always back up anything you store under **/wynton** +- Use **/gladstone** if you have access to it for all of your work since it is automatically backed up +- Use the scratch directories to store temporary files so they do not count against your group or user quotas + + +# Data Transfer + + + +## Secure Copy - scp + +- Local file to Wynton +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +echo "{local}$ scp /path/to/local_file.tsv alice@dt1.wynton.ucsf.edu:/destination/path" +``` + +- Copy a directory to a folder on Wynton +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +echo "{local}$ scp -r local_folder/ alice@dt1.wynton.ucsf.edu:/destination/path" +``` + +- Copy a single file to Wynton from your local machine +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +echo "{local}$ scp alice@dt1.wynton.ucsf.edu:/path/to/local_file.tsv /destination/path" +``` + + +## GUI SFTP Clients {.small-bullets} + +- These let you transfer files to and from Wynton using a GUI +- [2 factor authentication](https://wynton.ucsf.edu/hpc/get-started/duo-signup.html) may be required +- [Cyberduck](https://cyberduck.io/) + - Navigate to Preferences -> Transfers -> General + - change the Transfer Files setting "Use browser connection" instead of "Open Multiple connections" + +- [FileZilla](https://filezilla-project.org/) + - In the General tab, select ‘SFTP’ as the Protocol instead of ‘FTP’ + - For Logon Type, select ‘Interactive’ instead of ‘Ask for Password’ + - Under the Transfer Settings tab, you might need to click the ‘Limit number of simultaneous connections’ and make sure the ‘Maximum number of connections’ is set to 1 + + +## Globus + +- [Globus](https://wynton.ucsf.edu/hpc/transfers/globus.html) is a service for moving, syncing, and sharing large amounts of data +- Wynton Accounts are not required to transfer data with Globus +- Useful for transferring data between institutions + +## Rclone + +- Rclone is a command-line program to manage files on remote storage +- Can be used to transfer data from Wynton directly to [DropBox](https://rclone.org/dropbox/) or other storage systems (AWS, Azure, Google Drive etc.) + - Do this from a data transfer node using screen/tmux +- Do not use rclone for transfers to Box, follow the [Wynton to UCSF Box](https://wynton.ucsf.edu/hpc/transfers/ucsf-box.html) instructions + + + + +## Poll 1 + +Which of these can you **not** log in to from your computer? + +1. Login Nodes +2. Development Nodes +3. Data transfer Nodes +4. Compute Nodes + +## Poll 2 + +The **/wynton** directory is backed up on a nightly basis so do not need to back up the data you store here. + +1. True +2. False + + + + +# Installing Software + +## Basics + +- Check if the tool is already available in a [module](https://wynton.ucsf.edu/hpc/software/software-repositories.html#software-repositories) +- Ensure the software you are trying to install is compatible with Rocky 8 linux (use a container if not) +- Always install software in a development node +- Download a precompiled binary or [install from source](https://wynton.ucsf.edu/hpc/howto/install-from-source.html) + +## Install Samtools from Source {.small-list} + +1. Download and extract source code + +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +echo '[alice@dev1 ~]$ mkdir -p "/scratch/$USER" +[alice@dev1 ~]$ cd "/scratch/$USER" +[alice@dev1 alice]$ wget https://github.com/samtools/samtools/releases/download/1.19.2/samtools-1.19.2.tar.bz2 +[alice@dev1 alice]$ tar -x -f samtools-1.19.2.tar.bz2' +``` + +2. Create install location and configure + +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +echo '[alice@dev1 ~]$ mkdir -p $HOME/software/samtools-1.14' +echo '[alice@dev1 ~]$ ./configure --prefix=$HOME/software/samtools-1.14' +``` +3. Build and install + +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +echo '[alice@dev1 ~]$ make' +echo '[alice@dev1 ~]$ make install' +``` + +## Install Nextflow + +- Scientific workflow system with a community maintained set of [core bioinformatics analysis](https://nf-co.re/) pipelines + - We will cover an example RNA-seq pipeline in part 2 +- These can be configured to use the Wynton compute job submission system + + +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +echo '[alice@dev1 ~]$ cd ~/software' +echo '[alice@dev1 ~]$ curl -s "https://get.sdkman.io" | bash' +echo '[alice@dev1 ~]$ exit' +echo '[alice@log1 ~]$ ssh dev1' +echo '[alice@dev1 ~]$ sdk install java 17.0.6-tem' +echo '[alice@dev1 ~]$ wget -qO- https://get.nextflow.io | bash' +echo '[alice@dev1 ~]$ nextflow -v' +``` + + + + + +# Containers + + +## Motivation {.small-bullets} + +- Compute heavy jobs (high RAM, multiple cores) should be run on compute nodes +- Containers allow us to make additional software available to the compute nodes + - Also allows the use of software that might be hard to install on Rocky 8 Linux + - Improves reproducibility + +![Compute Jobs](slide_materials/compute_job_workflow.svg) + + +## Definitions {.small-bullets} + +- **Virtualization:** When software mimics the functions of physical hardware to run virtual machines +- **Containers:** Implements virtualization using an *image* as its base +- **Images:** An ordered collection of root filesystem changes and the corresponding execution parameters for use within a container runtime + + +## Apptainer {.small-bullets} + +- Wynton supports [Apptainer](https://wynton.ucsf.edu/hpc/software/apptainer.html) (formerly singularity) containers +- [Docker](https://docs.docker.com/) is a commonly used image creation software, these can be turned into apptainer image files (.sif) easily + +- apptainer run + - Run predefined script within container +- apptainer exec + - Execute any command within container +- apptainer shell + - Run bash shell within container + + +## Example Container - Hello World + +- Run this command to convert the public Docker image to a apptainer image file + +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +echo '[alice@dev1 ~]$ apptainer pull docker://natalie23gill/hello-world:1.0' +``` + +- Execute the "hi" command in the container + +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +echo '[alice@dev1 ~]$ apptainer exec hello-world_1.0.sif hi' +``` + +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +echo ' __ __ ____ _ __ __ __ __ + / / / /__ / / /___ | | / /___ _____/ /___/ / / / + / /_/ / _ \/ / / __ \ | | /| / / __ \/ ___/ / __ / / / + / __ / __/ / / /_/ / | |/ |/ / /_/ / / / / /_/ / /_/ +/_/ /_/\___/_/_/\____/ |__/|__/\____/_/ /_/\__,_/ (_) ' +``` + + +## Example Container + +- This container has **figlet** installed which creates ASCII art from text input +- Try running this command to create your own using *exec* + +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +echo '[alice@dev1 ~]$ apptainer exec hello-world_1.0.sif figlet your_text' +``` + +## Docker {.small-bullets} + +- Docker uses Dockerfiles to specify image creation +- Preferred by the Gladstone Bioinformatics Core to create new images +- In part 2, we will go over how to build custom container images from DockerFiles + - If you want to follow along, [install the docker engine](https://docs.docker.com/engine/install/) following the instructions for your OS + - Set up a free [DockerHub](https://hub.docker.com/) account to store your images +- To see the Dockerfile used to create the hello-world image, run: + +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +echo '[alice@dev1 ~]$ apptainer exec hello-world_1.0.sif cat /Dockerfile' +``` + + +# End of Part 1 + +## Thank You! + +- Please take some time to fill out the workshop survey if you are not attending part 2: +[https://www.surveymonkey.com/r/F75J6VZ](https://www.surveymonkey.com/r/F75J6VZ) + + +## Upcoming Data Science Training Program Workshops + + +[Introduction to Linear Mixed Effects Models](https://gladstone.org/events/introduction-linear-mixed-effects-models) +April 25-April 26, 2024 1-3pm PDT + +[Single Cell RNA-Seq Data Analysis](https://gladstone.org/events/single-cell-rna-seq-data-analysis) +April 29-April 30, 2024 9am-4pm PDT + +[Single Cell ATAC-Seq Data Analysis Part 1](https://gladstone.org/events/single-cell-atac-seq-data-analysis-part-1-1) +May 6-May 7, 2024 1-4pm PDT + + +[Complete Schedule](https://gladstone.org/events?series=189) + + diff --git a/working-on-wynton-hpc/Working_on_Wynton_Part_2.Rmd b/working-on-wynton-hpc/Working_on_Wynton_Part_2.Rmd new file mode 100644 index 0000000..341da38 --- /dev/null +++ b/working-on-wynton-hpc/Working_on_Wynton_Part_2.Rmd @@ -0,0 +1,449 @@ +--- +title: "Working on Wynton - Part 2" +author: "Natalie Elphick" +date: "April 16th, 2024" +knit: (function(input, ...) { + rmarkdown::render( + input, + output_dir = "../docs" + ) + }) +output: + revealjs::revealjs_presentation: + theme: simple + highlight: default + css: style.css +--- + +```{r, setup, include=FALSE} + +``` + +## + +
*Press the ? key for tips on navigating these slides*
+ +## Introductions + +**Natalie Elphick** +Bioinformatician I + +**Alex Pico** +Bioinformatics Core Director + + +## Target Audience +- Prior experience with UNIX command-line + + + +## Part 2: + +1. Custom Containers +2. Submitting Compute Jobs +3. Array Jobs +4. GPU Jobs +5. Running Pipelines +6. Jupyter Notebooks +7. RStudio Server +8. How to get help + + + + +# Custom Containers + +## Motivation {.small-bullets} + +- Compute heavy jobs (high RAM, multiple cores) should be run on compute nodes +- Containers allow us to make additional software available to the compute nodes + - Also allows the use of software that might be hard to install on Rocky 8 Linux + - Improves reproducibility + +![Compute Jobs](slide_materials/compute_job_workflow.svg) + + + + +## Dockerfile Basics + +- Dockerfiles contain instructions to build an image in **layers** +- Layers are added using Dockerfile instruction syntax +- Images are built by navigating to the directory that contains the Dockerfile and running: + +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +echo 'docker build .' +``` + +## Dockerfile Instructions {.small-bullets} +- First instruction is always **FROM** which specifies the base image + - Base images are a starting point with some basics already installed like the OS and build tools, find them on [DockerHub](https://hub.docker.com/) +- **RUN** : Use before running any shell commands +- **SHELL** : Set the shell +- **USER** : Set the user (within the image) +- **CMD** : Set the default instruction to be run by the image +- **COPY** : COPY files into the image + + +See the [Dockerfile documentation](https://docs.docker.com/reference/dockerfile/) for a full list of instructions + +## Example Dockerfile {.code-alt} + +- Click [here](https://www.dropbox.com/scl/fi/mdbefp3h8ahdvxtgjypqo/Dockerfile?rlkey=7d4zd9ge1m3wwszlfy78712ky&dl=1) to download the example Dockerfile +- Open in your preffered text editor + + +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +curl -s -L -o Dockerfile 'https://www.dropbox.com/scl/fi/mdbefp3h8ahdvxtgjypqo/Dockerfile?rlkey=7d4zd9ge1m3wwszlfy78712ky&dl=0' +cat Dockerfile +rm Dockerfile +``` + +## Building Example Image + +- Do not run this during the workshop + - It requires a lot of RAM +- On macOS, make sure you have the Docker Desktop App running +- We can provide an additional argument to the **build** command, -t, to set the name of the docker image + - We can add version tags after the name using ":" +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +echo "docker build -t docker_hub_user/seurat-harmony:1.0 ." +``` + + +## Pushing Images to DockerHub {.small-bullets} + +- Make sure you are signed in to your DockerHub account locally (Docker Desktop for macOS) +- The image name must start with your user name + +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +echo "docker push docker_hub_user/seurat-harmony:1.0" +``` + +- These can then be "pulled" on to Wynton as apptainer image files (image must be public) +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +echo "[alice@dev1 ~]$ apptainer pull docker://docker_hub_user/seurat-harmony:1.0" +``` + +## Notes on Building Custom Images {.code-small} + +- Time consuming process and uses a lot of RAM on your local machine +- A good base image can save you a lot of time +- You must run `apt-get update` and `apt-get install` in the same command + - Otherwise you will encounter caching issues +- Remember to use `apt-get install -y` + - You will have no control over the process while it's building + +# Compute Jobs + + +## Submission Script - Basics {.small-bullets .code-alt} + +- [Download](https://www.dropbox.com/scl/fi/fzp33y1ojslw005q8epuz/simple_submission_script.sh?rlkey=xmg3lqec962y3i57a1bkriosx&dl=1) this example job submission script +- Read the full Wynton [job submission guide](https://wynton.ucsf.edu/hpc/scheduler/submit-jobs.html) +- Wynton uses the [Sun Grid Engine](https://web.archive.org/web/20210826212738/https://arc.liv.ac.uk/SGE/howto/howto.html) job scheduler + + + +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +curl -s -L -o submission.sh 'https://www.dropbox.com/scl/fi/fzp33y1ojslw005q8epuz/simple_submission_script.sh?rlkey=xmg3lqec962y3i57a1bkriosx&dl=0' +cat submission.sh +rm submission.sh +``` + +## Submission Script - Apptainer {.small-bullets .code-alt} + +- [Download](https://www.dropbox.com/scl/fi/zzl9fnfcoxu3pyrx5ffd1/apptainer_submission_script.sh?rlkey=w05e18ahw4hvbvaucac379za9&dl=1) this example job submission script that uses a container +- Paths that the container needs read/write access to need to be mounted with APPTAINER_BINDPATH + +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +curl -s -L -o submission.sh 'https://www.dropbox.com/scl/fi/zzl9fnfcoxu3pyrx5ffd1/apptainer_submission_script.sh?rlkey=w05e18ahw4hvbvaucac379za9&dl=1' +cat submission.sh +rm submission.sh +``` + +## Parallel Processing Jobs {.small-bullets} + +- By default jobs run on a single core +- Multicore jobs must run in a SGE parallel environment (PE) and tell SGE how many cores the job will use +- **Do not use more cores than requested** + + +- There are four parallel environments on Wynton: + - **smp**: for single-host parallel jobs using [Symmetric multiprocessing](https://en.wikipedia.org/wiki/Symmetric_multiprocessing) (SMP) + - **mpi**: for multiple-host parallel jobs based on [MPI parallelization](https://en.wikipedia.org/wiki/Message_Passing_Interface) + - **mpi_onehost**: for single-host parallel jobs based on MPI parallelization + - **mpi-8**: for multi-threaded multi-host jobs based on MPI parallelization + +## Example Parallel Job {.small-bullets .code-alt} + +- The simplest parallel environment on Wynton is **smp**, a single node with *n* cores +- [Download](https://www.dropbox.com/scl/fi/71xo0cioh266pj3uwcdps/smp_submission_script.sh?rlkey=kw7qaz8pip6jveqv317b5swqr&dl=1) this example smp job submission script +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +curl -s -L -o submission.sh 'https://www.dropbox.com/scl/fi/71xo0cioh266pj3uwcdps/smp_submission_script.sh?rlkey=kw7qaz8pip6jveqv317b5swqr&dl=0' +cat submission.sh +rm submission.sh +``` + + +## Array Jobs {.small-bullets .code-alt} + +- This is a good option if the script you want to run operates on discrete sets of data + - e.g. sample or chromosome +- [Download](https://www.dropbox.com/scl/fi/upl71jeny62fxfzkxao1f/array_job_submission_script.sh?rlkey=ggkyjxx8nz400e1t96mif5t34&dl=1) this example array job submission script + +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +curl -s -L -o submission.sh 'https://www.dropbox.com/scl/fi/upl71jeny62fxfzkxao1f/array_job_submission_script.sh?rlkey=ggkyjxx8nz400e1t96mif5t34&dl=0' +cat submission.sh +rm submission.sh +``` + +## GPU Jobs {.small-bullets} + +- To run a [GPU job](https://wynton.ucsf.edu/hpc/scheduler/gpu.html), specify **-q gpu.q** (queue) as a GPU queue + - Other GPU queues may be available to you depending on your lab +- It is important to specify the GPU using the **SGE_GPU** variable so that your job uses its assigned GPU + - For CUDA based tools, add **export CUDA_VISIBLE_DEVICES=$SGE_GPU** to your submission script +- GPU jobs must include a runtime request or they will be removed from the queue + + +## Submitting and Querying jobs + +- Use **qsub** to submit jobs +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +echo '[alice@dev1 ~]$ qsub job1.sh +Your job 714888 ("job1.sh") has been submitted' +``` + + + + +- Use **qstat** to check the status of your jobs + +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +echo '[alice@dev1 ~]$ qstat +job-ID prior name user state submit/start at queue slots ja-task-ID +----------------------------------------------------------------------------------------------------------------- + 714888 0.06532 job1 alice r 03/25/2024 19:54:18 member.q@msg-hmio1 1 + 714889 0.06532 job2 alice r 03/25/2024 19:54:19 member.q@msg-hmio1 1 +' +``` + + +Read the [querying jobs](https://wynton.ucsf.edu/hpc/scheduler/list-jobs.html) Wynton documentation for more information. + + +## Estimating Job Resources + +- Try to estimate the amount of RAM needed using a small test dataset +- Request a little more RAM than you need to avoid having your job cancelled +- Check on jobs you are running for the first time with **qstat -j ** to make sure they are not going over + + +## Poll 3 + +Anything that you can run on a compute node can be run on a development node. + +1. True +2. False + + + +# Running Pipelines + + +## Nextflow RNA-seq {.small-bullets .big-picture} + +- Scientific workflow system with a community maintained set of core bioinformatics [analysis pipelines](https://nf-co.re/) +- The most commonly used one is the [RNA-seq pipeline](https://nf-co.re/rnaseq/3.14.0) + +![RNA-seq](slide_materials/nf-core-rnaseq_metro_map_grey.png) + + +## Example - RNA-seq Pipeline {.small-bullets} + +**Do not run this during the workshop as it will fill up the Wynton SGE queue** + +- Download the [testing script](https://www.dropbox.com/scl/fi/3c9qdmnwg8vw9x517mo05/nextflow_example.sh?rlkey=e9nxbvpcdtdyi5w0y16z9k7bq&dl=0) + - Runs a minimal test on the RNA-seq pipeline +- Download the [config file](https://www.dropbox.com/scl/fi/befhl3z6nipn1fqcxpqth/nextflow.config?rlkey=pd8d9vup6pnvb7bbrmekayn2j&dl=0) + - Configures nextflow to use the SGE job scheduler and sets limits on compute job resources for each process +- Put these in the same directory (do not use your user home directory for this) and run the script in a screen/tmux session +- When not running the test, the **-profile** should be apptainer + + + + + + + +# Jupyter Notebooks + +## Installing Jupyter Notebooks +- The preferred way to install and use [Jupyter notebooks](https://wynton.ucsf.edu/hpc/howto/jupyter.html) on Wynton is though pip, not conda +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +echo 'python3 -m pip install --user notebook' +``` +- Jupyter notebooks can only be run on development nodes +- See the Wynton [python documentation](https://wynton.ucsf.edu/hpc/howto/python.html) for more info on managing python environments on Wynton + + +## Running Jupyter Notebooks - Step 1 + +- You cannot connect from outside Wynton HPC directly to a development node + - Instead we need to use SSH port forwarding to establish the connection with a local web browser +- Find an available TCP port: +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +echo '[alice@dev1 ~]$ module load CBI port4me +[alice@dev1 ~]$ port4me --tool=jupyter +47467' +``` + +Note the port number returned by port4me, you will need this later. + + +## Running Jupyter Notebooks - Step 2 {.code-small} +- Launch Jupyter notebook using the port numer from step 1 +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +echo '[alice@dev1]$ jupyter notebook --no-browser --port 47467 +[I 2024-03-20 14:48:45.693 ServerApp] jupyter_lsp | extension was successfully linked. +[I 2024-03-20 14:48:45.698 ServerApp] jupyter_server_terminals | extension was successfully linked. +[I 2024-03-20 14:48:45.703 ServerApp] jupyterlab | extension was successfully linked. +[I 2024-03-20 14:48:45.708 ServerApp] notebook | extension was successfully linked. +[I 2024-03-20 14:48:46.577 ServerApp] notebook_shim | extension was successfully linked. +[I 2024-03-20 14:48:46.666 ServerApp] notebook_shim | extension was successfully loaded. +[I 2024-03-20 14:48:46.668 ServerApp] jupyter_lsp | extension was successfully loaded. +[I 2024-03-20 14:48:46.669 ServerApp] jupyter_server_terminals | extension was successfully loaded. +[I 2024-03-20 14:48:46.675 LabApp] JupyterLab extension loaded from /wynton/home/boblab/alice/.local/lib/python3.11/site-packages/jupyterlab +[I 2024-03-20 14:48:46.675 LabApp] JupyterLab application directory is /wynton/home/boblab/alice/.local/share/jupyter/lab +[I 2024-03-20 14:48:46.677 LabApp] Extension Manager is 'pypi'. +[I 2024-03-20 14:48:46.707 ServerApp] jupyterlab | extension was successfully loaded. +[I 2024-03-20 14:48:46.711 ServerApp] notebook | extension was successfully loaded. +[I 2024-03-20 14:48:46.712 ServerApp] Serving notebooks from local directory: /wynton/home/boblab/alice +[I 2024-03-20 14:48:46.712 ServerApp] Jupyter Server 2.13.0 is running at: +[I 2024-03-20 14:48:46.712 ServerApp] http://localhost:44214/tree?token=8e37f8d62fca6a1c9b2da429f27df5ebcec706a808c3a8f2 +[I 2024-03-20 14:48:46.712 ServerApp] http://127.0.0.1:44214/tree?token=8e37f8d62fca6a1c9b2da429f27df5ebcec706a808c3a8f2 +[I 2024-03-20 14:48:46.712 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). +[C 2024-03-20 14:48:46.725 ServerApp] + + To access the server, open this file in a browser: + file:///wynton/home/boblab/alice/.local/share/jupyter/runtime/jpserver-2853162-open.html + Or copy and paste one of these URLs: + http://localhost:44214/tree?token=8e37f8d62fca6a1c9b2da429f27df5ebcec706a808c3a8f2 + http://127.0.0.1:44214/tree?token=8e37f8d62fca6a1c9b2da429f27df5ebcec706a808c3a8f2' +``` + +## Running Jupyter Notebooks - Step 3 + +- Set up SSH port forwarding on your local machine in a separate terminal, leave both terminals open +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +echo '{local}$ ssh -J alice@log1.wynton.ucsf.edu -L 47467:localhost:47467 alice@dev1 +... +[alice@dev1 ~]$ ' +``` + +The notebook should now be available at the URL from step 2 + + +# RStudio Server + +## RStudio Server +- [RStudio server](https://wynton.ucsf.edu/hpc/howto/rstudio.html) is already available in the CBI module +- This allows you to set up a personal RStudio instance that only you can access +- Requires two separate SSH connections to the cluster:\ + - One to launch RStudio Server + - One to connect to it + + +## RStudio Server - Step 1 {.code-small} + +- Launch your own RStudio Server instance +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +echo '[alice@dev1 ~]$ module load CBI rstudio-server-controller +[alice@dev1 ~]$ rsc start +alice, your personal RStudio Server 2023.09.1-494 running R 4.3.2 is available on: + + + +Importantly, if you are running from a remote machine without direct access +to dev1, you need to set up SSH port forwarding first, which you can do by +running: + + ssh -L 20612:dev1:20612 alice@log1.wynton.ucsf.edu + +in a second terminal from your local computer. + +Any R session started times out after being idle for 120 minutes. +WARNING: You now have 10 minutes, until 2023-11-15 17:06:50-08:00, to +connect and log in to the RStudio Server before everything times out. +Your one-time random password for RStudio Server is: y+IWo7rfl7Z7MRCPI3Z4' +``` + +Note the password and URL, they will be needed to log in to the server instance. + + +## RStudio Server - Step 2 + +- Connect to your personal RStudio Server instance from your local machine in a separate terminal +```{r, engine='bash', eval=TRUE, results='markup',comment=NA, highlight=TRUE, echo=FALSE} +echo '{local}$ ssh -L 20612:dev1:20612 alice@log1.wynton.ucsf.edu +alice1@log1.wynton.ucsf.edu:s password: XXXXXXXXXXXXXXXXXXX +[alice@log1 ~]$ ' +``` + +## RStudio Server - Step 3 +- Open RStudio Server in your local web browser +- Open the link from step 1 +- Enter your Wynton user name +- Enter the password from step 1 + +# How to Get Help + +## Wynton Questions + +- Follow the Wynton [question checklist](https://wynton.ucsf.edu/hpc/support/index.html) +- Email + - [support@wynton.ucsf.edu](mailto:support@wynton.ucsf.edu) +- Slack + - [ucsf-wynton](https://join.slack.com/t/ucsf-wynton/signup) + - Sign-up using a UCSF email address + - Email support if that does not work +- Zoom office hours every **Tuesday at 11-12pm** + - Zoom URL in the message-of-the-day (MOTD) that you get when you log into Wynton + + + +## Bioinformatics Questions + +- Email + - [bioinformatics@gladstone.ucsf.edu](mailto:bioinformatics@gladstone.ucsf.edu) +- Slack channel #questions-about-bioinformatics + - Contact us at the email above to be added to the channel + + + +# End of Part 2 + +## Thank You! + +- Please take some time to fill out the workshop survey: +[https://www.surveymonkey.com/r/F75J6VZ](https://www.surveymonkey.com/r/F75J6VZ) + + +## Upcoming Data Science Training Program Workshops + + +[Introduction to Linear Mixed Effects Models](https://gladstone.org/events/introduction-linear-mixed-effects-models) +April 25-April 26, 2024 1-3pm PDT + +[Single Cell RNA-Seq Data Analysis](https://gladstone.org/events/single-cell-rna-seq-data-analysis) +April 29-April 30, 2024 9am-4pm PDT + +[Single Cell ATAC-Seq Data Analysis Part 1](https://gladstone.org/events/single-cell-atac-seq-data-analysis-part-1-1) +May 6-May 7, 2024 1-4pm PDT + + +[Complete Schedule](https://gladstone.org/events?series=189) + + + diff --git a/working-on-wynton-hpc/renv.lock b/working-on-wynton-hpc/renv.lock new file mode 100644 index 0000000..a000798 --- /dev/null +++ b/working-on-wynton-hpc/renv.lock @@ -0,0 +1,1551 @@ +{ + "R": { + "Version": "4.3.2", + "Repositories": [ + { + "Name": "CRAN", + "URL": "https://packagemanager.posit.co/cran/latest" + } + ] + }, + "Packages": { + "DBI": { + "Package": "DBI", + "Version": "1.2.2", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "methods" + ], + "Hash": "164809cd72e1d5160b4cb3aa57f510fe" + }, + "MASS": { + "Package": "MASS", + "Version": "7.3-60.0.1", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "grDevices", + "graphics", + "methods", + "stats", + "utils" + ], + "Hash": "b765b28387acc8ec9e9c1530713cb19c" + }, + "Matrix": { + "Package": "Matrix", + "Version": "1.6-5", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "grDevices", + "graphics", + "grid", + "lattice", + "methods", + "stats", + "utils" + ], + "Hash": "8c7115cd3a0e048bda2a7cd110549f7a" + }, + "R6": { + "Package": "R6", + "Version": "2.5.1", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R" + ], + "Hash": "470851b6d5d0ac559e9d01bb352b4021" + }, + "RColorBrewer": { + "Package": "RColorBrewer", + "Version": "1.1-3", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R" + ], + "Hash": "45f0398006e83a5b10b72a90663d8d8c" + }, + "askpass": { + "Package": "askpass", + "Version": "1.2.0", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "sys" + ], + "Hash": "cad6cf7f1d5f6e906700b9d3e718c796" + }, + "backports": { + "Package": "backports", + "Version": "1.4.1", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R" + ], + "Hash": "c39fbec8a30d23e721980b8afb31984c" + }, + "base64enc": { + "Package": "base64enc", + "Version": "0.1-3", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R" + ], + "Hash": "543776ae6848fde2f48ff3816d0628bc" + }, + "bit": { + "Package": "bit", + "Version": "4.0.5", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R" + ], + "Hash": "d242abec29412ce988848d0294b208fd" + }, + "bit64": { + "Package": "bit64", + "Version": "4.0.5", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "bit", + "methods", + "stats", + "utils" + ], + "Hash": "9fe98599ca456d6552421db0d6772d8f" + }, + "blob": { + "Package": "blob", + "Version": "1.2.4", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "methods", + "rlang", + "vctrs" + ], + "Hash": "40415719b5a479b87949f3aa0aee737c" + }, + "broom": { + "Package": "broom", + "Version": "1.0.5", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "backports", + "dplyr", + "ellipsis", + "generics", + "glue", + "lifecycle", + "purrr", + "rlang", + "stringr", + "tibble", + "tidyr" + ], + "Hash": "fd25391c3c4f6ecf0fa95f1e6d15378c" + }, + "bslib": { + "Package": "bslib", + "Version": "0.6.1", + "Source": "Repository", + "Repository": "RSPM", + "Requirements": [ + "R", + "base64enc", + "cachem", + "grDevices", + "htmltools", + "jquerylib", + "jsonlite", + "lifecycle", + "memoise", + "mime", + "rlang", + "sass" + ], + "Hash": "c0d8599494bc7fb408cd206bbdd9cab0" + }, + "cachem": { + "Package": "cachem", + "Version": "1.0.8", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "fastmap", + "rlang" + ], + "Hash": "c35768291560ce302c0a6589f92e837d" + }, + "callr": { + "Package": "callr", + "Version": "3.7.5", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "R6", + "processx", + "utils" + ], + "Hash": "9f0e4fae4963ba775a5e5c520838c87b" + }, + "cellranger": { + "Package": "cellranger", + "Version": "1.1.0", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "rematch", + "tibble" + ], + "Hash": "f61dbaec772ccd2e17705c1e872e9e7c" + }, + "cli": { + "Package": "cli", + "Version": "3.6.2", + "Source": "Repository", + "Repository": "RSPM", + "Requirements": [ + "R", + "utils" + ], + "Hash": "1216ac65ac55ec0058a6f75d7ca0fd52" + }, + "clipr": { + "Package": "clipr", + "Version": "0.8.0", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "utils" + ], + "Hash": "3f038e5ac7f41d4ac41ce658c85e3042" + }, + "colorspace": { + "Package": "colorspace", + "Version": "2.1-0", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "grDevices", + "graphics", + "methods", + "stats" + ], + "Hash": "f20c47fd52fae58b4e377c37bb8c335b" + }, + "conflicted": { + "Package": "conflicted", + "Version": "1.2.0", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "cli", + "memoise", + "rlang" + ], + "Hash": "bb097fccb22d156624fd07cd2894ddb6" + }, + "countdown": { + "Package": "countdown", + "Version": "0.4.0", + "Source": "Repository", + "Repository": "RSPM", + "Requirements": [ + "htmltools", + "prismatic", + "utils", + "whisker" + ], + "Hash": "b44ad5a5e287637c3b8ca0e9bb64bb18" + }, + "cpp11": { + "Package": "cpp11", + "Version": "0.4.7", + "Source": "Repository", + "Repository": "RSPM", + "Requirements": [ + "R" + ], + "Hash": "5a295d7d963cc5035284dcdbaf334f4e" + }, + "crayon": { + "Package": "crayon", + "Version": "1.5.2", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "grDevices", + "methods", + "utils" + ], + "Hash": "e8a1e41acf02548751f45c718d55aa6a" + }, + "curl": { + "Package": "curl", + "Version": "5.2.1", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R" + ], + "Hash": "411ca2c03b1ce5f548345d2fc2685f7a" + }, + "data.table": { + "Package": "data.table", + "Version": "1.15.2", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "methods" + ], + "Hash": "536dfe4ac4093b5d115caed7a1a7223b" + }, + "dbplyr": { + "Package": "dbplyr", + "Version": "2.5.0", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "DBI", + "R", + "R6", + "blob", + "cli", + "dplyr", + "glue", + "lifecycle", + "magrittr", + "methods", + "pillar", + "purrr", + "rlang", + "tibble", + "tidyr", + "tidyselect", + "utils", + "vctrs", + "withr" + ], + "Hash": "39b2e002522bfd258039ee4e889e0fd1" + }, + "digest": { + "Package": "digest", + "Version": "0.6.35", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "utils" + ], + "Hash": "698ece7ba5a4fa4559e3d537e7ec3d31" + }, + "dplyr": { + "Package": "dplyr", + "Version": "1.1.4", + "Source": "Repository", + "Repository": "RSPM", + "Requirements": [ + "R", + "R6", + "cli", + "generics", + "glue", + "lifecycle", + "magrittr", + "methods", + "pillar", + "rlang", + "tibble", + "tidyselect", + "utils", + "vctrs" + ], + "Hash": "fedd9d00c2944ff00a0e2696ccf048ec" + }, + "dtplyr": { + "Package": "dtplyr", + "Version": "1.3.1", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "cli", + "data.table", + "dplyr", + "glue", + "lifecycle", + "rlang", + "tibble", + "tidyselect", + "vctrs" + ], + "Hash": "54ed3ea01b11e81a86544faaecfef8e2" + }, + "ellipsis": { + "Package": "ellipsis", + "Version": "0.3.2", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "rlang" + ], + "Hash": "bb0eec2fe32e88d9e2836c2f73ea2077" + }, + "evaluate": { + "Package": "evaluate", + "Version": "0.23", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "methods" + ], + "Hash": "daf4a1246be12c1fa8c7705a0935c1a0" + }, + "fansi": { + "Package": "fansi", + "Version": "1.0.6", + "Source": "Repository", + "Repository": "RSPM", + "Requirements": [ + "R", + "grDevices", + "utils" + ], + "Hash": "962174cf2aeb5b9eea581522286a911f" + }, + "farver": { + "Package": "farver", + "Version": "2.1.1", + "Source": "Repository", + "Repository": "CRAN", + "Hash": "8106d78941f34855c440ddb946b8f7a5" + }, + "fastmap": { + "Package": "fastmap", + "Version": "1.1.1", + "Source": "Repository", + "Repository": "CRAN", + "Hash": "f7736a18de97dea803bde0a2daaafb27" + }, + "fontawesome": { + "Package": "fontawesome", + "Version": "0.5.2", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "htmltools", + "rlang" + ], + "Hash": "c2efdd5f0bcd1ea861c2d4e2a883a67d" + }, + "forcats": { + "Package": "forcats", + "Version": "1.0.0", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "cli", + "glue", + "lifecycle", + "magrittr", + "rlang", + "tibble" + ], + "Hash": "1a0a9a3d5083d0d573c4214576f1e690" + }, + "fs": { + "Package": "fs", + "Version": "1.6.3", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "methods" + ], + "Hash": "47b5f30c720c23999b913a1a635cf0bb" + }, + "gargle": { + "Package": "gargle", + "Version": "1.5.2", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "cli", + "fs", + "glue", + "httr", + "jsonlite", + "lifecycle", + "openssl", + "rappdirs", + "rlang", + "stats", + "utils", + "withr" + ], + "Hash": "fc0b272e5847c58cd5da9b20eedbd026" + }, + "generics": { + "Package": "generics", + "Version": "0.1.3", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "methods" + ], + "Hash": "15e9634c0fcd294799e9b2e929ed1b86" + }, + "ggplot2": { + "Package": "ggplot2", + "Version": "3.5.0", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "MASS", + "R", + "cli", + "glue", + "grDevices", + "grid", + "gtable", + "isoband", + "lifecycle", + "mgcv", + "rlang", + "scales", + "stats", + "tibble", + "vctrs", + "withr" + ], + "Hash": "52ef83f93f74833007f193b2d4c159a2" + }, + "glue": { + "Package": "glue", + "Version": "1.7.0", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "methods" + ], + "Hash": "e0b3a53876554bd45879e596cdb10a52" + }, + "googledrive": { + "Package": "googledrive", + "Version": "2.1.1", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "cli", + "gargle", + "glue", + "httr", + "jsonlite", + "lifecycle", + "magrittr", + "pillar", + "purrr", + "rlang", + "tibble", + "utils", + "uuid", + "vctrs", + "withr" + ], + "Hash": "e99641edef03e2a5e87f0a0b1fcc97f4" + }, + "googlesheets4": { + "Package": "googlesheets4", + "Version": "1.1.1", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "cellranger", + "cli", + "curl", + "gargle", + "glue", + "googledrive", + "httr", + "ids", + "lifecycle", + "magrittr", + "methods", + "purrr", + "rematch2", + "rlang", + "tibble", + "utils", + "vctrs", + "withr" + ], + "Hash": "d6db1667059d027da730decdc214b959" + }, + "gtable": { + "Package": "gtable", + "Version": "0.3.4", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "cli", + "glue", + "grid", + "lifecycle", + "rlang" + ], + "Hash": "b29cf3031f49b04ab9c852c912547eef" + }, + "haven": { + "Package": "haven", + "Version": "2.5.4", + "Source": "Repository", + "Repository": "RSPM", + "Requirements": [ + "R", + "cli", + "cpp11", + "forcats", + "hms", + "lifecycle", + "methods", + "readr", + "rlang", + "tibble", + "tidyselect", + "vctrs" + ], + "Hash": "9171f898db9d9c4c1b2c745adc2c1ef1" + }, + "highr": { + "Package": "highr", + "Version": "0.10", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "xfun" + ], + "Hash": "06230136b2d2b9ba5805e1963fa6e890" + }, + "hms": { + "Package": "hms", + "Version": "1.1.3", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "lifecycle", + "methods", + "pkgconfig", + "rlang", + "vctrs" + ], + "Hash": "b59377caa7ed00fa41808342002138f9" + }, + "htmltools": { + "Package": "htmltools", + "Version": "0.5.7", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "base64enc", + "digest", + "ellipsis", + "fastmap", + "grDevices", + "rlang", + "utils" + ], + "Hash": "2d7b3857980e0e0d0a1fd6f11928ab0f" + }, + "httr": { + "Package": "httr", + "Version": "1.4.7", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "R6", + "curl", + "jsonlite", + "mime", + "openssl" + ], + "Hash": "ac107251d9d9fd72f0ca8049988f1d7f" + }, + "ids": { + "Package": "ids", + "Version": "1.0.1", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "openssl", + "uuid" + ], + "Hash": "99df65cfef20e525ed38c3d2577f7190" + }, + "isoband": { + "Package": "isoband", + "Version": "0.2.7", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "grid", + "utils" + ], + "Hash": "0080607b4a1a7b28979aecef976d8bc2" + }, + "jquerylib": { + "Package": "jquerylib", + "Version": "0.1.4", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "htmltools" + ], + "Hash": "5aab57a3bd297eee1c1d862735972182" + }, + "jsonlite": { + "Package": "jsonlite", + "Version": "1.8.8", + "Source": "Repository", + "Repository": "RSPM", + "Requirements": [ + "methods" + ], + "Hash": "e1b9c55281c5adc4dd113652d9e26768" + }, + "knitr": { + "Package": "knitr", + "Version": "1.45", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "evaluate", + "highr", + "methods", + "tools", + "xfun", + "yaml" + ], + "Hash": "1ec462871063897135c1bcbe0fc8f07d" + }, + "labeling": { + "Package": "labeling", + "Version": "0.4.3", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "graphics", + "stats" + ], + "Hash": "b64ec208ac5bc1852b285f665d6368b3" + }, + "lattice": { + "Package": "lattice", + "Version": "0.22-6", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "grDevices", + "graphics", + "grid", + "stats", + "utils" + ], + "Hash": "cc5ac1ba4c238c7ca9fa6a87ca11a7e2" + }, + "lifecycle": { + "Package": "lifecycle", + "Version": "1.0.4", + "Source": "Repository", + "Repository": "RSPM", + "Requirements": [ + "R", + "cli", + "glue", + "rlang" + ], + "Hash": "b8552d117e1b808b09a832f589b79035" + }, + "lubridate": { + "Package": "lubridate", + "Version": "1.9.3", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "generics", + "methods", + "timechange" + ], + "Hash": "680ad542fbcf801442c83a6ac5a2126c" + }, + "magrittr": { + "Package": "magrittr", + "Version": "2.0.3", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R" + ], + "Hash": "7ce2733a9826b3aeb1775d56fd305472" + }, + "memoise": { + "Package": "memoise", + "Version": "2.0.1", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "cachem", + "rlang" + ], + "Hash": "e2817ccf4a065c5d9d7f2cfbe7c1d78c" + }, + "mgcv": { + "Package": "mgcv", + "Version": "1.9-1", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "Matrix", + "R", + "graphics", + "methods", + "nlme", + "splines", + "stats", + "utils" + ], + "Hash": "110ee9d83b496279960e162ac97764ce" + }, + "mime": { + "Package": "mime", + "Version": "0.12", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "tools" + ], + "Hash": "18e9c28c1d3ca1560ce30658b22ce104" + }, + "modelr": { + "Package": "modelr", + "Version": "0.1.11", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "broom", + "magrittr", + "purrr", + "rlang", + "tibble", + "tidyr", + "tidyselect", + "vctrs" + ], + "Hash": "4f50122dc256b1b6996a4703fecea821" + }, + "munsell": { + "Package": "munsell", + "Version": "0.5.0", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "colorspace", + "methods" + ], + "Hash": "6dfe8bf774944bd5595785e3229d8771" + }, + "nlme": { + "Package": "nlme", + "Version": "3.1-164", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "graphics", + "lattice", + "stats", + "utils" + ], + "Hash": "a623a2239e642806158bc4dc3f51565d" + }, + "openssl": { + "Package": "openssl", + "Version": "2.1.1", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "askpass" + ], + "Hash": "2a0dc8c6adfb6f032e4d4af82d258ab5" + }, + "pillar": { + "Package": "pillar", + "Version": "1.9.0", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "cli", + "fansi", + "glue", + "lifecycle", + "rlang", + "utf8", + "utils", + "vctrs" + ], + "Hash": "15da5a8412f317beeee6175fbc76f4bb" + }, + "pkgconfig": { + "Package": "pkgconfig", + "Version": "2.0.3", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "utils" + ], + "Hash": "01f28d4278f15c76cddbea05899c5d6f" + }, + "prettyunits": { + "Package": "prettyunits", + "Version": "1.2.0", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R" + ], + "Hash": "6b01fc98b1e86c4f705ce9dcfd2f57c7" + }, + "prismatic": { + "Package": "prismatic", + "Version": "1.1.1", + "Source": "Repository", + "Repository": "RSPM", + "Requirements": [ + "R", + "farver", + "grDevices", + "graphics" + ], + "Hash": "faa2193fdec94a45b4390aefc1280a62" + }, + "processx": { + "Package": "processx", + "Version": "3.8.4", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "R6", + "ps", + "utils" + ], + "Hash": "0c90a7d71988856bad2a2a45dd871bb9" + }, + "progress": { + "Package": "progress", + "Version": "1.2.3", + "Source": "Repository", + "Repository": "RSPM", + "Requirements": [ + "R", + "R6", + "crayon", + "hms", + "prettyunits" + ], + "Hash": "f4625e061cb2865f111b47ff163a5ca6" + }, + "ps": { + "Package": "ps", + "Version": "1.7.6", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "utils" + ], + "Hash": "dd2b9319ee0656c8acf45c7f40c59de7" + }, + "purrr": { + "Package": "purrr", + "Version": "1.0.2", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "cli", + "lifecycle", + "magrittr", + "rlang", + "vctrs" + ], + "Hash": "1cba04a4e9414bdefc9dcaa99649a8dc" + }, + "ragg": { + "Package": "ragg", + "Version": "1.3.0", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "systemfonts", + "textshaping" + ], + "Hash": "082e1a198e3329d571f4448ef0ede4bc" + }, + "rappdirs": { + "Package": "rappdirs", + "Version": "0.3.3", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R" + ], + "Hash": "5e3c5dc0b071b21fa128676560dbe94d" + }, + "readr": { + "Package": "readr", + "Version": "2.1.5", + "Source": "Repository", + "Repository": "RSPM", + "Requirements": [ + "R", + "R6", + "cli", + "clipr", + "cpp11", + "crayon", + "hms", + "lifecycle", + "methods", + "rlang", + "tibble", + "tzdb", + "utils", + "vroom" + ], + "Hash": "9de96463d2117f6ac49980577939dfb3" + }, + "readxl": { + "Package": "readxl", + "Version": "1.4.3", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "cellranger", + "cpp11", + "progress", + "tibble", + "utils" + ], + "Hash": "8cf9c239b96df1bbb133b74aef77ad0a" + }, + "rematch": { + "Package": "rematch", + "Version": "2.0.0", + "Source": "Repository", + "Repository": "CRAN", + "Hash": "cbff1b666c6fa6d21202f07e2318d4f1" + }, + "rematch2": { + "Package": "rematch2", + "Version": "2.1.2", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "tibble" + ], + "Hash": "76c9e04c712a05848ae7a23d2f170a40" + }, + "renv": { + "Package": "renv", + "Version": "1.0.5", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "utils" + ], + "Hash": "32c3f93e8360f667ca5863272ec8ba6a" + }, + "reprex": { + "Package": "reprex", + "Version": "2.1.0", + "Source": "Repository", + "Repository": "RSPM", + "Requirements": [ + "R", + "callr", + "cli", + "clipr", + "fs", + "glue", + "knitr", + "lifecycle", + "rlang", + "rmarkdown", + "rstudioapi", + "utils", + "withr" + ], + "Hash": "1425f91b4d5d9a8f25352c44a3d914ed" + }, + "revealjs": { + "Package": "revealjs", + "Version": "0.9", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "rmarkdown" + ], + "Hash": "08e30caa337e6d335d83df1a05d5b9f3" + }, + "rlang": { + "Package": "rlang", + "Version": "1.1.3", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "utils" + ], + "Hash": "42548638fae05fd9a9b5f3f437fbbbe2" + }, + "rmarkdown": { + "Package": "rmarkdown", + "Version": "2.26", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "bslib", + "evaluate", + "fontawesome", + "htmltools", + "jquerylib", + "jsonlite", + "knitr", + "methods", + "tinytex", + "tools", + "utils", + "xfun", + "yaml" + ], + "Hash": "9b148e7f95d33aac01f31282d49e4f44" + }, + "rstudioapi": { + "Package": "rstudioapi", + "Version": "0.15.0", + "Source": "Repository", + "Repository": "CRAN", + "Hash": "5564500e25cffad9e22244ced1379887" + }, + "rvest": { + "Package": "rvest", + "Version": "1.0.4", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "cli", + "glue", + "httr", + "lifecycle", + "magrittr", + "rlang", + "selectr", + "tibble", + "xml2" + ], + "Hash": "0bcf0c6f274e90ea314b812a6d19a519" + }, + "sass": { + "Package": "sass", + "Version": "0.4.9", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R6", + "fs", + "htmltools", + "rappdirs", + "rlang" + ], + "Hash": "d53dbfddf695303ea4ad66f86e99b95d" + }, + "scales": { + "Package": "scales", + "Version": "1.3.0", + "Source": "Repository", + "Repository": "RSPM", + "Requirements": [ + "R", + "R6", + "RColorBrewer", + "cli", + "farver", + "glue", + "labeling", + "lifecycle", + "munsell", + "rlang", + "viridisLite" + ], + "Hash": "c19df082ba346b0ffa6f833e92de34d1" + }, + "selectr": { + "Package": "selectr", + "Version": "0.4-2", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "R6", + "methods", + "stringr" + ], + "Hash": "3838071b66e0c566d55cc26bd6e27bf4" + }, + "stringi": { + "Package": "stringi", + "Version": "1.8.3", + "Source": "Repository", + "Repository": "RSPM", + "Requirements": [ + "R", + "stats", + "tools", + "utils" + ], + "Hash": "058aebddea264f4c99401515182e656a" + }, + "stringr": { + "Package": "stringr", + "Version": "1.5.1", + "Source": "Repository", + "Repository": "RSPM", + "Requirements": [ + "R", + "cli", + "glue", + "lifecycle", + "magrittr", + "rlang", + "stringi", + "vctrs" + ], + "Hash": "960e2ae9e09656611e0b8214ad543207" + }, + "sys": { + "Package": "sys", + "Version": "3.4.2", + "Source": "Repository", + "Repository": "CRAN", + "Hash": "3a1be13d68d47a8cd0bfd74739ca1555" + }, + "systemfonts": { + "Package": "systemfonts", + "Version": "1.0.6", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "cpp11" + ], + "Hash": "6d538cff441f0f1f36db2209ac7495ac" + }, + "textshaping": { + "Package": "textshaping", + "Version": "0.3.7", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "cpp11", + "systemfonts" + ], + "Hash": "997aac9ad649e0ef3b97f96cddd5622b" + }, + "tibble": { + "Package": "tibble", + "Version": "3.2.1", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "fansi", + "lifecycle", + "magrittr", + "methods", + "pillar", + "pkgconfig", + "rlang", + "utils", + "vctrs" + ], + "Hash": "a84e2cc86d07289b3b6f5069df7a004c" + }, + "tidyr": { + "Package": "tidyr", + "Version": "1.3.1", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "cli", + "cpp11", + "dplyr", + "glue", + "lifecycle", + "magrittr", + "purrr", + "rlang", + "stringr", + "tibble", + "tidyselect", + "utils", + "vctrs" + ], + "Hash": "915fb7ce036c22a6a33b5a8adb712eb1" + }, + "tidyselect": { + "Package": "tidyselect", + "Version": "1.2.1", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "cli", + "glue", + "lifecycle", + "rlang", + "vctrs", + "withr" + ], + "Hash": "829f27b9c4919c16b593794a6344d6c0" + }, + "tidyverse": { + "Package": "tidyverse", + "Version": "2.0.0", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "broom", + "cli", + "conflicted", + "dbplyr", + "dplyr", + "dtplyr", + "forcats", + "ggplot2", + "googledrive", + "googlesheets4", + "haven", + "hms", + "httr", + "jsonlite", + "lubridate", + "magrittr", + "modelr", + "pillar", + "purrr", + "ragg", + "readr", + "readxl", + "reprex", + "rlang", + "rstudioapi", + "rvest", + "stringr", + "tibble", + "tidyr", + "xml2" + ], + "Hash": "c328568cd14ea89a83bd4ca7f54ae07e" + }, + "timechange": { + "Package": "timechange", + "Version": "0.3.0", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "cpp11" + ], + "Hash": "c5f3c201b931cd6474d17d8700ccb1c8" + }, + "tinytex": { + "Package": "tinytex", + "Version": "0.50", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "xfun" + ], + "Hash": "be7a76845222ad20adb761f462eed3ea" + }, + "tzdb": { + "Package": "tzdb", + "Version": "0.4.0", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "cpp11" + ], + "Hash": "f561504ec2897f4d46f0c7657e488ae1" + }, + "utf8": { + "Package": "utf8", + "Version": "1.2.4", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R" + ], + "Hash": "62b65c52671e6665f803ff02954446e9" + }, + "uuid": { + "Package": "uuid", + "Version": "1.2-0", + "Source": "Repository", + "Repository": "RSPM", + "Requirements": [ + "R" + ], + "Hash": "303c19bfd970bece872f93a824e323d9" + }, + "vctrs": { + "Package": "vctrs", + "Version": "0.6.5", + "Source": "Repository", + "Repository": "RSPM", + "Requirements": [ + "R", + "cli", + "glue", + "lifecycle", + "rlang" + ], + "Hash": "c03fa420630029418f7e6da3667aac4a" + }, + "viridisLite": { + "Package": "viridisLite", + "Version": "0.4.2", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R" + ], + "Hash": "c826c7c4241b6fc89ff55aaea3fa7491" + }, + "vroom": { + "Package": "vroom", + "Version": "1.6.5", + "Source": "Repository", + "Repository": "RSPM", + "Requirements": [ + "R", + "bit64", + "cli", + "cpp11", + "crayon", + "glue", + "hms", + "lifecycle", + "methods", + "progress", + "rlang", + "stats", + "tibble", + "tidyselect", + "tzdb", + "vctrs", + "withr" + ], + "Hash": "390f9315bc0025be03012054103d227c" + }, + "withr": { + "Package": "withr", + "Version": "3.0.0", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "R", + "grDevices", + "graphics" + ], + "Hash": "d31b6c62c10dcf11ec530ca6b0dd5d35" + }, + "xfun": { + "Package": "xfun", + "Version": "0.42", + "Source": "Repository", + "Repository": "CRAN", + "Requirements": [ + "grDevices", + "stats", + "tools" + ], + "Hash": "fd1349170df31f7a10bd98b0189e85af" + }, + "xml2": { + "Package": "xml2", + "Version": "1.3.6", + "Source": "Repository", + "Repository": "RSPM", + "Requirements": [ + "R", + "cli", + "methods", + "rlang" + ], + "Hash": "1d0336142f4cd25d8d23cd3ba7a8fb61" + }, + "yaml": { + "Package": "yaml", + "Version": "2.3.8", + "Source": "Repository", + "Repository": "RSPM", + "Hash": "29240487a071f535f5e5d5a323b7afbd" + } + } +} diff --git a/working-on-wynton-hpc/renv/.gitignore b/working-on-wynton-hpc/renv/.gitignore new file mode 100644 index 0000000..0ec0cbb --- /dev/null +++ b/working-on-wynton-hpc/renv/.gitignore @@ -0,0 +1,7 @@ +library/ +local/ +cellar/ +lock/ +python/ +sandbox/ +staging/ diff --git a/working-on-wynton-hpc/renv/activate.R b/working-on-wynton-hpc/renv/activate.R new file mode 100644 index 0000000..9b2e7f1 --- /dev/null +++ b/working-on-wynton-hpc/renv/activate.R @@ -0,0 +1,1201 @@ + +local({ + + # the requested version of renv + version <- "1.0.5" + attr(version, "sha") <- NULL + + # the project directory + project <- getwd() + + # use start-up diagnostics if enabled + diagnostics <- Sys.getenv("RENV_STARTUP_DIAGNOSTICS", unset = "FALSE") + if (diagnostics) { + start <- Sys.time() + profile <- tempfile("renv-startup-", fileext = ".Rprof") + utils::Rprof(profile) + on.exit({ + utils::Rprof(NULL) + elapsed <- signif(difftime(Sys.time(), start, units = "auto"), digits = 2L) + writeLines(sprintf("- renv took %s to run the autoloader.", format(elapsed))) + writeLines(sprintf("- Profile: %s", profile)) + print(utils::summaryRprof(profile)) + }, add = TRUE) + } + + # figure out whether the autoloader is enabled + enabled <- local({ + + # first, check config option + override <- getOption("renv.config.autoloader.enabled") + if (!is.null(override)) + return(override) + + # if we're being run in a context where R_LIBS is already set, + # don't load -- presumably we're being run as a sub-process and + # the parent process has already set up library paths for us + rcmd <- Sys.getenv("R_CMD", unset = NA) + rlibs <- Sys.getenv("R_LIBS", unset = NA) + if (!is.na(rlibs) && !is.na(rcmd)) + return(FALSE) + + # next, check environment variables + # TODO: prefer using the configuration one in the future + envvars <- c( + "RENV_CONFIG_AUTOLOADER_ENABLED", + "RENV_AUTOLOADER_ENABLED", + "RENV_ACTIVATE_PROJECT" + ) + + for (envvar in envvars) { + envval <- Sys.getenv(envvar, unset = NA) + if (!is.na(envval)) + return(tolower(envval) %in% c("true", "t", "1")) + } + + # enable by default + TRUE + + }) + + # bail if we're not enabled + if (!enabled) { + + # if we're not enabled, we might still need to manually load + # the user profile here + profile <- Sys.getenv("R_PROFILE_USER", unset = "~/.Rprofile") + if (file.exists(profile)) { + cfg <- Sys.getenv("RENV_CONFIG_USER_PROFILE", unset = "TRUE") + if (tolower(cfg) %in% c("true", "t", "1")) + sys.source(profile, envir = globalenv()) + } + + return(FALSE) + + } + + # avoid recursion + if (identical(getOption("renv.autoloader.running"), TRUE)) { + warning("ignoring recursive attempt to run renv autoloader") + return(invisible(TRUE)) + } + + # signal that we're loading renv during R startup + options(renv.autoloader.running = TRUE) + on.exit(options(renv.autoloader.running = NULL), add = TRUE) + + # signal that we've consented to use renv + options(renv.consent = TRUE) + + # load the 'utils' package eagerly -- this ensures that renv shims, which + # mask 'utils' packages, will come first on the search path + library(utils, lib.loc = .Library) + + # unload renv if it's already been loaded + if ("renv" %in% loadedNamespaces()) + unloadNamespace("renv") + + # load bootstrap tools + `%||%` <- function(x, y) { + if (is.null(x)) y else x + } + + catf <- function(fmt, ..., appendLF = TRUE) { + + quiet <- getOption("renv.bootstrap.quiet", default = FALSE) + if (quiet) + return(invisible()) + + msg <- sprintf(fmt, ...) + cat(msg, file = stdout(), sep = if (appendLF) "\n" else "") + + invisible(msg) + + } + + header <- function(label, + ..., + prefix = "#", + suffix = "-", + n = min(getOption("width"), 78)) + { + label <- sprintf(label, ...) + n <- max(n - nchar(label) - nchar(prefix) - 2L, 8L) + if (n <= 0) + return(paste(prefix, label)) + + tail <- paste(rep.int(suffix, n), collapse = "") + paste0(prefix, " ", label, " ", tail) + + } + + startswith <- function(string, prefix) { + substring(string, 1, nchar(prefix)) == prefix + } + + bootstrap <- function(version, library) { + + friendly <- renv_bootstrap_version_friendly(version) + section <- header(sprintf("Bootstrapping renv %s", friendly)) + catf(section) + + # attempt to download renv + catf("- Downloading renv ... ", appendLF = FALSE) + withCallingHandlers( + tarball <- renv_bootstrap_download(version), + error = function(err) { + catf("FAILED") + stop("failed to download:\n", conditionMessage(err)) + } + ) + catf("OK") + on.exit(unlink(tarball), add = TRUE) + + # now attempt to install + catf("- Installing renv ... ", appendLF = FALSE) + withCallingHandlers( + status <- renv_bootstrap_install(version, tarball, library), + error = function(err) { + catf("FAILED") + stop("failed to install:\n", conditionMessage(err)) + } + ) + catf("OK") + + # add empty line to break up bootstrapping from normal output + catf("") + + return(invisible()) + } + + renv_bootstrap_tests_running <- function() { + getOption("renv.tests.running", default = FALSE) + } + + renv_bootstrap_repos <- function() { + + # get CRAN repository + cran <- getOption("renv.repos.cran", "https://cloud.r-project.org") + + # check for repos override + repos <- Sys.getenv("RENV_CONFIG_REPOS_OVERRIDE", unset = NA) + if (!is.na(repos)) { + + # check for RSPM; if set, use a fallback repository for renv + rspm <- Sys.getenv("RSPM", unset = NA) + if (identical(rspm, repos)) + repos <- c(RSPM = rspm, CRAN = cran) + + return(repos) + + } + + # check for lockfile repositories + repos <- tryCatch(renv_bootstrap_repos_lockfile(), error = identity) + if (!inherits(repos, "error") && length(repos)) + return(repos) + + # retrieve current repos + repos <- getOption("repos") + + # ensure @CRAN@ entries are resolved + repos[repos == "@CRAN@"] <- cran + + # add in renv.bootstrap.repos if set + default <- c(FALLBACK = "https://cloud.r-project.org") + extra <- getOption("renv.bootstrap.repos", default = default) + repos <- c(repos, extra) + + # remove duplicates that might've snuck in + dupes <- duplicated(repos) | duplicated(names(repos)) + repos[!dupes] + + } + + renv_bootstrap_repos_lockfile <- function() { + + lockpath <- Sys.getenv("RENV_PATHS_LOCKFILE", unset = "renv.lock") + if (!file.exists(lockpath)) + return(NULL) + + lockfile <- tryCatch(renv_json_read(lockpath), error = identity) + if (inherits(lockfile, "error")) { + warning(lockfile) + return(NULL) + } + + repos <- lockfile$R$Repositories + if (length(repos) == 0) + return(NULL) + + keys <- vapply(repos, `[[`, "Name", FUN.VALUE = character(1)) + vals <- vapply(repos, `[[`, "URL", FUN.VALUE = character(1)) + names(vals) <- keys + + return(vals) + + } + + renv_bootstrap_download <- function(version) { + + sha <- attr(version, "sha", exact = TRUE) + + methods <- if (!is.null(sha)) { + + # attempting to bootstrap a development version of renv + c( + function() renv_bootstrap_download_tarball(sha), + function() renv_bootstrap_download_github(sha) + ) + + } else { + + # attempting to bootstrap a release version of renv + c( + function() renv_bootstrap_download_tarball(version), + function() renv_bootstrap_download_cran_latest(version), + function() renv_bootstrap_download_cran_archive(version) + ) + + } + + for (method in methods) { + path <- tryCatch(method(), error = identity) + if (is.character(path) && file.exists(path)) + return(path) + } + + stop("All download methods failed") + + } + + renv_bootstrap_download_impl <- function(url, destfile) { + + mode <- "wb" + + # https://bugs.r-project.org/bugzilla/show_bug.cgi?id=17715 + fixup <- + Sys.info()[["sysname"]] == "Windows" && + substring(url, 1L, 5L) == "file:" + + if (fixup) + mode <- "w+b" + + args <- list( + url = url, + destfile = destfile, + mode = mode, + quiet = TRUE + ) + + if ("headers" %in% names(formals(utils::download.file))) + args$headers <- renv_bootstrap_download_custom_headers(url) + + do.call(utils::download.file, args) + + } + + renv_bootstrap_download_custom_headers <- function(url) { + + headers <- getOption("renv.download.headers") + if (is.null(headers)) + return(character()) + + if (!is.function(headers)) + stopf("'renv.download.headers' is not a function") + + headers <- headers(url) + if (length(headers) == 0L) + return(character()) + + if (is.list(headers)) + headers <- unlist(headers, recursive = FALSE, use.names = TRUE) + + ok <- + is.character(headers) && + is.character(names(headers)) && + all(nzchar(names(headers))) + + if (!ok) + stop("invocation of 'renv.download.headers' did not return a named character vector") + + headers + + } + + renv_bootstrap_download_cran_latest <- function(version) { + + spec <- renv_bootstrap_download_cran_latest_find(version) + type <- spec$type + repos <- spec$repos + + baseurl <- utils::contrib.url(repos = repos, type = type) + ext <- if (identical(type, "source")) + ".tar.gz" + else if (Sys.info()[["sysname"]] == "Windows") + ".zip" + else + ".tgz" + name <- sprintf("renv_%s%s", version, ext) + url <- paste(baseurl, name, sep = "/") + + destfile <- file.path(tempdir(), name) + status <- tryCatch( + renv_bootstrap_download_impl(url, destfile), + condition = identity + ) + + if (inherits(status, "condition")) + return(FALSE) + + # report success and return + destfile + + } + + renv_bootstrap_download_cran_latest_find <- function(version) { + + # check whether binaries are supported on this system + binary <- + getOption("renv.bootstrap.binary", default = TRUE) && + !identical(.Platform$pkgType, "source") && + !identical(getOption("pkgType"), "source") && + Sys.info()[["sysname"]] %in% c("Darwin", "Windows") + + types <- c(if (binary) "binary", "source") + + # iterate over types + repositories + for (type in types) { + for (repos in renv_bootstrap_repos()) { + + # retrieve package database + db <- tryCatch( + as.data.frame( + utils::available.packages(type = type, repos = repos), + stringsAsFactors = FALSE + ), + error = identity + ) + + if (inherits(db, "error")) + next + + # check for compatible entry + entry <- db[db$Package %in% "renv" & db$Version %in% version, ] + if (nrow(entry) == 0) + next + + # found it; return spec to caller + spec <- list(entry = entry, type = type, repos = repos) + return(spec) + + } + } + + # if we got here, we failed to find renv + fmt <- "renv %s is not available from your declared package repositories" + stop(sprintf(fmt, version)) + + } + + renv_bootstrap_download_cran_archive <- function(version) { + + name <- sprintf("renv_%s.tar.gz", version) + repos <- renv_bootstrap_repos() + urls <- file.path(repos, "src/contrib/Archive/renv", name) + destfile <- file.path(tempdir(), name) + + for (url in urls) { + + status <- tryCatch( + renv_bootstrap_download_impl(url, destfile), + condition = identity + ) + + if (identical(status, 0L)) + return(destfile) + + } + + return(FALSE) + + } + + renv_bootstrap_download_tarball <- function(version) { + + # if the user has provided the path to a tarball via + # an environment variable, then use it + tarball <- Sys.getenv("RENV_BOOTSTRAP_TARBALL", unset = NA) + if (is.na(tarball)) + return() + + # allow directories + if (dir.exists(tarball)) { + name <- sprintf("renv_%s.tar.gz", version) + tarball <- file.path(tarball, name) + } + + # bail if it doesn't exist + if (!file.exists(tarball)) { + + # let the user know we weren't able to honour their request + fmt <- "- RENV_BOOTSTRAP_TARBALL is set (%s) but does not exist." + msg <- sprintf(fmt, tarball) + warning(msg) + + # bail + return() + + } + + catf("- Using local tarball '%s'.", tarball) + tarball + + } + + renv_bootstrap_download_github <- function(version) { + + enabled <- Sys.getenv("RENV_BOOTSTRAP_FROM_GITHUB", unset = "TRUE") + if (!identical(enabled, "TRUE")) + return(FALSE) + + # prepare download options + pat <- Sys.getenv("GITHUB_PAT") + if (nzchar(Sys.which("curl")) && nzchar(pat)) { + fmt <- "--location --fail --header \"Authorization: token %s\"" + extra <- sprintf(fmt, pat) + saved <- options("download.file.method", "download.file.extra") + options(download.file.method = "curl", download.file.extra = extra) + on.exit(do.call(base::options, saved), add = TRUE) + } else if (nzchar(Sys.which("wget")) && nzchar(pat)) { + fmt <- "--header=\"Authorization: token %s\"" + extra <- sprintf(fmt, pat) + saved <- options("download.file.method", "download.file.extra") + options(download.file.method = "wget", download.file.extra = extra) + on.exit(do.call(base::options, saved), add = TRUE) + } + + url <- file.path("https://api.github.com/repos/rstudio/renv/tarball", version) + name <- sprintf("renv_%s.tar.gz", version) + destfile <- file.path(tempdir(), name) + + status <- tryCatch( + renv_bootstrap_download_impl(url, destfile), + condition = identity + ) + + if (!identical(status, 0L)) + return(FALSE) + + renv_bootstrap_download_augment(destfile) + + return(destfile) + + } + + # Add Sha to DESCRIPTION. This is stop gap until #890, after which we + # can use renv::install() to fully capture metadata. + renv_bootstrap_download_augment <- function(destfile) { + sha <- renv_bootstrap_git_extract_sha1_tar(destfile) + if (is.null(sha)) { + return() + } + + # Untar + tempdir <- tempfile("renv-github-") + on.exit(unlink(tempdir, recursive = TRUE), add = TRUE) + untar(destfile, exdir = tempdir) + pkgdir <- dir(tempdir, full.names = TRUE)[[1]] + + # Modify description + desc_path <- file.path(pkgdir, "DESCRIPTION") + desc_lines <- readLines(desc_path) + remotes_fields <- c( + "RemoteType: github", + "RemoteHost: api.github.com", + "RemoteRepo: renv", + "RemoteUsername: rstudio", + "RemotePkgRef: rstudio/renv", + paste("RemoteRef: ", sha), + paste("RemoteSha: ", sha) + ) + writeLines(c(desc_lines[desc_lines != ""], remotes_fields), con = desc_path) + + # Re-tar + local({ + old <- setwd(tempdir) + on.exit(setwd(old), add = TRUE) + + tar(destfile, compression = "gzip") + }) + invisible() + } + + # Extract the commit hash from a git archive. Git archives include the SHA1 + # hash as the comment field of the tarball pax extended header + # (see https://www.kernel.org/pub/software/scm/git/docs/git-archive.html) + # For GitHub archives this should be the first header after the default one + # (512 byte) header. + renv_bootstrap_git_extract_sha1_tar <- function(bundle) { + + # open the bundle for reading + # We use gzcon for everything because (from ?gzcon) + # > Reading from a connection which does not supply a 'gzip' magic + # > header is equivalent to reading from the original connection + conn <- gzcon(file(bundle, open = "rb", raw = TRUE)) + on.exit(close(conn)) + + # The default pax header is 512 bytes long and the first pax extended header + # with the comment should be 51 bytes long + # `52 comment=` (11 chars) + 40 byte SHA1 hash + len <- 0x200 + 0x33 + res <- rawToChar(readBin(conn, "raw", n = len)[0x201:len]) + + if (grepl("^52 comment=", res)) { + sub("52 comment=", "", res) + } else { + NULL + } + } + + renv_bootstrap_install <- function(version, tarball, library) { + + # attempt to install it into project library + dir.create(library, showWarnings = FALSE, recursive = TRUE) + output <- renv_bootstrap_install_impl(library, tarball) + + # check for successful install + status <- attr(output, "status") + if (is.null(status) || identical(status, 0L)) + return(status) + + # an error occurred; report it + header <- "installation of renv failed" + lines <- paste(rep.int("=", nchar(header)), collapse = "") + text <- paste(c(header, lines, output), collapse = "\n") + stop(text) + + } + + renv_bootstrap_install_impl <- function(library, tarball) { + + # invoke using system2 so we can capture and report output + bin <- R.home("bin") + exe <- if (Sys.info()[["sysname"]] == "Windows") "R.exe" else "R" + R <- file.path(bin, exe) + + args <- c( + "--vanilla", "CMD", "INSTALL", "--no-multiarch", + "-l", shQuote(path.expand(library)), + shQuote(path.expand(tarball)) + ) + + system2(R, args, stdout = TRUE, stderr = TRUE) + + } + + renv_bootstrap_platform_prefix <- function() { + + # construct version prefix + version <- paste(R.version$major, R.version$minor, sep = ".") + prefix <- paste("R", numeric_version(version)[1, 1:2], sep = "-") + + # include SVN revision for development versions of R + # (to avoid sharing platform-specific artefacts with released versions of R) + devel <- + identical(R.version[["status"]], "Under development (unstable)") || + identical(R.version[["nickname"]], "Unsuffered Consequences") + + if (devel) + prefix <- paste(prefix, R.version[["svn rev"]], sep = "-r") + + # build list of path components + components <- c(prefix, R.version$platform) + + # include prefix if provided by user + prefix <- renv_bootstrap_platform_prefix_impl() + if (!is.na(prefix) && nzchar(prefix)) + components <- c(prefix, components) + + # build prefix + paste(components, collapse = "/") + + } + + renv_bootstrap_platform_prefix_impl <- function() { + + # if an explicit prefix has been supplied, use it + prefix <- Sys.getenv("RENV_PATHS_PREFIX", unset = NA) + if (!is.na(prefix)) + return(prefix) + + # if the user has requested an automatic prefix, generate it + auto <- Sys.getenv("RENV_PATHS_PREFIX_AUTO", unset = NA) + if (auto %in% c("TRUE", "True", "true", "1")) + return(renv_bootstrap_platform_prefix_auto()) + + # empty string on failure + "" + + } + + renv_bootstrap_platform_prefix_auto <- function() { + + prefix <- tryCatch(renv_bootstrap_platform_os(), error = identity) + if (inherits(prefix, "error") || prefix %in% "unknown") { + + msg <- paste( + "failed to infer current operating system", + "please file a bug report at https://github.com/rstudio/renv/issues", + sep = "; " + ) + + warning(msg) + + } + + prefix + + } + + renv_bootstrap_platform_os <- function() { + + sysinfo <- Sys.info() + sysname <- sysinfo[["sysname"]] + + # handle Windows + macOS up front + if (sysname == "Windows") + return("windows") + else if (sysname == "Darwin") + return("macos") + + # check for os-release files + for (file in c("/etc/os-release", "/usr/lib/os-release")) + if (file.exists(file)) + return(renv_bootstrap_platform_os_via_os_release(file, sysinfo)) + + # check for redhat-release files + if (file.exists("/etc/redhat-release")) + return(renv_bootstrap_platform_os_via_redhat_release()) + + "unknown" + + } + + renv_bootstrap_platform_os_via_os_release <- function(file, sysinfo) { + + # read /etc/os-release + release <- utils::read.table( + file = file, + sep = "=", + quote = c("\"", "'"), + col.names = c("Key", "Value"), + comment.char = "#", + stringsAsFactors = FALSE + ) + + vars <- as.list(release$Value) + names(vars) <- release$Key + + # get os name + os <- tolower(sysinfo[["sysname"]]) + + # read id + id <- "unknown" + for (field in c("ID", "ID_LIKE")) { + if (field %in% names(vars) && nzchar(vars[[field]])) { + id <- vars[[field]] + break + } + } + + # read version + version <- "unknown" + for (field in c("UBUNTU_CODENAME", "VERSION_CODENAME", "VERSION_ID", "BUILD_ID")) { + if (field %in% names(vars) && nzchar(vars[[field]])) { + version <- vars[[field]] + break + } + } + + # join together + paste(c(os, id, version), collapse = "-") + + } + + renv_bootstrap_platform_os_via_redhat_release <- function() { + + # read /etc/redhat-release + contents <- readLines("/etc/redhat-release", warn = FALSE) + + # infer id + id <- if (grepl("centos", contents, ignore.case = TRUE)) + "centos" + else if (grepl("redhat", contents, ignore.case = TRUE)) + "redhat" + else + "unknown" + + # try to find a version component (very hacky) + version <- "unknown" + + parts <- strsplit(contents, "[[:space:]]")[[1L]] + for (part in parts) { + + nv <- tryCatch(numeric_version(part), error = identity) + if (inherits(nv, "error")) + next + + version <- nv[1, 1] + break + + } + + paste(c("linux", id, version), collapse = "-") + + } + + renv_bootstrap_library_root_name <- function(project) { + + # use project name as-is if requested + asis <- Sys.getenv("RENV_PATHS_LIBRARY_ROOT_ASIS", unset = "FALSE") + if (asis) + return(basename(project)) + + # otherwise, disambiguate based on project's path + id <- substring(renv_bootstrap_hash_text(project), 1L, 8L) + paste(basename(project), id, sep = "-") + + } + + renv_bootstrap_library_root <- function(project) { + + prefix <- renv_bootstrap_profile_prefix() + + path <- Sys.getenv("RENV_PATHS_LIBRARY", unset = NA) + if (!is.na(path)) + return(paste(c(path, prefix), collapse = "/")) + + path <- renv_bootstrap_library_root_impl(project) + if (!is.null(path)) { + name <- renv_bootstrap_library_root_name(project) + return(paste(c(path, prefix, name), collapse = "/")) + } + + renv_bootstrap_paths_renv("library", project = project) + + } + + renv_bootstrap_library_root_impl <- function(project) { + + root <- Sys.getenv("RENV_PATHS_LIBRARY_ROOT", unset = NA) + if (!is.na(root)) + return(root) + + type <- renv_bootstrap_project_type(project) + if (identical(type, "package")) { + userdir <- renv_bootstrap_user_dir() + return(file.path(userdir, "library")) + } + + } + + renv_bootstrap_validate_version <- function(version, description = NULL) { + + # resolve description file + # + # avoid passing lib.loc to `packageDescription()` below, since R will + # use the loaded version of the package by default anyhow. note that + # this function should only be called after 'renv' is loaded + # https://github.com/rstudio/renv/issues/1625 + description <- description %||% packageDescription("renv") + + # check whether requested version 'version' matches loaded version of renv + sha <- attr(version, "sha", exact = TRUE) + valid <- if (!is.null(sha)) + renv_bootstrap_validate_version_dev(sha, description) + else + renv_bootstrap_validate_version_release(version, description) + + if (valid) + return(TRUE) + + # the loaded version of renv doesn't match the requested version; + # give the user instructions on how to proceed + remote <- if (!is.null(description[["RemoteSha"]])) { + paste("rstudio/renv", description[["RemoteSha"]], sep = "@") + } else { + paste("renv", description[["Version"]], sep = "@") + } + + # display both loaded version + sha if available + friendly <- renv_bootstrap_version_friendly( + version = description[["Version"]], + sha = description[["RemoteSha"]] + ) + + fmt <- paste( + "renv %1$s was loaded from project library, but this project is configured to use renv %2$s.", + "- Use `renv::record(\"%3$s\")` to record renv %1$s in the lockfile.", + "- Use `renv::restore(packages = \"renv\")` to install renv %2$s into the project library.", + sep = "\n" + ) + catf(fmt, friendly, renv_bootstrap_version_friendly(version), remote) + + FALSE + + } + + renv_bootstrap_validate_version_dev <- function(version, description) { + expected <- description[["RemoteSha"]] + is.character(expected) && startswith(expected, version) + } + + renv_bootstrap_validate_version_release <- function(version, description) { + expected <- description[["Version"]] + is.character(expected) && identical(expected, version) + } + + renv_bootstrap_hash_text <- function(text) { + + hashfile <- tempfile("renv-hash-") + on.exit(unlink(hashfile), add = TRUE) + + writeLines(text, con = hashfile) + tools::md5sum(hashfile) + + } + + renv_bootstrap_load <- function(project, libpath, version) { + + # try to load renv from the project library + if (!requireNamespace("renv", lib.loc = libpath, quietly = TRUE)) + return(FALSE) + + # warn if the version of renv loaded does not match + renv_bootstrap_validate_version(version) + + # execute renv load hooks, if any + hooks <- getHook("renv::autoload") + for (hook in hooks) + if (is.function(hook)) + tryCatch(hook(), error = warnify) + + # load the project + renv::load(project) + + TRUE + + } + + renv_bootstrap_profile_load <- function(project) { + + # if RENV_PROFILE is already set, just use that + profile <- Sys.getenv("RENV_PROFILE", unset = NA) + if (!is.na(profile) && nzchar(profile)) + return(profile) + + # check for a profile file (nothing to do if it doesn't exist) + path <- renv_bootstrap_paths_renv("profile", profile = FALSE, project = project) + if (!file.exists(path)) + return(NULL) + + # read the profile, and set it if it exists + contents <- readLines(path, warn = FALSE) + if (length(contents) == 0L) + return(NULL) + + # set RENV_PROFILE + profile <- contents[[1L]] + if (!profile %in% c("", "default")) + Sys.setenv(RENV_PROFILE = profile) + + profile + + } + + renv_bootstrap_profile_prefix <- function() { + profile <- renv_bootstrap_profile_get() + if (!is.null(profile)) + return(file.path("profiles", profile, "renv")) + } + + renv_bootstrap_profile_get <- function() { + profile <- Sys.getenv("RENV_PROFILE", unset = "") + renv_bootstrap_profile_normalize(profile) + } + + renv_bootstrap_profile_set <- function(profile) { + profile <- renv_bootstrap_profile_normalize(profile) + if (is.null(profile)) + Sys.unsetenv("RENV_PROFILE") + else + Sys.setenv(RENV_PROFILE = profile) + } + + renv_bootstrap_profile_normalize <- function(profile) { + + if (is.null(profile) || profile %in% c("", "default")) + return(NULL) + + profile + + } + + renv_bootstrap_path_absolute <- function(path) { + + substr(path, 1L, 1L) %in% c("~", "/", "\\") || ( + substr(path, 1L, 1L) %in% c(letters, LETTERS) && + substr(path, 2L, 3L) %in% c(":/", ":\\") + ) + + } + + renv_bootstrap_paths_renv <- function(..., profile = TRUE, project = NULL) { + renv <- Sys.getenv("RENV_PATHS_RENV", unset = "renv") + root <- if (renv_bootstrap_path_absolute(renv)) NULL else project + prefix <- if (profile) renv_bootstrap_profile_prefix() + components <- c(root, renv, prefix, ...) + paste(components, collapse = "/") + } + + renv_bootstrap_project_type <- function(path) { + + descpath <- file.path(path, "DESCRIPTION") + if (!file.exists(descpath)) + return("unknown") + + desc <- tryCatch( + read.dcf(descpath, all = TRUE), + error = identity + ) + + if (inherits(desc, "error")) + return("unknown") + + type <- desc$Type + if (!is.null(type)) + return(tolower(type)) + + package <- desc$Package + if (!is.null(package)) + return("package") + + "unknown" + + } + + renv_bootstrap_user_dir <- function() { + dir <- renv_bootstrap_user_dir_impl() + path.expand(chartr("\\", "/", dir)) + } + + renv_bootstrap_user_dir_impl <- function() { + + # use local override if set + override <- getOption("renv.userdir.override") + if (!is.null(override)) + return(override) + + # use R_user_dir if available + tools <- asNamespace("tools") + if (is.function(tools$R_user_dir)) + return(tools$R_user_dir("renv", "cache")) + + # try using our own backfill for older versions of R + envvars <- c("R_USER_CACHE_DIR", "XDG_CACHE_HOME") + for (envvar in envvars) { + root <- Sys.getenv(envvar, unset = NA) + if (!is.na(root)) + return(file.path(root, "R/renv")) + } + + # use platform-specific default fallbacks + if (Sys.info()[["sysname"]] == "Windows") + file.path(Sys.getenv("LOCALAPPDATA"), "R/cache/R/renv") + else if (Sys.info()[["sysname"]] == "Darwin") + "~/Library/Caches/org.R-project.R/R/renv" + else + "~/.cache/R/renv" + + } + + renv_bootstrap_version_friendly <- function(version, shafmt = NULL, sha = NULL) { + sha <- sha %||% attr(version, "sha", exact = TRUE) + parts <- c(version, sprintf(shafmt %||% " [sha: %s]", substring(sha, 1L, 7L))) + paste(parts, collapse = "") + } + + renv_bootstrap_exec <- function(project, libpath, version) { + if (!renv_bootstrap_load(project, libpath, version)) + renv_bootstrap_run(version, libpath) + } + + renv_bootstrap_run <- function(version, libpath) { + + # perform bootstrap + bootstrap(version, libpath) + + # exit early if we're just testing bootstrap + if (!is.na(Sys.getenv("RENV_BOOTSTRAP_INSTALL_ONLY", unset = NA))) + return(TRUE) + + # try again to load + if (requireNamespace("renv", lib.loc = libpath, quietly = TRUE)) { + return(renv::load(project = getwd())) + } + + # failed to download or load renv; warn the user + msg <- c( + "Failed to find an renv installation: the project will not be loaded.", + "Use `renv::activate()` to re-initialize the project." + ) + + warning(paste(msg, collapse = "\n"), call. = FALSE) + + } + + renv_json_read <- function(file = NULL, text = NULL) { + + jlerr <- NULL + + # if jsonlite is loaded, use that instead + if ("jsonlite" %in% loadedNamespaces()) { + + json <- tryCatch(renv_json_read_jsonlite(file, text), error = identity) + if (!inherits(json, "error")) + return(json) + + jlerr <- json + + } + + # otherwise, fall back to the default JSON reader + json <- tryCatch(renv_json_read_default(file, text), error = identity) + if (!inherits(json, "error")) + return(json) + + # report an error + if (!is.null(jlerr)) + stop(jlerr) + else + stop(json) + + } + + renv_json_read_jsonlite <- function(file = NULL, text = NULL) { + text <- paste(text %||% readLines(file, warn = FALSE), collapse = "\n") + jsonlite::fromJSON(txt = text, simplifyVector = FALSE) + } + + renv_json_read_default <- function(file = NULL, text = NULL) { + + # find strings in the JSON + text <- paste(text %||% readLines(file, warn = FALSE), collapse = "\n") + pattern <- '["](?:(?:\\\\.)|(?:[^"\\\\]))*?["]' + locs <- gregexpr(pattern, text, perl = TRUE)[[1]] + + # if any are found, replace them with placeholders + replaced <- text + strings <- character() + replacements <- character() + + if (!identical(c(locs), -1L)) { + + # get the string values + starts <- locs + ends <- locs + attr(locs, "match.length") - 1L + strings <- substring(text, starts, ends) + + # only keep those requiring escaping + strings <- grep("[[\\]{}:]", strings, perl = TRUE, value = TRUE) + + # compute replacements + replacements <- sprintf('"\032%i\032"', seq_along(strings)) + + # replace the strings + mapply(function(string, replacement) { + replaced <<- sub(string, replacement, replaced, fixed = TRUE) + }, strings, replacements) + + } + + # transform the JSON into something the R parser understands + transformed <- replaced + transformed <- gsub("{}", "`names<-`(list(), character())", transformed, fixed = TRUE) + transformed <- gsub("[[{]", "list(", transformed, perl = TRUE) + transformed <- gsub("[]}]", ")", transformed, perl = TRUE) + transformed <- gsub(":", "=", transformed, fixed = TRUE) + text <- paste(transformed, collapse = "\n") + + # parse it + json <- parse(text = text, keep.source = FALSE, srcfile = NULL)[[1L]] + + # construct map between source strings, replaced strings + map <- as.character(parse(text = strings)) + names(map) <- as.character(parse(text = replacements)) + + # convert to list + map <- as.list(map) + + # remap strings in object + remapped <- renv_json_read_remap(json, map) + + # evaluate + eval(remapped, envir = baseenv()) + + } + + renv_json_read_remap <- function(json, map) { + + # fix names + if (!is.null(names(json))) { + lhs <- match(names(json), names(map), nomatch = 0L) + rhs <- match(names(map), names(json), nomatch = 0L) + names(json)[rhs] <- map[lhs] + } + + # fix values + if (is.character(json)) + return(map[[json]] %||% json) + + # handle true, false, null + if (is.name(json)) { + text <- as.character(json) + if (text == "true") + return(TRUE) + else if (text == "false") + return(FALSE) + else if (text == "null") + return(NULL) + } + + # recurse + if (is.recursive(json)) { + for (i in seq_along(json)) { + json[i] <- list(renv_json_read_remap(json[[i]], map)) + } + } + + json + + } + + # load the renv profile, if any + renv_bootstrap_profile_load(project) + + # construct path to library root + root <- renv_bootstrap_library_root(project) + + # construct library prefix for platform + prefix <- renv_bootstrap_platform_prefix() + + # construct full libpath + libpath <- file.path(root, prefix) + + # run bootstrap code + renv_bootstrap_exec(project, libpath, version) + + invisible() + +}) diff --git a/working-on-wynton-hpc/renv/settings.json b/working-on-wynton-hpc/renv/settings.json new file mode 100644 index 0000000..ffdbb32 --- /dev/null +++ b/working-on-wynton-hpc/renv/settings.json @@ -0,0 +1,19 @@ +{ + "bioconductor.version": null, + "external.libraries": [], + "ignored.packages": [], + "package.dependency.fields": [ + "Imports", + "Depends", + "LinkingTo" + ], + "ppm.enabled": null, + "ppm.ignored.urls": [], + "r.version": null, + "snapshot.type": "implicit", + "use.cache": true, + "vcs.ignore.cellar": true, + "vcs.ignore.library": true, + "vcs.ignore.local": true, + "vcs.manage.ignores": true +} diff --git a/working-on-wynton-hpc/Gladstone_Wynton_Docs.pdf b/working-on-wynton-hpc/slide_materials/Gladstone_Wynton_Docs.pdf similarity index 100% rename from working-on-wynton-hpc/Gladstone_Wynton_Docs.pdf rename to working-on-wynton-hpc/slide_materials/Gladstone_Wynton_Docs.pdf diff --git a/working-on-wynton-hpc/slide_materials/HPC_diagram.png b/working-on-wynton-hpc/slide_materials/HPC_diagram.png new file mode 100644 index 0000000..ad22bf6 Binary files /dev/null and b/working-on-wynton-hpc/slide_materials/HPC_diagram.png differ diff --git a/working-on-wynton-hpc/slide_materials/compute_job_workflow.svg b/working-on-wynton-hpc/slide_materials/compute_job_workflow.svg new file mode 100644 index 0000000..27614ec --- /dev/null +++ b/working-on-wynton-hpc/slide_materials/compute_job_workflow.svg @@ -0,0 +1,3 @@ + + +
User
Compute
Node
Submit
Job
Query
Job Status
User
Job Script
Container
 \ No newline at end of file diff --git a/working-on-wynton-hpc/slide_materials/nf-core-rnaseq_metro_map_grey.png b/working-on-wynton-hpc/slide_materials/nf-core-rnaseq_metro_map_grey.png new file mode 100644 index 0000000..0dbf23f Binary files /dev/null and b/working-on-wynton-hpc/slide_materials/nf-core-rnaseq_metro_map_grey.png differ diff --git a/working-on-wynton-hpc/status-chart.png b/working-on-wynton-hpc/slide_materials/status-chart.png similarity index 100% rename from working-on-wynton-hpc/status-chart.png rename to working-on-wynton-hpc/slide_materials/status-chart.png diff --git a/working-on-wynton-hpc/style.css b/working-on-wynton-hpc/style.css new file mode 100644 index 0000000..3963e02 --- /dev/null +++ b/working-on-wynton-hpc/style.css @@ -0,0 +1,175 @@ + +/* Set font size and alignment for the message */ +.bottom-message { + font-size: 0.8em !important; + font-style: italic !important; + text-align: center; + position: relative; + bottom: 0 !important; + left: 0; + right: 0; +} + +.reveal code { + background-color: #1e1e1eef; /* Dark background for code chunks */ + color: white; /* White text for code */ + font-size: 1.2em; + line-height: 1.2; +} + +.code-small code { + background-color: #1e1e1eef; /* Dark background for code chunks */ + color: white; /* White text for code */ + font-size: 1em; + line-height: 1; +} + +.reveal code::selection { + background-color: #d97306 !important; /* Dark orange background for selected text */ +} + + +/* Specific styles for code output: background */ +.reveal code.output { + background-color: black; /* Black background for code outputs */ +} + +/* Custom class for code alt display */ +.code-alt code { + background-color: #ffecd0ac; /* Dark background for code outputs */ + max-height: 400px !important; + font-family: 'Menlo', sans-serif; + font-size: 0.8em; + color: rgb(76, 76, 76) +} + + + +/* Code output text color */ +.reveal pre code.output { + color: white; +} + + +/* Left-align all code outputs */ +.reveal pre code { + text-align: left !important; +} + + +/* Add horizontal scrolling to all code outputs */ +.reveal pre code.output { + white-space: pre !important; + overflow-x: auto !important; +} +/* Change the font family used for code blocks */ +pre, code, kbd, samp { + font-family: "Courier New", Courier, monospace; +} + + +/* Add horizontal scrolling to all code chunks */ +.reveal pre code { + white-space: pre !important; + overflow-x: auto !important; +} + +/* Change the font family used for all text except code */ +.reveal p, .reveal li, .reveal h1, .reveal h2, .reveal h3, .reveal h4, .reveal h5, .reveal h6 { + font-family: "Helvetica", sans-serif; +} + +.reveal h3 { + color: black; + font-size: 0.7em; +} +/* Bold slide titles and change color */ +.reveal h2 { + font-weight: bold !important; + color: #9c0366; + font-size: 1.3em; +} +/* Bold slide titles and change color */ +.reveal h1 { + font-weight: bold !important; + color: #9c0366; + font-size: 2.0em; +} + +.reveal .slides>section:first-child h2 { + color: black; + font-weight: normal !important; +} + +.reveal .slides>section:first-child h1 { + font-weight: bold !important; + color: #9c0366; +} + + +.reveal p { + text-align: left; + margin-left: 20px !important; + } +.reveal ul { + display: block; + margin-left: 75px !important; + margin-right: 50px !important; +} + +.reveal ul ul { + font-size: 0.75em; /* Smaller font size */ + margin-bottom: 5px !important; +} +.reveal ol { + display: block; + margin-bottom: 20px; + margin-left: 75px; + margin-right: 50px +} + + +/* Decrease size of image, remove border, shadow and center align*/ +.reveal img { + max-width: 60%; + border: none !important; + box-shadow: none !important; + display: block !important; + margin: 0 auto !important; +} + +small { + font-size: 70%; +} + +/* Create a custom class for the small bullets, increase the spacing between list items */ +.small-bullets ul { + font-size: 85%; +} +.small-list ol { + font-size: 80%; +} + +.reveal li { + margin-bottom: 10px !important; +} + + +.less-small-bullets ul { + font-size: 80%; +} + +.big-picture img{ + max-width: 95%; + +} + +/* Chage link color to sky blue */ +.reveal a { + color: #0c74dc; +} + +/* Change link color to magenta on hover */ +.reveal a:hover { + color: #9c0366 !important; +} diff --git a/working-on-wynton-hpc/working-on-wynton-hpc.Rproj b/working-on-wynton-hpc/working-on-wynton-hpc.Rproj new file mode 100644 index 0000000..8e3c2eb --- /dev/null +++ b/working-on-wynton-hpc/working-on-wynton-hpc.Rproj @@ -0,0 +1,13 @@ +Version: 1.0 + +RestoreWorkspace: Default +SaveWorkspace: Default +AlwaysSaveHistory: Default + +EnableCodeIndexing: Yes +UseSpacesForTab: Yes +NumSpacesForTab: 2 +Encoding: UTF-8 + +RnwWeave: Sweave +LaTeX: pdfLaTeX