Wynton

• A HPC Linux environment available to all UCSF researchers for free
  
• Uses the Rocky 8 linux OS
• Includes several hundred compute nodes and a large shared storage system (Cluster specifications)
  
• Funded and administered cooperatively by UCSF campus IT and key research groups

https://wynton.ucsf.edu

Node Types

• Login: Submit and query jobs. SSH to development nodes. File management.
• Development: Compile and install software. Test job scripts. Submit and query jobs. Version control. File management.
• Compute: Running job scripts.
• Transfer: Fast in- & outbound file transfers. File management.

The Login Nodes

• Only capable of basic tasks (file management, submitting and checking on jobs)
• Lacks access to pre-installed software tools that the development nodes have
• The primary method to log in is to use an SSH client application
• The Wynton HPC is up to date with information on logging in: Access Cluster

Names:

log1, log2 and plog1 (for PHI users)

Login

• Connect to the UCSF or Gladstone WiFi networks (or the respective VPN) or using 2FA
  
• ssh [your-username]@[node].wynton.ucsf.edu

{local}$ ssh alice@log1.wynton.ucsf.edu
alice@log1.wynton.ucsf.edu's password: 
[alice@log1 ~]$

• There will not be any visual feedback when typing your password

The Development Nodes

• Has a set of core software installed
  • e.g. git, vim, nano, make and python
• Also has access to software repositories some which are maintained by other users or research groups
  • e.g. matlab, R and openjdk
• Cannot SSH in to directly, only from a login node

ssh dev1

Names:

dev[1-3], gpudev1, pdev1 (PHI) and pgpudev1 (PHI)

Data Transfer Nodes

• Can SSH in to directly
• Fast network speed
• Limited software
• Use for transferring files to and from Wynton

Example:

{local}$ scp local_file.tsv alice@dt1.wynton.ucsf.edu:~/

Names:

dt1 and dt2

Compute Nodes

• Can not SSH in to directly
• No internet or UCSF network access
  
• Used to run non-interactive compute job scripts
• The software to run the job script is provided using a container

The File System

• A file system how information is stored and retrieved on a computer
  • Consists of files and directories
• A local file system is function of the operating system and only accessible from a single computer
• A shared file system is accessible from multiple computers

BeeGFS

• Wynton uses a parallel shared file system called BeeGFS
  • The files are stored as “chunks” spread across many different servers
• BeeGFS has multiple services that work together to manage the file system
  • Storage (stores the chunks)
  • Metadata (tracks the chunks and information about their file)
  • Management (tracks all of the services)
  • Client (provides linux access to the file system)

BeeGFS - Takehome Message

• Prefer fewer, large files over many small ones
• Distribute reading and writing over several directories
• Use local scratch (/scratch) when possible
• Don’t include anything in /wynton in your default LD_LIBRARY_PATH
• If using conda, putting the conda application inside a Apptainer (formerly singularity) container will result in better performance

Storage

• Wynton storage is not backed up
• /wynton/home/[group_name]/[user]
  • PHI users : /wynton/protected/home/[group_name]/[user]
  • User home directory - limited to 500 GiB
• /wynton/group/[group_name]
  • PHI users : /wynton/protected/group/[group_name]
  • User group directory - disk quota varies by group
  • Use this directory for any analysis you want to share with your lab
• More information on disk quotas

To check your group disk quota run:

beegfs-ctl --getquota --storagepoolid=12 --gid "$(id --group)"

Scratch - Temporary Storage

• Local /scratch - 0.1-1.8 TiB/node storage unique to each compute node
  • Can only be accessed from the specific compute node
  • Use this to store intermediate files only needed for a job
    
• /wynton/scratch and /wynton/protected/scratch (for PHI users)
  • 703 TiB storage accessible from everywhere
• No quotas

Files not used for 2 weeks are automatically deleted

Storage Advice

• Always back up anything you store under /wynton
• If you have access to it keep all of your data on /gladstone
  • A large number of jobs reading and writing to these directories may be slower since it is NFS mounted not BeeGFS
• Use the scratch directories to store temporary files
  • e.g. A large amount of .fastq that you do not need after the alignment step

GUI SFTP Clients

• These let you transfer files to and from Wynton using a GUI
• 2 factor authentication may be required
• Cyberduck
  • Navigate to Preferences -> Transfers -> General
  • change the Transfer Files setting “Use browser connection” instead of “Open Multiple connections”
• FileZilla
  • In the General tab, select ‘SFTP’ as the Protocol instead of ‘FTP’
  • For Logon Type, select ‘Interactive’ instead of ‘Ask for Password’
  • Under the Transfer Settings tab, you might need to click the ‘Limit number of simultaneous connections’ and make sure the ‘Maximum number of connections’ is set to 1

Definitions

• Containers: An isolated environment for running software that is created from an image file, preventing conflicts with the host system.
• Images: An ordered collection of root filesystem changes that contain all necessary dependencies, ensuring software run identically across various computing platforms.

Apptainer

• Wynton supports Apptainer (formerly singularity) containers

• Docker is a commonly used image creation software, these can be turned into apptainer image files (.sif) easily

• apptainer run

  • Run predefined script within container

• apptainer exec

  • Execute any command within container

• apptainer shell

  • Run bash shell within container

Docker

• Docker uses Dockerfiles to specify image creation
• Preferred by the Gladstone Bioinformatics Core to create new images
• In part 2, we will go over how to build custom container images from DockerFiles
  • If you want to follow along, install the docker engine following the instructions for your OS
  • Set up a free DockerHub account to store your images
• To see the Dockerfile used to create the hello-world image, run:

[alice@dev1 ~]$ apptainer exec hello-world_1.0.sif cat /Dockerfile