O2 is a Linux-based platform for high performance computer that we can use at Harvard. This should be used for things that require a lot of computing power and therefore are slow to run locally. An example is registration of imaging sessions. Quarterly computation and storage charges apply to non-Quad groups, but not us.

Usage#

To start using it, you need to create an account.

If you use Linux or Mac, you can use the native terminal application to interact with the O2 server. If you use Windows, you will need to install a program like MobaXterm to interact with it.

You will enter O2 by (in Terminal/MobaXterm):

ssh your_ecommons@02.hms.harvard.edu

To display graphics back to your desktop (requires XQuartz on Mac):

ssh -XY your_ecommons@02.hms.harvard.edu

Where your eCommons is in all lowercase. If you are outside of the HMS building, you can access through the HMS VPN. You may be prompted for 2FA authentication via Duo.

SLURM is the resource identifier, e.g. how many cores are available on he many computer ’nodes’, on O2. The specs for O2 are:

Cores#

  • 12000+ cores
  • 32, 28 or 20 cores per ‘compute node' 256-160 GiB RAM per node (but 9 high memory, ‘highmem’ nodes, 756 GiB and one ith 1 TiB)
  • 133 GPUs 6 ‘login’ Linux head nodes, form which to dispatch jobs / do not computational tasks

Storage#

  • Active
    • For data you are actively using in computation. Please only trasnfer the data you need as space is fairly limited. Never store data here that is not backed up elsewhere.
      • /n/data1/institution/dept/lab or /n/groups/lab
        • Quota is larger than with /home/user/
      • In /home/user/
        • 100 GiB
        • In side the hidden .snapshot:
          • 14 day, daily backup
          • 60 day, weekly backup
      • Scratch
        • /n/scratch3/users/m/user/
        • Purged every 30 days
        • 10 TiBs of storage pere user!
        • No backups, often purged
      • Wilson lab group storage
        • /n/groups/wilson is currently full (1TB). Please don’t add stuff.
        • /n/data1/hms/neurobio/wilson is available for lab members (20TB).
  • Standby
    • /n/standby/institution/dept/lab on the transfer cluster
    • Infrequently accessed data.
  • Cold
    • Rarely accessed archival data, for regulatroy/historical purposes

Data Tansfers#

  • Research.files active filesystem can transfer data to O2 using, e.g. rsync. Data must be transfered to be computed against with O2. An example workflow of transferring the data from O2 to the lab server is
    • (1) #log into @transfer.rc.hms.harvard.edu (e.g. using MobaXterm if you’re in Windows)
    • (2) rsync -avz /path/to/source /files/Neurobio/Wilson\ Lab/user_name
  • Use these specs with a FTP client, e.g. Filezilla:
    • host: transfer.rc.hms.harvard.edu
    • username: your_ecommons
    • password: your_password
    • port: 22
  • Two-factor: will use default option setup in ~/.bashrc as export DUO_PASSCODE=push/phone

Common commands#

See possible arguments for any of the below using: command —help

  • quota - check your storage quota
  • /n/cluster/bin/scratch3_quota.sh - check scrath storage
  • mkdir - make directory (avoid spaces and caps where possible)
  • pwd - resent working directoy
  • ls - list directory contents
  • nano - enter text editor
  • cp - copy files
  • rm - remove files, with -r also remove directories recursively
  • * - for pattern matching
  • which - where is your executable, e.g. which gcc
  • wget - download from the web, e.g. a .tar.gz to install some software, e.g. R
  • less - check file contents, e.g. a.txtfile
  • sshare -Uu $User - see your fair share usage
  • sprio -u $User - see your job priorities
  • 02sacct - see details on past jon
  • sacct - past job usage
  • scancel - cancel job, folowed by jobID
  • scontrol hold - pasue job
  • scontrol release - release held job
  • chmod - change permissions
    • user/group/others/all can have read/write/execute
    • So chmod u+rwx my file - give your users complete access to file

LMOD and public tools#

Adds directory paths of software into $PATH. Resolves conflicts. E.g.:

  • module load gcc/6.2.0 - load the compiler gcc, version. Some modules cannot be loaded, without others loaded first.
  • module avail - available software through LMOD
  • module spider gcc - see all available versions of gcc
  • module help - see everything module can do
  • module purge - unload all modules
  • Languages available through LMOD:
    • R
      • Use O2-specific personal R library
    • Python 2 and 3
      • Use virtualenv to maintain packages, install with pip
    • Perl
    • MATLAB
      • Set up cluster profile specific to O2
  • You can also compile software in your home or group directories. LMOD will not find this (?)

Public databases#

  • /n/shared_db/

MPI#

  • Message Passing Interface
  • Distribute work over multiple nodes, allowing for the utilization of more cores
  • openMPI-3.1.0 compiled against GCC 6.2.0
  • MATLAB, Python, R, Perl, Java, C++, Fortran implementations
  • Needs wrapper function mpirun to dispatch to compute nodes with SLURM
  • Run in “mpi” partition -p mpi after obtaining access to partition
  • Core cap: 640 processors, 5-day runtime

Jobs#

  • Serial
    • priority
      • Small numberof jobs you want done fast
    • transfer
      • File transfers
    • Example:
      • srun - interactive job, go to compute node, example: run —pty -p interactive -t 0-2:00 —mem 2G bash
        • -p interactive - interactive partition
        • -t 0-2.00 - time, asking for 2 hours, your best guess
        • —mem 2G - Asking for 2 GiB of memory
        • bash - shell that we are running
        • —x11 - send graphics to your computer. Not used in batch.
  • Batch, sbatch -p partition -t 0-1:00 --wrap=“command_here”
    • sbatch yourSLurmJob.sh
  • No job can run for more than 30 days, but see limits below.
  • You cannot use CPUs you do not request, request with -c, followed by number of cores
  • One GiB by default, ask for more with —men XG where X is the number of GiB you want
  • Core usage does not necessarily scal linearly with compute time.
  • Possible .sh script to run:
#!/bin/bash 
#SBATCH -p short #partition 
#SBATCH -t 0-01:00 #time days-hr:min 
#SBATCH -c X #number of cores 
#SBATCH --mem=XG #memory per job (all cores), GiB
#SBATCH -o %j.out #out file 
#SBATCH -e %j.err #error file
#SBATCH --mail-type=BEGIN,END,FAIL,ALL 
#SBATCH --mail-user=mfk8@med.harvard.edu 
# put any module load commands here 
# put any analysis commands you want to run here
# When you run sbatch yourSLurmJob.sh any following variables can be assed as arugments in order, e.g. $1

Further learning#

Research Computing @ HMS regularly holds a training session, and you can also see the slides here.

You could also run Jupyter notebooks and RStudio on O2.

HMS IT has a GitHub page hosting O2 learning marerials, including their presentation. There is also a wiki for the O2.

Research Computing Core: ccore@hms.harvard.edu. Office hours: Wednesdays 1-3p for pressing needs, but appointments encouraged.

Look up Omero and the HMS image management core for help with microscopy data on the server, imc-support@hms.harvrad.edu.

Matlab#

If you’re using MEX files in MATLAB (e.g. ScanImageTiffReader), make sure to use the Unix version (.mexa64) and not the Windows version (.mexw64). It might be best to not have the Windows version to avoid potential path issues.

Setting files in unix mode#

If you have generated your scripts in windows, you might come across an error of the form

‘/bin/sh^M: bad interpreter’

To fix this, you need to transform your file to unix. You can do this by:

  1. Accessing your file through mobaXterm (or whatever platform you’re using) typing ‘vi filename’. This will let you see and edit your file’s contents.
  2. Enter your file in vi command mode by pressing ‘Esc’
  3. Type ‘:set fileformat=unix’.
  4. Save with ‘:x!’
Backward Image analysis resources Learning coding, data science, & stats Forward