Amarel

Using Amarel

Getting Started

This wiki page is a brief introduction to get started running the ROMS model, and analyze model output, on the Rutgers Office of Advanced Research Computing (OARC) cluster computer Amarel. There is a comprehensive user guide at https://sites.google.com/view/cluster-user-guide to which you can refer for more detailed information.

Rutgers VPN

If you are not on campus, you will need to connect to the Rutgers VPN. To use Rutgers VPN, you must first enroll in Duo 2 Factor Authentication (2FA) (NetID+). Most users have probably already done this. If not, navigate to https://netid.rutgers.edu/setupTwoFactorAuthentication.htm and follow the instructions.

Once you are enrolled in NetID+ you will need to activate the VPN service, if you have not already. Navigate to https://soc.rutgers.edu/vpn/ and click the gray button titled Service Activation. If you are not already logged in you may be asked to login and/or approve your login with the Duo Mobile app. To activate the VPN service, click the checkbox next to Remote Access VPN, Cisco AnyConnect Access for Rutgers and click the Activate Services button.

You are now ready to download the Cisco VPN client and connect to the Rutgers VPN. Complete instructions can be found here by clicking the red button titled General VPN Instructions to download the PDF. The instructions are geared towards Windows so if you are using a Mac you might find this page more helpful. In most cases, regardless of your operating system, pointing your browser to https://vpn.rutgers.edu/ and logging in with your NetID will lead you to downloading the correct VPN client.

Once installed, open the Cisco AnyConnect client and type vpn.rutgers.edu in the box and click Connect:

In the next window, the Username will be your NetID, the password field for either will be your NetID Password and for 2FA, you have 4 options to enter in the Second Password/Duo Action field:

• Enter a 6 digit Duo Passcode. These are generated either by a Hard Token, showing the passcode in the Duo Mobile App, or from a previous “SMS” request. Simply type in the 6 numbers and hit OK.
• Type the word “push”. This will send a push notification to the primary device you have enrolled with Duo through NetID+ with the option to Accept or Deny.
• Type the word “phone”. You will receive a phone call to the primary device you have enrolled with Duo through NetID+ with touch tone options to Accept or Deny.
• Type the word “sms”. You will receive a text message to the primary device you have enrolled with Duo through NetID+ containing passcodes you can use to logon.

Click OK and you should be connected to Rutgers VPN. You should have a small AnyConnect icon with a lock on it in the task tray (Windows ) or in the menu bar (Mac OS ).

Connecting to Amarel with SSH

In order to connect to amarel and compile and run ROMS you will need an SSH/terminal client. If you already use an SSH/terminal application you are comfortable with the stick with that and adapt the instructions below accordingly. For these instructions to work you need to either be on campus or connected to the Rutgers VPN.

Mac OS

If you are on a Mac, open Terminal (found in the Applications -> Utilities folder) or iTerm2 and type:

ssh fakeuser@amarel.rutgers.edu

replacing fakeuser with your NetID. When it asks if you want continue connecting enter yes, then enter your NetID password. Having to type the above command and password every time can be annoying and time consuming. Luckily there a couple things you can do to make things easier.

1. If you are still connected to amarel, disconnect by entering exit
2. At the prompt for your local machine, enter nano ~/.ssh/config and paste the block below at the end of the file:
   Host amarel
   Hostname amarel.rutgers.edu
   HostKeyAlias amarel
   User [NetID]
   replacing [NetID] with your own NetID. Save and exit (Ctrl+o Ctrl+x). This allows you to type ssh amarel to connect.
3. Enter ls ~/.ssh if there is a file called id_rsa.pub skip to 5
4. Enter ssh-keygen -t rsa, then hit return to accept the default location, enter a passphase twice (or leave blank for no passphrase)
5. Copy the public portion of the key to amarel by entering scp ~/.ssh/id_rsa.pub amarel:.
6. SSH to amarel (ssh amarel) and enter the following cat ~/id_rsa.pub >> ~/.ssh/authorized_keys then exit back to your local machine and execute ssh amarel again. It should no longer ask for your password.

Windows

For Windows, we recommend MobaXterm installer edition.

1. Once installed open it, choose light or dark theme and click the Session button in the upper left.
2. Choose SSH and enter amarel.rutgers.edu for Remote host, check the Specify username box and enter your NetID in the box and click OK.
3. You will be asked to type your NetID password and then asked if you want to save the password. If you choose yes you will be asked to set a master password to encrypt all your saved passwords.
4. In the future you can click Sessions (not Session) and select amarel.rutgers.edu and not have to enter your password.

Setting up your .bashrc

The main Amarel software setup by OARC does not include the software needed to compile and run ROMS. In order to use the software needed for running ROMS and other models, you will need to add a couple lines to the login script (.bashrc, .cshrc, etc.) in your home directory. Log in to amarel with Terminal/iTerm2 or MobaXterm and edit your login script as described below.

Unless otherwise requested your default shell will be bash and the following three lines (shown in red) should be added (with nano or your preferred editor) near the top of your .bashrc after the import of the global definitions as shown:

# Source global definitions

if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi

ulimit -s unlimited

export MODULEPATH=/projects/dmcs_1/sw/modulefiles/Core:${MODULEPATH}
export SQUEUE_FORMAT=“%.18i %.9P %.8j %.8u %.2t %.10M %.10l %.4C %.6D %R”

The first line allows you to take full advantage of computing resources (may be obsolete now but it won't hurt anything). The first export line will tell Lmod where to find the modules that will load the custom software. The second export line makes the squeue command (explained later) more useful. Once you have added these lines, log out and back in for them to take effect.

Checking Out the ROMS Source Code

If you are not already a ROMS user, you will need to fill out the registration form (found here) and wait for approval. Once approved, you will be able to checkout the ROMS source code. There is a GIT repository available but Subversion (SVN) is still the preferred way to obtain the ROMS source code. If your prefer git, the initial process is a little more involved but you can follow the directions found on the Git page.

There is nothing special about checking out the ROMS source code on Amarel. The same svn commands you’re used to will work on Amarel. However, the first time you check the code out you will will need to use the ‘--username’ flag unless your NetID matches you ROMS username. Navigate to the directory where you want your ROMS source code to reside and execute this svn checkout command:

svn --username <user> co https://www.myroms.org/svn/src/trunk <my_src_dir>

replacing <user> with your ROMS username and <my_src_dir> with what you want the source code directory to be called. After typing your password it will ask you if you want to store your password. I would suggest typing yes but it’s up to you. If you answer no you will have to type your password every time you do an svn checkout or svn update. After your first checkout you will no longer need the --username flag for svn operations to any of the myroms.org subversion repositories.

Loading and Unloading Modules

Like many clusters, Amarel uses environment modules to load and unload software and configure the environment. Some commands you will find useful are:

module help Display help message
module help <m1> Show help information for module <m1>
module available Show all modules currently available
module whatis <m1> Show brief information about module <m1>
module spider List all possible modules even if not currently available
module list List currently loaded modules
module load Load the specified module(s)
module unload Unload the specified module(s)
module swap <m1> <m2> Unload <m1> and then load <m2> (for switching versions of the same software)
module purge Unload all loaded modules

Loading the ROMS Module

Setting up your environment to compile and run ROMS is as simple as loading the roms module.

module load roms
module list
Currently Loaded Modules:
1) intel/17.0.4 2) mvapich2/2.2 3) mct-roms/2.6.0 4) netcdf/4.6.2
5) esmf/8.0.0_nc4 6) parpack-roms/2.1 7) hdf5/1.10.4 8) roms/intel_nc4

Notice that loading the roms module will actually load 5 other modules.

This will setup your environment to use the Intel compiler so remember to set FORT to ifort in your build script.

Configuring and Compiling ROMS

Using the build script is the recommended method for compiling ROMS on Amarel. Some of the modules that load with the roms module also set environment variables that help simplify your roms build script. Starting from the latest build_roms.sh you will likely only need to change ROMS_APPLICATION, MY_SRC_DIR, set FORT to ifort, and uncomment USE_NETCDF4.

It is important to make sure USE_MY_LIBS is set to no or your compilation will fail.

Remembering the name you gave above for <my_src_dir>, you will find latest build_roms.sh script in subdirectory ROMS/Bin. Copy that script to the directory you will work in to run ROMS.

Configure build_roms.sh by setting the line MY_SRC_DIR to point to your choice for <my_roms_src_dir>.

If this is your first time working with ROMS, a good starting place is to compile the default UPWELLING test case that is indicated by the build_roms.sh setting ROMS_APPLICATION=UPWELLING.

Copy to your working directory the file upwelling.h from subdirectory ROMS/Include of your source code.

Once your build script is configured and you have upwelling.h in your working directory you can compile ROMS by typing:

./build_roms.sh -j 4

where the number after -j indicates the number of compute cores to use in parallel to execute the compilation. The greater the number, the faster it goes.

However, the login node you will be compiling on is shared for the entire Amarel system. If you use a number larger than 4, or omit it altogether (which says use all cores on the login node) your build might be terminated by an administrator. Be a considerate user and keep the number low.