Computational Science Community Wiki

Installing WRF on ARCHER

These instructions describe how to compile and run WRF on the new national supercomputing facility, ARCHER. ARCHER officially replaced the HECToR supercomputer on 21st March 2014. The following instructions have been tested against WRF versions 3.3.1 and 3.4.1.

If you had an account on HECToR, then you can request an account on ARCHER by following the instructions at http://www.archer.ac.uk/support/getting-started/

Requirements

* WRF model source code (including WPS); see http://www.mmm.ucar.edu/wrf/users/download/get_source.html.

* netcdf v4.3.0 (pre-installed on ARCHER and available at /opt/cray/netcdf/4.3.0/cray/81);

* Cray compiler suite (NB - this is currently the default compiler suite on ARCHER, and so you don't need to do anything here)

Compilation Instructions

Copy the WRF tar file over to your work directory on ARCHER, and extract using the command:

Within the newly extracted WRF directory, create a bash script called configure_wrf.sh, containing the following:

Give your file executable permissions, i.e.

Then run the script by typing

When prompted, select the option:

This will select the cray fortran compiler, with 'dmpar' option (distributed memory) which uses MPI, thus allowing you to run WRF as a parallel job across multiple cores.

Upon completion, a file called configure.wrf will be generated, containing the settings needed for the compilation to proceed.

It is preferable to submit the compilation via a serial job submission script. It is possible to run the compile command interactively direct from the command line, although the /tmp directories on the login nodes are periodically emptied so with a long compile process such as WRF, temporary files required by the compiler sometimes go missing sporadically, leading to apparent random failures. To get around this, you can set the $TMPDIR environment variable to point to a location in your own workspace, e.g. $HOME/tmp.

An example serial job submission script to compile WRF is given below:

In the above script, be sure to replace [budget_code] with the string specifying your budget. For most CAS people, this will be n02-weat. Save your script with the name compile_wrf.sh, and then submit it using qsub:

Output from the compilation will be saved in a text file called compile_log_out.txt. To check on the status of your serial job, use qstat:

To check for any errors in the compile log file, use grep:

Upon successful completion, you should see the wrf.exe and real.exe executables in the 'main' directory.

Running WRF

Below is an example of a parallel job submission script called wrf_exe.sh for running WRF on ARCHER. Note that pbs scripts from HECToR will not work on ARCHER. Also ARCHER has 24 cores per compute node, whereas HECToR phase 3 had 32 cores per node. So running a job across 12 nodes on ARCHER equates to 288 cores in total.

Submit using qsub:

As before, you can check on the status of your job using the 'qstat' command. When the job starts to run, you can check on progress using the command:

Scroll to the end of this file to see how many timesteps have been completed.