Computational Science Community Wiki

Building WRF on the CSF


WRF v3.6 - PGI 13.6-acml-fma4 - AMD Bulldozer


GWL - June 2014

WRF Packages (from http://www2.mmm.ucar.edu/wrf/users/download/get_sources.html)

WPS Static Data: Need the Complete dataset and then the 3.6 update:

Target config

Libraries that will be built with same PGI compiler as WRF by us

Existing libraries on CSF not compiled by us (e.g., in /usr/lib64)

Dependency / Library Compilation

Setup Build Environment

All compilation below assumes you have done the following first to get on to a backend AMD Bulldozer node and to set some environment vars for convenience (to make the instructions generic). If you're doing a local build simply change the APPS and LDIR directories, which can be the same dir if you prefer (e.g., to ~/local)

On the CSF zlib, hdf5 and netcdf are installed in the libs area. We create an installation directory, an archive directory where we store the .tgz file for reference and a build directory (where we compile). You can arrange directories as you prefer. So we'll have something like:

Now build the dependencies (stick to the order given here - NetCDF must be last)

ZLIB

See http://prdownloads.sourceforge.net/libpng/zlib-1.2.8.tar.gz?download

HDF5

See http://www.hdfgroup.org/ftp/HDF5/current/src/hdf5-1.8.13.tar.gz

NetCDF (C libraries)

See ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-4.3.2.tar.gz

NetCDF (FORTRAN libraries)

See ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-fortran-4.2.tar.gz

Note: We build and install NetCDF fortran in the NetCDF (C) area. They have different version numbers (FORTRAN=v4.2, C=v4.3.2) but it makes building WRF easier if everything is in the same NetCDF installation and the FORTRAN libraries rely on the C libraries so they can be considered all part of the same package. You must do FORTRAN after C.

OpenMPI 1.6

Note, this is very CSF-specific. If you're doing a local build on the CSF you can simply load the modulefile (module load mpi/pgi-13.6-acml-fma4/openmpi/1.6-ib-amd-bd) or if you're on a different system you can probably just install MPICH2 or some other MPI implementation from a .rpm or .deb package. You don't necessarily need OpenMPI and you may not have InfiniBand networking.

WRF Compilation

Now the good stuff - WRF and WPS. On CSF we use modulefiles to set up the environment (e.g., add the previous dependencies' bin and lib directories to PATH and LD_LIBRARY_PATH.

You must now redo the steps to Setup Your Environment to get on a backend node and set the ZLIB, ZROOT etc variables. DO THIS NOW.

WRF

See top of page for download links

WPS

See top of page for download links

Now install the WPS static geography data

ncl

See top of page for download link

We are going to install a binary (pre-compiled) version because of the large number of dependencies. This is a gcc version so can be run on any compute node on the CSF. If it isn't fast enough we'll look at doing a Bulldozer build from source but that may take a while.

Running an Example

See the CSF WRF Example for step-by-step instruction on running a test case (thanks to Jonathan Fairman.)


WRF v3.4.1 - PGI 12.6 - AMD Bulldozer


MKB - work in progress (May 2013)

Target config

Required libraries

hdf (serial)

  1. download http://www.hdfgroup.org/ftp/HDF5/current/src/hdf5-1.8.11.tar.bz2

build ON Bulldozer

  1. qrsh -l bulldozer -l short

  2. module load mpi/pgi-12.10/openmpi/1.6-ib-amd-bd which also loads PGI v12 compiler

  3. cd /software/libs/pgi-12.10/hdf/5/1.8.11/build/hdf5-1.8.11

  4. ./configure --prefix=/software/libs/pgi-12.10/hdf/5/1.8.11/dist --enable-fortran --enable-cxx CC=pgcc CXX=pgCC FC=pgf90 CFLAGS="-tp=bulldozer-64" CXXFLAGS="-tp=bulldozer-64" FCFLAGS="-tp=bulldozer-64"  2>&1 | tee mkb_configure.txt (note the -tp=bulldozer-64, also needed to explicitly state compilers)

  5. make -j 64 2>&1 | tee mkb_make-j64.txt

  6. make check - passed (seems to be serial only?)

  7. make install

  8. make check-install

create modulefile libs/pgi-12.10/hdf/5/1.8.11-ib-amd-bd (ie include IB/AMD/Bulldozer in file name)

  1. ensure HDF5HOME is set

Notes

  1. serial build (by default)
  2. not sure if zlib picked up

netcdf

  1. download http://www.unidata.ucar.edu/downloads/netcdf/ftp/netcdf-4.3.0.tar.gz

This is the main (C) lib. Build this (and then see if the FORTRAN lib also needed) ***TO COMPLETE***

build ON Bulldozer

  1. qrsh -l bulldozer -l short

  2. cd /software/libs/pgi-12.10/netcdf/4.3/build/netcdf-4.3.0

  3. module load compilers/PGI/12.10  libs/pgi-12.10/hdf/5/1.8.11-ib-amd-bd

  4. CC=pgcc CPPFLAGS="-tp=bulldozer-64 -I${HDF5HOME}/include -L${HDF5HOME}/lib " ./configure --prefix=/software/libs/pgi-12.10/netcdf/4.3/dist --disable-dap 2>&1 | tee mkb_configure.txt (not sure if picked curl up; failed tests if didn't disable DAP; explicitly need to state PGI compiler/s)

  5. make check install

netcdf-fortran

See

build ON Bulldozer

  1. qrsh -l bulldozer -l short

  2. cd /software/apps/pgi-12.10/WRF/3.4.1/build/WRFV3

  3. load modules: compilers/PGI/12.10 libs/pgi-12.10/hdf/5/1.8.11-ib-amd-bd libs/pgi-12.10/netcdf/4.3-ib-amd-bd
  4. `CC=mpicc CPPFLAGS="-tp=bulldozer-64 -I${NETCDFHOME}/include" FC=pgf90 FCFLAGS="-tp=bulldozer-64" LDFLAGS=-L${NETCDFHOME}/lib ./configure --prefix=${NETCDFHOME}

WRF

build ON Bulldozer

  1. qrsh -l bulldozer -l short

  2. cd /software/libs/pgi-12.10/netcdf/fortran-4.2/build/netcdf-fortran-4.2

  3. load modules: compilers/PGI/12.10 libs/pgi-12.10/hdf/5/1.8.11-ib-amd-bd libs/pgi-12.10/netcdf/4.3-ib-amd-bd mpi/pgi-12.10/openmpi/1.6-ib-amd-bd
  4. edit arch/Config_new.pl (set $I_really_want_to_output_grib2_from_WRF = "TRUE" ;

as per configure comments)

  1. set env

    export JASPERLIB=/usr/lib64
    export JASPERINC=/usr/include/jasper
    export WRFIO_NCD_LARGE_FILE_SUPPORT=1
    export NETCDF=$NETCDFHOME
  2. .configure (options 7, 1)

  3. edit configure.wrf to use mpicc/mpif90 for DM_CC/FC (otherwise $MPI_ROOT unset so cannot find .h files). Also DM_CC needs -DMPI2_SUPPORT. NB this edit needs doing after each ./configure (or add new arch/configure_new_defaults)

WPS

build ON Bulldozer

  1. qrsh -l bulldozer -l short

  2. cd /software/apps/pgi-12.10/WPS/3.4.1/build/WPS

  3. load modules: compilers/PGI/12.10 libs/pgi-12.10/hdf/5/1.8.11-ib-amd-bd libs/pgi-12.10/netcdf/4.3-ib-amd-bd mpi/pgi-12.10/openmpi/1.6-ib-amd-bd
  4. set env vars

    export NETCDF=$NETCDFHOME
    export JASPERLIB=/usr/lib64
    export JASPERINC=/usr/include/jasper
  5. ./configure (option 11. Linux x86_64, PGI compiler, SGI MPT (dmpar))

  6. To pick up WRF, edit the resultant configure.wps so WRF_DIR is set to above build (longer term we would load the WRF module and use something like $WRFHOME). WRF_DIR set to /software/apps/pgi-12.10/WRF/3.4.1/build/WRFV3/

  7. also edit configure.wps so DM_FC/DM_CC set as per WRF build;

  8. for WPS, it appears to not always use DM_FC (for linking) but rather $FC $LD_FLAGS which requires MPI_ROOT. Sort by setting LD as $DM_FC rather than FC and unsetting LDFLAGS
  9. ./compile


open64 build

WRF does not support open64 but we built the hdf and netcdf libs anyhow

Details of Each Lib Build

netcdf

  1. download http://www.unidata.ucar.edu/downloads/netcdf/ftp/netcdf-4.3.0.tar.gz

This is the main (C) lib. Build this (and then see if the FORTRAN lib also needed) ***TO COMPLETE***

build ON Bulldozer

  1. qrsh -l bulldozer -l short

  2. cd /software/libs/open64-4.5.2/netcdf/4.3/build/netcdf-4.3.0

  3. module load compilers/amd/4.5.2 libs/open64-4.5.2/hdf/5/1.8.11

  4. CPPFLAGS="-march=bdver1 -I${HDF5HOME}/include -L${HDF5HOME}/lib " ./configure --prefix=/software/libs/open64-4.5.2/netcdf/4.3/dist --disable-dap 2>&1 | tee mkb_configure.txt (not sure if picked curl up; failed tests if didn't disable DAP)

netcdf for Fortran

  1. download http://www.unidata.ucar.edu/downloads/netcdf/ftp/netcdf-fortran-4.2.tar.gz

build ON Bulldozer

  1. qrsh -l bulldozer -l short

  2. cd /software/libs/open64-4.5.2/netcdf/4.2/build/netcdf-fortran-4.2

  3. modules: /compilers/amd/4.5.2 /libs/open64-4.5.2/hdf/5/1.8.11
  4. *** TO COMPLETE IFF NEEDED ***

hdf5

  1. download http://www.hdfgroup.org/ftp/HDF5/current/src/hdf5-1.8.11.tar.bz2

build ON Bulldozer

  1. qrsh -l bulldozer -l short

  2. cd /software/libs/open64-4.5.2/hdf/5/1.8.11/build/hdf5-1.8.11/

  3. module load compilers/amd/4.5.2

  4. ./configure --prefix=/software/libs/open64-4.5.2/hdf/5/1.8.11/dist --enable-fortran --enable-cxx CFLAGS="-march=bdver1" CXXFLAGS="-march=bdver1" FCFLAGS="-march=bdver1" 2>&1 | tee mkb_configure.txt (note the -march=dbver1 (might not be necessary if on Bulldozer but will be required if compiling from login node)

  5. make -j 64 2>&1 | tee mkb_make-j64.txt

  6. make check - passed (seems to be serial only?)

  7. make install

  8. make check-install

create modulefile: libs/open64-4.5.2/hdf/5/1.8.11

  1. ensure HDF5HOME is set

Notes

  1. not sure if zlib picked up