Computational Science Community Wiki

Redqueen

Compiling v3.2 Using Modules

MKB, Winter & Summer 2011

  1. extract files
    1. make a clean new directory and cd in to it
    2. untar WRFV3.2.TAR (55 MB), which creates and populates directory WRFV3

    3. cd in to WRFV3 and then untar WRFV3-Chem-3.2.TAR (11 MB) which creates a new subdir chem

  2. build WRF-Chem (v3.2, default options) from the WRFV3 directory (NB as of Spring 2012 there is now a newer openmpi build) module load use.own compilers/intel/11.1.075 tau/2.21.1/openmpi-1.5.4/intel/11.1.075/iowrapper openmpi/1.5.4/intel/11.1.075 libs/intel-11.1/netcdf/4.1.3

    module load compilers/intel/fortran compilers/intel/c  libs/intel-11.1/netcdf  mpi/intel-11/openmpi/1.4.1
    export NETCDF=/software/netcdf/4.1.3/dist
    export WRFIO_NCD_LARGE_FILE_SUPPORT=1 
    export WRF_CHEM=1
    ./configure (options 11,1)
    ./compile em_real 

where option 11 is "Linux x86_64 i486 i586 i686, ifort compiler with icc (dmpar)" and for nesting option 1 (basic); for some builds it is safer to make serially: ./compile -j 1 em_real

Compiling V3.4.1 on Redqueen

SAN - 07/01/12

i. extract files

  1. make a clean new directory and cd in to it
  2. untar WRFV3.4.1.TAR, which creates and populates directory WRFV3

  3. cd in to WRFV3 and then untar WRFV3-Chem-3.4.1.TAR, which creates a new subdir chem

ii. Make changes to chem/KPP/configure_kpp:

-- # YACC=yacc -d
++ YACC=yacc -d

iii. copy lexx file to the KPP directory: cp /atmos-scratch-1/mbessdl2/WRFCHEM_src/CRIMECH-N2O5-v3.4.1-ATMOS_IB/lexx.yy.c v3.4.1_scottedits/chem/KPP/kpp/kpp-2.1/src/

iv. use configue and compile scripts for compiling WRF-Chem on Redqueen:

  1. ./config_wrfchem.sh (options: 15,1). Option 15 = Linux x86_64 i486 i586 i686, ifort compiler with icc (dmpar). Nesting option = 1 (basic)
  2. ./compile_wrfchem.sh

config_wrfchem.sh:

 #!/bin/bash

#load module commands
. /usr/share/Modules/init/bash

 # load required modules
module load compilers/intel/12.0.5/
module load libs/intel-11.1/netcdf
module load mpi/intel-12/openmpi/1.6-qlc

 # export system variables
export NETCDF=/software/netcdf/4.1.3/dist
export WRFIO_NCD_LARGE_FILE_SUPPORT=1
export WRF_CHEM=1
export WRF_KPP=1
export CHEM_DBG_I=20
export CHEM_DBG_J=20
export CHEM_DBG_K=1
export FLEX_LIB_DIR=/usr/lib64

./configure

FCOPTIM         =       -O1

 #!/bin/bash

 # Adjust the PATH environment to include the present directory.
 # This is required by the KPP chemistry script
export PATH=$PATH:.

#load module commands
. /usr/share/Modules/init/bash

 # load required modules
module load compilers/intel/12.0.5
module load libs/intel-11.1/netcdf
module load mpi/intel-12/openmpi/1.6-qlc

 # export system variables
export NETCDF=/software/netcdf/4.1.3/dist
export WRFIO_NCD_LARGE_FILE_SUPPORT=1
export WRF_CHEM=1
export WRF_KPP=1
export CHEM_DBG_I=20
export CHEM_DBG_J=20
export CHEM_DBG_K=1
export FLEX_LIB_DIR=/usr/lib64

./compile em_real &> wrf_compile_log.txt

Running on Redqueen

module load compilers/intel/c compilers/intel/fortran libs/intel-11.1/netcdf libs/intel-11.1/hdf mpi/intel-11/openmpi/1.4.1 
qsub -q r3-atmos-parallel-16.q -cwd -V -pe orte.pe 100 -l h_rt=04:00:00
 ./runWrfChemScript.sh

will run on 100 cores with max wall clock of 4 hrs

Doug - 25 Aug 2011 - running wrfchem

These are the commands I used for running wrf.exe on the 25th Aug 2011 (our new version of WRF/CHEM):

module load compilers/intel/c compilers/intel/fortran libs/intel-11.1/netcdf libs/intel-11.1/hdf mpi/intel-11/openmpi/1.4.1 
qsub -q r3-atmos-parallel-16.q -cwd -V -pe orte.pe 100 -l h_rt=04:00:00 <<EOF
set -x
ulimit -s unlimited
time mpirun -np 100 ./wrf.exe
EOF

The executable file should be in the current working directory.

Doug - 10 Sept 2011 - ulimit -s unlimited is not required - the error was caused by array overflow from bad coding...


Setting up Prequisite Libs (ITSfR/MKB)


Previously...

  1. configure (option 11 (dmpar) and basic nesting)

Jonathan Fairman- Running WRF 3.4.1 Fresh Compile

There was a slight issue with a fresh compile of WRF 3.4.1 involving this error -

'undefined reference to intel_sse2_strcpy' _

This was with Intel 11.1.075, MPI 1.5.4, netcdf, curl, HDF5 all loaded in. The error was in read_grib1.c. The solution was to add this line to configure.wrf, changing LD_FLAGS_LOCAL

'LDFLAGS_LOCAL = -ip -L/opt/intel/Compiler/11.1/075/lib/intel64/-lifcoremt -Bstatic -lirc -Bdynamic'

Then cleaning out the GRIB external libraries and recompiling.