Computational Science Community Wiki

Differences between revisions 29 and 31 (spanning 2 versions)
Revision 29 as of 2017-11-02 17:50:31
Size: 6503
Editor: DouglasLowe
Comment:
Revision 31 as of 2017-11-07 15:10:10
Size: 6902
Editor: DouglasLowe
Comment:
Deletions are marked like this. Additions are marked like this.
Line 26: Line 26:
  * NOTE: this is no longer required for version 3.9.1 (and onwards?)   * NOTE: the Makefile in version 3.9.1 (and onwards?) has already had this option removed, so you don't need to do this anymore.
Line 72: Line 72:

== WRF-Chem 3.9.1 ==


 * Configuration:
  * 50: INTEL (ftn/icc): Cray XC (dmpar)
  * Compile for nesting: select 1(=basic)
 * WRF-Chem can be compiled with optimisation levels 2 & 3 (-O2 and -O3), if we ensure that floating point maths is precise (-fp-model precise) rather than fast (-fp-model fast=1 --- which is the default)

Installing WRF-Chem on ARCHER

The big change in moving from HECToR to ARCHER is that the PGI compiler is not available on the new platform. WRF can be compiled using the CRAY compiler, but this does not have any options for creating executables which run on the login (or serial) nodes, so to compile WRF-Chem we will have to use the Intel Compiler.

Currently we have to compile on the login nodes to avoid hardware incompatibilities. The serial nodes are not the same architecture as the login and compute nodes, and so some compiler options which work on these nodes will not work for executables compiled to run on the serial nodes. As the compilation process for WRF-Chem involves compiling (and using) temporary executables we cannot do this on the serial nodes.

Common Requirements

  • The common temp directories get cleared out regularly - to avoid our compilation process failing due to deleted temporary files we should make sure that we've defined our own temp directory in the .bashrc file first, e.g.:

    • export TMPDIR='/work/n02/n02/[username]/TMPDIR'
  • environmental settings to use (for the moment we use the default modules - this may change later though):
    • module swap PrgEnv-cray PrgEnv-intel

    • module load netcdf
    • export NETCDF=$NETCDF_DIR
    • export WRFIO_NCD_LARGE_FILE_SUPPORT=1
    • export WRF_CHEM=1
    • export WRF_KPP=1
    • export FLEX_LIB_DIR=/usr/lib64
  • YACC is not installed - instead the "yacc" command points to the BISON program, which does not process the KPP source files properly ("y.tab.h" is not produced from the "scan.y" file). To get around this we have to install (and point to) our own YACC installation.
    • define the path to the YACC install in "chem/KPP/configure_kpp", e.g.:
      • YACC=/nerc/n02/n02/lowe/WRF-CHEM_Compiling/archer/byacc/yacc -d
  • The compiler options for KPP are hard-coded in the KPP Makefile (not set in the configuration file as the rest of the compiler options for WRF are), so we need to edit these options by hand:
    • delete the "-ll" option from the file "chem/KPP/kpp/kpp-2.1/src/Makefile" (2 lines need modifying, as part of the make config for the "kpp" option).
    • NOTE: the Makefile in version 3.9.1 (and onwards?) has already had this option removed, so you don't need to do this anymore.
  • NOTE: the following step was needed when ARCHER was first used. It is not needed now (Jan 2017) for v3.8.1, and probably isn't needed for v3.4.1 either.
    • When compiling the write_decomp program (which is used while compiling WRF-Chem, for modifying the KPP generated code, and is recompiled many times during the process of compiling WRF-Chem) we need to specify explicitly the compiler used, rather than using the default compiler, as doing so includes extra compiler options that we don't want for the login nodes.
      • In "chem/KPP/util/write_decomp/Makefile" replace "$(SFC)" with "ifort" (on lines 36 and 54)

WRF-Chem 3.4.1 (and derivatives)

  • Configuration:
    • No option for compiling on Cray with Intel exists in the configuration file for 3.4.1 (and earlier?) - so this option needs to be copied from the configuration file for 3.5.1
    • (37): Cray XC30 CLE/Linux x86_64, Xeon ifort compiler (dmpar)
      • option number will depend on where you've pasted the settings into the "arch/configure_new.defaults" file
      • I've deleted "$(TRADFLAG)" from the "CPP" options - this might not be necessary though?
    • Compile for nesting: select 1(=basic)
  • KPP does not work properly with the latest version of FLEX - to get round this we copy a pre-generated version of the "lex.yy.c" file to "chem/KPP/kpp/kpp-2.1/src/"
  • I have retro-fitted changes made to WRF-Chem 3.6 to our copy of 3.4.1 (for the setting of SOILW and SOILM under glaciers, and for ensuring that some variables within the radiation calculation scheme are +ive and non-zero)
  • Compilation when configuring for debugging (using the ./configure -d command) takes (roughly) 14 minutes on the login nodes.
  • Compilation with standard options takes (roughly) 45 minutes on the login nodes.
    • WRF-Chem can be compiled with optimisation level 1 without changing the configuration file
    • WRF-Chem can be compiled with optimisation levels 2 & 3 (-O2 and -O3), if we ensure that floating point maths is precise (-fp-model precise) rather than fast (-fp-model fast=1 --- which is the default) - example configuration files with these changes are included in the svn code

      • NOTE: These example configuration files are for netcdf 4.3.0 --- as the ARCHER package system is upgraded with new packages you should check to see if the example configuration files are still valid!

WRF-Chem 3.5.1

  • Configuration:
    • 54: Cray XC30 CLE/Linux x86_64, Xeon ifort compiler (dmpar)
    • Compile for nesting: select 1(=basic)
  • Compilation when configuring for debugging (using the ./configure -d command) takes (roughly) 10 minutes on the login nodes. I've not yet checked if the compiled executables work correctly though (25/1/2014).
  • Compilation with standard options takes (roughly) 40 minutes on the login nodes. As for the debugging executables, I've not yet checked if the compiled executables work correctly though (25/1/2014).

WRF-Chem 3.8.1

  • Configuration:
    • 50: INTEL (ftn/icc): Cray XC (dmpar)
    • Compile for nesting: select 1(=basic)
  • Debugging:
    • Can't use "-g" option --- this causes this error for module_io_domain.f90:
      • Fatal compilation error: Out of memory asking for 2097152

    • "-traceback" option is okay though
  • Standard:
    • compiles okay, not tried running the code though (19/10/2016)
  • UoM Modified Code
    • The registry file created once all VBS variables are added is too large for compiling on the login nodes (once we hit ~3.7Gb of memory allocation, malloc refuses to allocate any more, and a segmentation fault occurs)
    • To compile successfully we need a 2 step process:
      • First generate the KPP chemistry code on the login nodes: chem/KPP/compile_wkc 2>&1 | tee z_chem_kpp_precompile_log.txt

      • Then run the standard compile command on the serial nodes, within a batch script. This will skip the KPP code generation (as it's already been done), so avoiding the issues with running the linking program on the serial nodes, and will carry on the rest of the compile, without the memory issues seen on the login nodes.

WRF-Chem 3.9.1

  • Configuration:
    • 50: INTEL (ftn/icc): Cray XC (dmpar)
    • Compile for nesting: select 1(=basic)
  • WRF-Chem can be compiled with optimisation levels 2 & 3 (-O2 and -O3), if we ensure that floating point maths is precise (-fp-model precise) rather than fast (-fp-model fast=1 --- which is the default)