Computational Science Community Wiki

WRFOUT writing by patch

Writing times for wrfout* files can vary a lot, depending both on the complexity of the chemical scheme you're using, and on the number of MPI processes that you're using (because communication between MPI processes, required for regrouping the data from each of the patches which make up the model domain, takes a, very real, finite amount of time). In order to avoid the MPI communication costs we can write out the model data for each patch independently, by setting the namelist data formatting option to be 102 instead of 2, i.e.: io_form_history = 102

This creates output files of the form wrfout_d01_2012-09-11_00:00:00_XXXX, where XXXX is the MPI process ID number. When running with the CBMZ and MOSAIC/VBS configuration over South America, using 192 MPI processes on ARCHER, this output method reduces write times from 110-120 seconds down to 3-10 seconds. When outputting data every hour this equates roughly to a saving of 15-20% of compute time (as each 2 minute model step takes 10-13 seconds to process, giving a compute time, without writing output data, of 300-390 seconds per model hour).

To combine these outputs into a single wrfout* file again we can use software from the Advanced Regional Prediction (ARPS) system. Their joinwrfh program will perform this task for us. To compile this program use the command: makearps -io net joinwrfh (within the ARPS code directory).

Their code isn't optimised though - in order to make it run faster you should replace their fjoinwrf.h file with the file available here: jharlass website. Or here: joinwrh github repo.

To run joinwrfh you need a namelist file - an example of which is available here: namelist file on github. I have got this to run successfully, and can confirm that this process perfectly combines the patches to give a data file containing the exact same data arrays for a single variable as the standard method for writing wrfout* files.

However, current limitations are:

  1. Setting nvarout = 0 does not seem to copy the list of variables out from the wrfout file and write them all out. Currently it seems we have to list all variables by hand in the varlist array.

  2. There's a hard limit of 300 for the number of variables that can be processed (this will require simple modification of the code to get round, though).
  3. I don't know how well this handles stepping through multiple times - my quick check of the code for a single variable seemed to suggest it will slow down after each wrfout file has been processed. It might be better to run a shell script which replaces the times from a separate list, and runs single instances of this program in serial (or limited parallel) to avoid memory issues.