wiki:FpCtbtoOverview

CTBTO Project Wiki Page

Update: 2015-03-18

This project is carried out in collaboration by the following partners:

The aim of the project is to enhance the FLEXPART modeling system for CTBTO operations, and to contribute the enhancements into the mainstream FLEXPART distribution for the benefit of others. This should enable CTBTO to benefit from new science and techniques used in future FLEXPART versions, while insuring that their own enhancements remain part of the distribution.

The scope of the project includes the following functionality areas:

  1. Allow GRIB input data to be pre-processed before FLEXPART runs
  2. Allow reading of both NCEP and ECMWF GRIB files using the same executable
  3. Allow dynamic allocation of simulation grid and particles
  4. Add configuration option for outputting same units in backward runs as in forward runs
  5. Support reading ECMWF GRIB files in various sub-formats
  6. Add Makefiles for ifort

Work is currently ongoing on the second functionality-area above. More specifically, this entails:

  • Unifying the GRIB reading routines in FLEXPART so that one executable may use either NCEP or ECMWF met files.
  • Building a simple, initial regression testing system by which modifications to FLEXPART can be validated for consistency in output and performance
  • Setting up a private developmental repository and ticketing system for the project in a way that developmental releases (a CTBTO-developmental branch) can be made accessible to the mainstream flexpart.eu community for evaluation and integration into the main branch
  • Planning and providing a presentation on the project at the FLEXPART Developers meeting to be held in Vienna during EGU week.

Updates on the project will be described in this public wikipage and will also be introduced in the ticketing system for easy follow up from FLEXPART developers and the community.

Update: 2015-06-30

To perform the unification in the GRIB reading routines in FLEXPART to achieve a single executable, the following steps have taken place:

  1. A testing environment has been developed. FLEXtest, is constituted of a set of simple modules that can be independently called but that are meant to be in a high level structure. http://flexpart.eu:8088/svn/CTBTOdevel/trunk/flexpart-testing/doc/ for readily available mark down documentation. An example of the functioning and outcome of the testing is added below. http://flexpart.eu:8088/svn/CTBTOdevel/trunk/flexpart-testing/doc/Overview.md provides information and structure of the testing environment and information on its usage.
  1. The FLEXPART version in the trunk (as of date 16 February 2015) has been used to program the unification of routines. The code has been modified as follows:
  • renaming of duplicated routines so that both versions could be used in a single build
  • adding a routine to detect the format to be used (detectformat.f90)
  1. Documentation, test environment and test cases have been added into the distribution within their respective directories doc, flextest, tests. In his way, the user may test compilation, running and execution right after download.

The code, documentation and test environment may be obtained in http://flexpart.eu:8088/svn/CTBTOdevel/trunk/ -- 2015.11.02 OBSOLETE --- see below for updates and code into the git system

Update: 2015-07-13

Work is currently ongoing on the first functionality-area above: "Allow GRIB input data to be pre-processed before FLEXPART runs". The upcoming task to be addressed is to allow GRIB input data to be pre-processed before FLEXPART runs. This task shall tackle the following aspects:

  1. A command line utility (grib2flexpart) shall be developed that, given two arguments, namely the GRIB input filename and the output filename in FLEXPART native arrays, produces the files in the native FLEXPART format so that the on-the-fly processing is avoided.
  1. The utility shall reuse and share existing code in FLEXPART for reading GRIB format files and processing them into native arrays.
  2. A switch shall be added to the COMMAND configuration file to control whether data files specified in the AVAILABLE configuration file is in original GRIB format or pre-processed native arrays.

The aforementioned tasks will require the generation of appropriate test cases for verifications. Documentation shall be produced at all steps.

Update: 2015-11-05

Within work package 4, the following developments have been performed:

  • CONCEPT OF INTERMEDIATE FORMAT AND IMPLEMENTATION Concept of implementation, Documentation on .fp format
    • Documentation on the existing and new workflows was produced
    • The new .fp format was described in detail
    • A number of surprises were encountered, including the discovery that met data in a forward run is not exactly the same data that will be generated in a backward run with the same GRIB files.
  • UPDATES FLEXtest ENVIRONMENT: Update of the testing environment
    • For development we need to attack the problem through a Test Driven Development (TDD) paradigm, and chose to invest heavily in the expansion of our earlier work, developing a semi-automated approach to rapidly testing new software modifications for the introduction of bugs
    • The environment consists of small tests embedded in the FLEXPART distribution that allows developers to immediately determine if their changes have had detrimental effects. It is also suitable for "outside the distribution" environments to perform large-scale regression tests. It will be a necessary component in future FLEXPART enhancements.
  • GRIB2FLEXPART UTILITY: Pre-processing command-line utility
    • A flexible command-line utility, grib2flexpart, was developed to pre-process met files outside of a FLEXPART simulation.
    • Simultaneously, FLEXPART was modified to optionally use these .fp files as input, providing one to two orders of magnitude greater efficiency in input ingest.
    • An exciting potential of this work is that this approach may be used to define an intermediate .fp met format, to which any met data may be converted, and then used to drive FLEXPART simulations.
  • VERIFICATION TESTS: Verification tests
    • The testing environment was used in various ways to
      • Verify that the results of using grib2flexpart to generate .fp files, and then use the .fp files to drive a FLEXPART run, were identical to those obtained by using FLEXPART with traditional GRIB met input
      • Test the performance (with the addition of conditionally-compiled timing routines) of the pre-processing paradigm relative to the traditional paradigm of GRIB met file ingest. Results suggest that a huge amount of time is spent processing the GRIB met files, and that once this is done, the dump to .fp format, and loading from .fp format gives an estimated order of magnitude gain in performance.

The code can be downloaded from http://borealscicomp.com/CTBTO_FLEXPART/snapshot_flexpart_distros/stable-wo4.tgz

Update: 2015-11-06

The upcoming work order requires the improvement of the treatment of the GRIB 1 and GRIB 2 data. A new approach is suggested and described here?

Update: 2015-12-11

The implementation of a VTABLE concept prototype has been programmed for ECMWF GRIB1/GRIB2 versions. Updates of the concept and its implementation can be found here?

Update: 2016-01-10

Within Work package 8, the work stated in Work package 5 will continue. The Vtables prototype will be extended and generalised to additionally support NCEP input. As a result, a new version of the code, testing environment and documentation shall be made available.

Update: 2016-04-05

Currently Work package 8 has successfully finished and a FLEXPART version fully working with the Vtables approach is ready. Within this WP the following tasks have been achieved:

1. Enhanced testing environment and its documentation

Increasingly, the use of a robust test environment in this project has been vital. Although there is a time element of investing in its development and maintenance, it has allowed us to develop code with much greater confidence, finding errors quickly rather than having them pop up unexpectedly months or years later. When bugs are discovered, the testing environment often helps to quickly isolate the cause and test fixes.

The general motivation behind the environment is to protect the entire FLEXPART code base from the introduction of unexpected bugs, while allowing users to determine readily whether their current implementation is working in full. An environment that can run through a comprehensive set of tests is the primary goal, coupled with a discipline of running the full set of tests every time a change is introduced into the master branch. The enhancements to the testing environment that were introduced in this work order helped us to build a code base that we have increased confidence in, as well as allowing us to discover several problems in our own code, and in the original FLEXPART v9.2 code.

  • Prior to the start of this work order, a number of changes were made to the testing environment based on initial experiences in its use
    • The ability to specify a makefile in the XML test specification file was removed, and the makefile must now be specified on the command line when invoking the test. This was done because makefile are very specific to the system they are running on, and we didn't want users to have to edit the XML test specification files every time they downloaded a new test environment. It made more sense for users to be able to simply specify a makefile that they already had on their system. After facilitating the command-line specification of makefile, we removed the ability to specify it in XML files because we felt that having both capabilities would get very confusing
    • Added flags for specifying that temporary test directories could be saved in the event of a successful test. By default they would be removed, to save space
    • Other minor changes meant to get the testing environment up to speed for beginning Work Package 8 included the addition of nested input test cases, as well as general code ehancements
  • One of the major problems that was facing our testing environment was that it had a broad collection of test cases - many with poor naming conventions - and we frequently got confused over which cases should be run, or had been run, after majour project milestones. We realized that we wanted the general FLEXPART user to be able to run a single command and have a comprehensive set of tests performed, giving an immediate (within 2-15 minutes, depending on depth of tests) indication of whether the system was running correctly or not. Anything short of this would make it difficult to encourage the discipline of Test Driven Development (TDD) and frequent testing of the system after modifications A description of this enhancement is described in Consolidation of tests
  • The existing test environment documentation was reviewed and modified. A number of cosmetic and code-based changes were made, but the primary changes came from the realization that the documentation was very technical, and would be difficult for new users to understand. Therefore, the primary document, Overview.md was modified with two links at the beginning which pointed to more "hands-on" documents, including a quick-start page Quick Start. Note - these two pages are best viewed in a browser with a markdown extension.
  • In a credit to the value of the testing environment, while trying to add deposition test cases some problems with the Work Package 4 implementation of met input preprocessing was discovered. They are described in more detail at Discussion on Deposition aspects

2. Completed FLEXPART code with the VTABLES

Update: 2017-12-01: LAST ACTIONS OF THE PROJECT- PHASE III

Executive summary

This final phase has focused on FLEXPART 10, which includes serial and parallel versions, to include the modifications and new features required by CTBTO and produced in former phases for FLEXPART version 9. The five basic modifications/features are (OPTIONAL or NON-OPTIONAL depending on whether the user can select to use it or not):

  1. Unification of the GFS and ECMWF executables by adding automatic detection of the meteorological data. A single executable (one for MPI and one for serial) are produced after compilation and detectformat.f90 routine handles the usage of routines according to the type of driving data - [NON-OPTIONAL]
  1. Vtables approach. This is a paradigm change on how internally FLEXPART handles different incoming meteorological data, facilitating the implementation of new data and variations in already used data. This approach is based on the usage of Variable tables (Vtables) with the specifications of different parameters inside the GRIB files that then are handled by the corresponding library (class_vtable_mod.f90). Vtables are already provided with the distribution for ECMWF data (GRIB2, GRIB1, GRIB1-2) and GFS/NCEP data (GRIB1, GRIB2) - [NON-OPTIONAL]
  1. Implementation of a testing environment that includes control data for a variety of cases including ECMWF and GFS data, serial and parallel versions, GRIB2FLEXPART intermediate (NetCDF4) formats and the CTBTO-specific modifications. All the control cases operate under the basis of the Vtables approach and unified executable. - [OPTIONAL]
  1. Implementation of the add-on tool GRIB2FLEXPART. This tool, currently only valid for non-depositing species, allows the user to pre-process the meteorological data (in essence, performing all the calculations inside the subroutines called by getfields.f90) and have it available for multiple runs. The pre-processed data are stored in NetCDF4 format and read in by a modified getfields and the corresponding libraries. It is an option and does not affect the normal usage of FLEXPART other than adding a new field in the COMMAND file. If namelists format is used, the default option is not to use the intermediate files and therefore should only be added explicitly in the COMMAND files when the intermediate format is required. It is implemented for both serial and parallel, GFS and ECMWF data. - [OPTIONAL]
  1. Implementation of receptor concentration possibility and concentration gridded output in backward runs. These are specific CTBTO requirements that are activated through a parameter (lctbto) in par_mod.f90 file. - [OPTIONAL]

In the process some bugs were encountered and (many) fixed. A list is provided below and tickets for some of them have been created:

  1. verttransform.f90 routine had a problem in calling “ew” function that was written as a multiplication instead [FIXED]
  1. MPI_IN_PLACE is used instead of MPI_REDUCE if directed during compilation time. This required the modification of mpi_mod.f90, unc_mod.f90, outgrid_init.f90 and the makefiles. This was needed because some issues appeared due to the original MPI_Reduce calls. Now MPI_IN_PLACE is used instead but only if directed during compilation. Otherwise, a safer option is used. [FIXED]
  1. Flexpart has been modified so that the version string in the header is 256-character static [FIXED]
  1. Modification of the getting of the size and data of the values array after checking of the dimensions. Section:

!  !get the size and data of the values array
  if (isec1(6).ne.-1) then
    call grib_get_real4_array(igrib,'values',zsec4,iret)
    call grib_check(iret,gribFunction,gribErrorMsg)
  endif

Should be moved downwards in the code [NOT YET FIXED]

  1. MPI error message stating MPI runs were not possible in backward runs has been removed. [FIXED]
  1. MPI runs generate duplicated timestamps in the dates file [NOT FIXED]

Unification of executables

FLEXPART v10 has been modified in order to have automatic detection of the driving meteorological data. Therefore, the current version will only require two executables: FLEXPART and FLEXPART_MPI and they will both run with NCEP and ECMWF data. In order to achieve this, the modifications described below have been done.

  • gributils directory: this directory, provided inside src, includes the class_gribfile_mod.f90 module. This routine is used by FLEXPART to identify the type of incoming driving data.
  • detectformat.f90: this new routine, added in src, is called by FLEXPART.f90 and FLEXPART_MPI.f90 and it is used to detect the format of the meteorological driving data by setting the variable metdata_format. This variable is used in timemanager.f90 and timemanager_mpi.f90 (and subsequently by get_fields.f90 and get_fiels_mpi.f90) routines to call the appropriate *gfs or *ecmwf routines.
  • each of the *gfs and *ecmwf routines now have the internal subroutine name changed adding _ecmwf and _gfs. For example: gridcheck_ecmwf.f90 routine had "subroutine gridcheck" in the code. Now it reads "subroutine gridcheck_ecmwf". This is needed to ensure we can have a single executable that calls the appropriate subroutines.
  • the makefile has been adapted accordingly: only one executable name for serial and mpi (this modifies the compilation command line, as described in the description. Namely, one does not need to write make gfs serial but only make serial. This will generate an executable that will work for both ECMWF and NCEP data. The same for the mpi executable. In addition, the makefile has been adapted to use the gributils class_gribfile_mod.f90 module.

Notes:

  • The code assumes (which is a reasonable assumption) that the type of meteorological data does not change during the run (which would not be possible anyway in previous FLEXPARTs). The first GRIB file is used to identify the center and it assumes all the GRIB files used will be of the same type. If neither ECMWF nor NCEP are not found as center type, then the code exists with a No wind file available message.
  • The original code included two modules, ecmwf_mod.f90 and gfs_mod.f90. Those have now been erased and the variables brought back to the par_mod.f90 file. This has been done with NILU's agreement.

Vtables

One of the main structural modifications is to modify the routines that read the meteorological input data from the GRIB files so that all the variable references are made to names of the corresponding meteorological fields rather than a combination of GRIB parameter codes. In order to do this, we have borrowed the approach from the NWP model WRF (Weather Research and Forecasting model) whereby a Variable Table (Vtable) provides the mapping between the FLEXPART meteorological field names (e.g.. TT, U, U10, …) and the respective GRIB codes. Vtables for ECMWF and GFS data are provided inside the Vtables directory in the usual options directory. The layout of the Vtables is such as:

GRIB1 | Level| flexpart | flexpart | flexpart               |GRIB2|GRIB2|GRIB2|GRIB2|
Param | Type | Name     | Units    | Description            |Discp|Catgy|Param|Level|
------+------+----------+----------+------------------------+-----------------------+
 130  | 109  | TT       | K        | Temperature            |  0  |  0  |  0  | 105 |
 131  | 109  | UU       | m s-1    | U                      |  0  |  2  |  2  | 105 |
 132  | 109  | VV       | m s-1    | V                      |  0  |  2  |  3  | 105 |
 133  | 109  | QV       | kg kg-1  | Specific humidity      |  0  |  1  |  0  | 105 |
 134  | 1    | PS       | Pa       | Surface pressure       |  0  |  3  |  0  |  1  |
 135  | 109  | ETADOT   | Pa s-1   | Vertical velocity      |  0  |  2  |  32 | 105 |
 141  | 1    | SD       | m        | Snow depth             |  0  |  1  |  11 |  1  |
 151  | 1    | MSL      | Pa       | Sea-level Pressure     |  0  |  3  |  0  | 101 |
 164  | 1    | TCC      | (0-1)    | Total cloud cover      | 192 | 128 | 164 |  1  |
 165  | 1    | U10      | m s-1    | 10m U                  |  0  |  2  |  2  | 103 |
 166  | 1    | V10      | m s-1    | 10m V                  |  0  |  2  |  3  | 103 |
 167  | 1    | T2       | K        | 2m Temperature         |  0  |  0  |  0  | 103 |
 168  | 1    | TD2      | K        | 2m Dewpoint            |  0  |  0  |  6  | 103 |

…

 172  | 1    | LSM      | 0/1 Flag | Land/Sea flag          |  2  |  0  |  0  |  1  |
 246  | 1    | CLWC     | kg kg-1  | Spec. cld liq water    |  0  |  1  |  83 | 103 |
 247  | 1    | CIWC     | kg kg-1  | Spec. cld ice water    |  0  |  1  |  84 | 103 |
201031| 1    | QC       | kg kg-1  | Spec. cld water cont.  |  0  | 201 |  31 | 105 |
------+------+----------+----------+------------------------+-----------------------+

In order to use the Vtables, the routines that read the meteorological data (gridcheck and readwind) are modified and, the structures like:

if(isec1(6).eq.131) uuh(i,j,nlev_ec-k+2)= &!! U VELOCITY
zsec4(nxfield*(ny-j-1)+i+1)

Are now modified to simply call the actual variable name:

.
ELSE IF(TRIM(fpname) .EQ. 'UU') THEN
    iumax=max(iumax,nlev_ec-k+1)
        DO j=0,nymin1
            DO i=0,nxfield-1
                uuh(i,j,nlev_ec-k+2) = zsec4(nxfield*(ny-j-1)+i+1)
            END DO
        END DO

This has required the generation of an additional library, provided with the code, that handles the conversion from variable names to corresponding GRIB parameters. The library is provided in src/gributils/class_vtable_mod.f90 . This library handles as well the GRIB2 to GRIB1 conversions of parameters.

The benefits of this approach are:

  • The code itself is more understandable and not dependent on specific GRIB parameter

values

  • Users can experiment with new sources of met fields by creating a new Vtable, mapping

to the new GRIB parameters, without the need to modify code

This approach, however, adds a level of indirection to the code (a table lookup) that has been compensated by restructuring the code and rearranging the loops in readwind, leading to even better performance in comparison with the original code.

From the user perspective, the only difference is that a link to the correct Vtable has to be done in the run directory (where pathnames sits)

The code has been tested against our testing environment showing zero differences for the test cases.

In order to implement and deploy the Vtable approach, the following main modifications/additions have been carried out:

  • Creation of an independent gributils/class_Vtable_mod.f90 to abstract all the operations of using the Vtables and matching GRIB file parameter codes to the appropriate Vtable entry. Gributils is provided inside src/
  • Modification of the code in the following source files that deal directly with GRIB parameter codes - gridcheck_ecmwf.f90, readwind_ecmwf.f90, gridcheck_nests.f90, and readwind_nests.f90 . These modifications include:
    • Added code that initializes a Vtable object
    • Added code that used Vtable object routines to determine the fpname (e.g. UU,T2, CONVPREC , etc.) of a GRIB message
    • Modified code to look for a specific fpname rather than specific GRIB parameters

Testing environment

A new and independent top-level directory, ctbto-testing-env was added to the directory tree. This directory contains the following:

  • case_data: this directory includes the different test cases used in the testing environment, with the corresponding tree structure and control data. The cases are organised first according to the meteorological type of data (e.g. ecmwf_02hr_1p0deg) and then, several cases underneath depending on the type of FLEXPART run with this data (eg. case_ecmwf_02hr_1p0deg_1species_1emission_bwd, case_ecmwf_02hr_1p0deg_1species_1emission_fwd). In each of the case directories, the user finds the control output (the output the testing environment will test against), the rundir_template directory (with the right options, pathnames and Vtable) and the par_mod file.
  • check: this directory contains all the high level routines to control the tests and includes the files the user should invoke to use the testing environment. The test driver, *check.py* is a Python program that expects an XML file as an argument, then creates a *TestSuite?* consisting of all the components needed and performs all the compilations, simulations and testing. XML files for the case data are provided. In addition, run_selected_cases.py adds the possibility to run multiple cases defined in a text file (-lst) and multi_configtest.py calls in the run_selected_cases.py but using different makefiles provided as well by the user. How the code is executed is provided through the help function.
  • distrotest: includes all the routines that prepares all the information and populate the directories to perform the testing
  • doc: documentation for this environment is in the ctbto-testing-env/doc/ directory, in Markdown format. With appropriate Markdown Extension, pointing the browser to ctbto-testing-env/doc/Overview.md should provide a usable browsing of the documentation.
  • flextest: lower level components for compiling, running, and evaluating FLEXPART output. These provide low level access routines for running FLEXPART programs, as well as numerous tools for accessing, evaluating and comparing FLEXPART outputs.
  • metfetch: the testing environment requires to have meteorological data to drive the test cases. metfetch automatically gets the NCEP data and prompts information on restricted ECMWF data.
  • test: set of unit tests for the testing environment development
  • utilities: includes examples of additional output one could obtain from the testing environment.
  • distropath: link to where the source code to run the testing environment is located.

Apart from the documentation inside the testing environment itself, an updated version is kept in: ​http://borealscicomp.com/CTBTO_FLEXPART/WO17/ctbto_testenv_docs/Overview.md

The testing environment includes control case data using the original NILU code for serial tests but with the modifications in the header . For the MPI control data, the new version with the modifications detailed below, has been used to create it

GRIB2FLEXPART

GRIB2FLEXPART is an add-on utility that, currently only for non-depositing species, allows the user to pre-process each of the meteorological files and have it available for multiple runs. The utility makes use of FLEXPART source code (getfields.f90 and related subroutines) and generates a set of intermediate files in netCDF4 that can be optionally used to run FLEXPART. The utility consists of the following:

  • grib2flexpart directory: includes the main routines that a) will handle the calling of the FLEXPART routines to convert calculate and interpolate the meteorological data needed to drive FLEXPART, b) the routines to produce the intermediate output in netCDF4 format and c) a test_preprocmet_mod routine that includes testing routines.
  • preproc_gridcheck.f90 and preproc_getfields.f90 (in the usual src directory of the FLEXPART distribution): these two routines substitute the usual getfields.f90 and readwind.f90 ones and directly read in the intermediate files generated by the user in a former step
  • An adapted makefile that includes the dependencies for the compilation of the preprocessing tool and those to use the pre-processed fields.

The utility is thought to process one grib file at a time. The user should generate a batch script for generating multiple files or a sequence of files. The tool is called by:

 grib2flexpart <path to input gribfile> <path to output NetCDF4 file>

Once the intermediate files are generated, the user will need to prepare the usual AVAILABLE in the usual format and simply adapt the pathnames file to point to where the netCDF4 files are stored and include in COMMAND the option “1” for the preproc_met field. In the namelist format, preprocmet is set to 0 by default.

Limitations:

  • In FLEXPART, the internal vertical coordinate system is initialised from the first meteorological file read in (that is, from its surface pressure and the orography field at the lower left corner of the met domain). If this is not treated in a consistent way, and the first file read in is different every time, then the vertical coordinate system is defined differently with every read in and, when passed through the testing environment, differences will appear. We have opted for looking for a trade-off solution and assume the potential l inconsistency in the treatment of the vertical coordinates. In this option we guarantee that only one single day of data needs to be processed (for backward and forward) nevertheless, although the number of internal vertical levels is kept the same, each time a file is generated a new system is initialised. For this case, each meteorological file (or in case of nests, mother and nest at the same time) may be processed via command line.
  • Depositing species cannot be handled in this approach mainly because of the deposition parameters calculated in calcpar without actually reading information about the species to be simulated.

CTBTO specific software

As explained in Ticket on CTBTO specific code in Fp10 . For Version 9.3 the implementation of such code was made using some compile IFDEF directives. However, for version 10 and, after consultation with the Flexpart developers, a better solution has been found: a new logical variable “lctbto” has been added in par_mod.f90 file. The variable is by default set to False and only those who want to use the CTBTO specific code will have to set it to True and recompile. The specific code includes the following:

  • Add the possibility to generate receptors output in backward runs
  • Add the possibility to obtain concentration as backwards unit (when using IND_SOURCE and IND_RECEPTOR set to one) - this includes the 10E+12 factor as well.
  • Allow for aggregated output in backward mode (allowing to have IFOR = 0 for bwd runs)
  • Small modification in the welcome string indicating you are using the CTBTO specific version

TODOs

Dynamical allocation of many of the meteorological variables has been tested and prototyped. This has not yet been fully implemented and remains as a TODO.

Last modified 6 days ago Last modified on Dec 5, 2017, 9:27:59 AM
hosted by ZAMG