wiki:FpCtbtoOverview

Version 24 (modified by dearn, 8 years ago) (diff)

--

CTBTO Project Wiki Page

Update: 2015-03-18

This project is carried out in collaboration by the following partners:

The aim of the project is to enhance the FLEXPART modeling system for CTBTO operations, and to contribute the enhancements into the mainstream FLEXPART distribution for the benefit of others. This should enable CTBTO to benefit from new science and techniques used in future FLEXPART versions, while insuring that their own enhancements remain part of the distribution.

The scope of the project includes the following functionality areas:

  1. Allow GRIB input data to be pre-processed before FLEXPART runs
  2. Allow reading of both NCEP and ECMWF GRIB files using the same executable
  3. Allow dynamic allocation of simulation grid and particles
  4. Add configuration option for outputting same units in backward runs as in forward runs
  5. Support reading ECMWF GRIB files in various sub-formats
  6. Add Makefiles for ifort

Work is currently ongoing on the second functionality-area above. More specifically, this entails:

  • Unifying the GRIB reading routines in FLEXPART so that one executable may use either NCEP or ECMWF met files.
  • Building a simple, initial regression testing system by which modifications to FLEXPART can be validated for consistency in output and performance
  • Setting up a private developmental repository and ticketing system for the project in a way that developmental releases (a CTBTO-developmental branch) can be made accessible to the mainstream flexpart.eu community for evaluation and integration into the main branch
  • Planning and providing a presentation on the project at the FLEXPART Developers meeting to be held in Vienna during EGU week.

Updates on the project will be described in this public wikipage and will also be introduced in the ticketing system for easy follow up from FLEXPART developers and the community.

Update: 2015-06-30

To perform the unification in the GRIB reading routines in FLEXPART to achieve a single executable, the following steps have taken place:

  1. A testing environment has been developed. FLEXtest, is constituted of a set of simple modules that can be independently called but that are meant to be in a high level structure. http://flexpart.eu:8088/svn/CTBTOdevel/trunk/flexpart-testing/doc/ for readily available mark down documentation. An example of the functioning and outcome of the testing is added below. http://flexpart.eu:8088/svn/CTBTOdevel/trunk/flexpart-testing/doc/Overview.md provides information and structure of the testing environment and information on its usage.
  1. The FLEXPART version in the trunk (as of date 16 February 2015) has been used to program the unification of routines. The code has been modified as follows:
  • renaming of duplicated routines so that both versions could be used in a single build
  • adding a routine to detect the format to be used (detectformat.f90)
  1. Documentation, test environment and test cases have been added into the distribution within their respective directories doc, flextest, tests. In his way, the user may test compilation, running and execution right after download.

The code, documentation and test environment may be obtained in http://flexpart.eu:8088/svn/CTBTOdevel/trunk/ -- 2015.11.02 OBSOLETE --- see below for updates and code into the git system

Update: 2015-07-13

Work is currently ongoing on the first functionality-area above: "Allow GRIB input data to be pre-processed before FLEXPART runs". The upcoming task to be addressed is to allow GRIB input data to be pre-processed before FLEXPART runs. This task shall tackle the following aspects:

  1. A command line utility (grib2flexpart) shall be developed that, given two arguments, namely the GRIB input filename and the output filename in FLEXPART native arrays, produces the files in the native FLEXPART format so that the on-the-fly processing is avoided.
  1. The utility shall reuse and share existing code in FLEXPART for reading GRIB format files and processing them into native arrays.
  2. A switch shall be added to the COMMAND configuration file to control whether data files specified in the AVAILABLE configuration file is in original GRIB format or pre-processed native arrays.

The aforementioned tasks will require the generation of appropriate test cases for verifications. Documentation shall be produced at all steps.

Update: 2015-11-05

Within work package 4, the following developments have been performed:

  • CONCEPT OF INTERMEDIATE FORMAT AND IMPLEMENTATION Concept of implementation?, Documentation on .fp format?
    • Documentation on the existing and new workflows was produced
    • The new .fp format was described in detail
    • A number of surprises were encountered, including the discovery that met data in a forward run is not exactly the same data that will be generated in a backward run with the same GRIB files.
  • UPDATES FLEXtest ENVIRONMENT: Update of the testing environment?
    • For development we need to attack the problem through a Test Driven Development (TDD) paradigm, and chose to invest heavily in the expansion of our earlier work, developing a semi-automated approach to rapidly testing new software modifications for the introduction of bugs
    • The environment consists of small tests embedded in the FLEXPART distribution that allows developers to immediately determine if their changes have had detrimental effects. It is also suitable for "outside the distribution" environments to perform large-scale regression tests. It will be a necessary component in future FLEXPART enhancements.
  • GRIB2FLEXPART UTILITY: Pre-processing command-line utility?
    • A flexible command-line utility, grib2flexpart, was developed to pre-process met files outside of a FLEXPART simulation.
    • Simultaneously, FLEXPART was modified to optionally use these .fp files as input, providing one to two orders of magnitude greater efficiency in input ingest.
    • An exciting potential of this work is that this approach may be used to define an intermediate .fp met format, to which any met data may be converted, and then used to drive FLEXPART simulations.
  • VERIFICATION TESTS: Verification tests?
    • The testing environment was used in various ways to
      • Verify that the results of using grib2flexpart to generate .fp files, and then use the .fp files to drive a FLEXPART run, were identical to those obtained by using FLEXPART with traditional GRIB met input
      • Test the performance (with the addition of conditionally-compiled timing routines) of the pre-processing paradigm relative to the traditional paradigm of GRIB met file ingest. Results suggest that a huge amount of time is spent processing the GRIB met files, and that once this is done, the dump to .fp format, and loading from .fp format gives an estimated order of magnitude gain in performance.

The code can be downloaded from http://borealscicomp.com/CTBTO_FLEXPART/snapshot_flexpart_distros/stable-wo4.tgz

Update: 2015-11-06

The upcoming work order requires the improvement of the treatment of the GRIB 1 and GRIB 2 data. A neew approach is suggested and described here?

hosted by ZAMG