CTBTO Project Wiki Page
This project is carried out in collaboration by the following partners:
- Comprehensive Nuclear-Test-Ban Treaty Organization (CTBTO)
- Geophysical Institute at the University of Alaska Fairbanks (UAF) (http://www.gi.alaska.edu/)
- Zentralanstalt für Meteorologie und Geodynamik (ZAMG) (http://www.zamg.ac.at/), Vienna
- Angewandte Wissenschaft Software und Technologie (AWST) GmbH (http://www.awst.at/), also in Vienna
The aim of the project is to enhance the FLEXPART modeling system for CTBTO operations, and to contribute the enhancements into the mainstream FLEXPART distribution for the benefit of others. This should enable CTBTO to benefit from new science and techniques used in future FLEXPART versions, while insuring that their own enhancements remain part of the distribution.
The scope of the project includes the following functionality areas:
- Allow GRIB input data to be pre-processed before FLEXPART runs
- Allow reading of both NCEP and ECMWF GRIB files using the same executable
- Allow dynamic allocation of simulation grid and particles
- Add configuration option for outputting same units in backward runs as in forward runs
- Support reading ECMWF GRIB files in various sub-formats
- Add Makefiles for ifort
Work is currently ongoing on the second functionality-area above. More specifically, this entails:
- Unifying the GRIB reading routines in FLEXPART so that one executable may use either NCEP or ECMWF met files.
- Building a simple, initial regression testing system by which modifications to FLEXPART can be validated for consistency in output and performance
- Setting up a private developmental repository and ticketing system for the project in a way that developmental releases (a CTBTO-developmental branch) can be made accessible to the mainstream flexpart.eu community for evaluation and integration into the main branch
- Planning and providing a presentation on the project at the FLEXPART Developers meeting to be held in Vienna during EGU week.
Updates on the project will be described in this public wikipage and will also be introduced in the ticketing system for easy follow up from FLEXPART developers and the community.
To perform the unification in the GRIB reading routines in FLEXPART to achieve a single executable, the following steps have taken place:
- A testing environment has been developed. FLEXtest, is constituted of a set of simple modules that can be independently called but that are meant to be in a high level structure. http://flexpart.eu:8088/svn/CTBTOdevel/trunk/flexpart-testing/doc/ for readily available mark down documentation. An example of the functioning and outcome of the testing is added below. http://flexpart.eu:8088/svn/CTBTOdevel/trunk/flexpart-testing/doc/Overview.md provides information and structure of the testing environment and information on its usage.
- The FLEXPART version in the trunk (as of date 16 February 2015) has been used to program the unification of routines. The code has been modified as follows:
- renaming of duplicated routines so that both versions could be used in a single build
- adding a routine to detect the format to be used (detectformat.f90)
- Documentation, test environment and test cases have been added into the distribution within their respective directories doc, flextest, tests. In his way, the user may test compilation, running and execution right after download.
The code, documentation and test environment may be obtained in
http://flexpart.eu:8088/svn/CTBTOdevel/trunk/ -- 2015.11.02 OBSOLETE --- see below for updates and code into the git system
Work is currently ongoing on the first functionality-area above: "Allow GRIB input data to be pre-processed before FLEXPART runs". The upcoming task to be addressed is to allow GRIB input data to be pre-processed before FLEXPART runs. This task shall tackle the following aspects:
- A command line utility (grib2flexpart) shall be developed that, given two arguments, namely the GRIB input filename and the output filename in FLEXPART native arrays, produces the files in the native FLEXPART format so that the on-the-fly processing is avoided.
- The utility shall reuse and share existing code in FLEXPART for reading GRIB format files and processing them into native arrays.
- A switch shall be added to the COMMAND configuration file to control whether data files specified in the AVAILABLE configuration file is in original GRIB format or pre-processed native arrays.
The aforementioned tasks will require the generation of appropriate test cases for verifications. Documentation shall be produced at all steps.
Within work package 4, the following developments have been performed:
- CONCEPT OF INTERMEDIATE FORMAT AND IMPLEMENTATION Concept of implementation, Documentation on .fp format
- Documentation on the existing and new workflows was produced
- The new .fp format was described in detail
- A number of surprises were encountered, including the discovery that met data in a forward run is not exactly the same data that will be generated in a backward run with the same GRIB files.
- UPDATES FLEXtest ENVIRONMENT: Update of the testing environment
- For development we need to attack the problem through a Test Driven Development (TDD) paradigm, and chose to invest heavily in the expansion of our earlier work, developing a semi-automated approach to rapidly testing new software modifications for the introduction of bugs
- The environment consists of small tests embedded in the FLEXPART distribution that allows developers to immediately determine if their changes have had detrimental effects. It is also suitable for "outside the distribution" environments to perform large-scale regression tests. It will be a necessary component in future FLEXPART enhancements.
- GRIB2FLEXPART UTILITY: Pre-processing command-line utility
- A flexible command-line utility, grib2flexpart, was developed to pre-process met files outside of a FLEXPART simulation.
- Simultaneously, FLEXPART was modified to optionally use these .fp files as input, providing one to two orders of magnitude greater efficiency in input ingest.
- An exciting potential of this work is that this approach may be used to define an intermediate .fp met format, to which any met data may be converted, and then used to drive FLEXPART simulations.
- VERIFICATION TESTS: Verification tests
- The testing environment was used in various ways to
- Verify that the results of using grib2flexpart to generate .fp files, and then use the .fp files to drive a FLEXPART run, were identical to those obtained by using FLEXPART with traditional GRIB met input
- Test the performance (with the addition of conditionally-compiled timing routines) of the pre-processing paradigm relative to the traditional paradigm of GRIB met file ingest. Results suggest that a huge amount of time is spent processing the GRIB met files, and that once this is done, the dump to .fp format, and loading from .fp format gives an estimated order of magnitude gain in performance.
- The testing environment was used in various ways to
The code can be downloaded from http://borealscicomp.com/CTBTO_FLEXPART/snapshot_flexpart_distros/stable-wo4.tgz
The upcoming work order requires the improvement of the treatment of the GRIB 1 and GRIB 2 data. A new approach is suggested and described here?
The implementation of a VTABLE concept prototype has been programmed for ECMWF GRIB1/GRIB2 versions. Updates of the concept and its implementation can be found here?
Within Work package 8, the work stated in Work package 5 will continue. The Vtables prototype will be extended and generalised to additionally support NCEP input. As a result, a new version of the code, testing environment and documentation shall be made available.
Currently Work package 8 has successfully finished and a FLEXPART version fully working with the Vtables approach is ready. Within this WP the following tasks have been achieved:
1. Enhanced testing environment and its documentation
Increasingly, the use of a robust test environment in this project has been vital. Although there is a time element of investing in its development and maintenance, it has allowed us to develop code with much greater confidence, finding errors quickly rather than having them pop up unexpectedly months or years later. When bugs are discovered, the testing environment often helps to quickly isolate the cause and test fixes.
The general motivation behind the environment is to protect the entire FLEXPART code base from the introduction of unexpected bugs, while allowing users to determine readily whether their current implementation is working in full. An environment that can run through a comprehensive set of tests is the primary goal, coupled with a discipline of running the full set of tests every time a change is introduced into the master branch. The enhancements to the testing environment that were introduced in this work order helped us to build a code base that we have increased confidence in, as well as allowing us to discover several problems in our own code, and in the original FLEXPART v9.2 code.
- Prior to the start of this work order, a number of changes were made to the testing environment based on initial experiences in its use
- The ability to specify a makefile in the XML test specification file was removed, and the makefile must now be specified on the command line when invoking the test. This was done because makefile are very specific to the system they are running on, and we didn't want users to have to edit the XML test specification files every time they downloaded a new test environment. It made more sense for users to be able to simply specify a makefile that they already had on their system. After facilitating the command-line specification of makefile, we removed the ability to specify it in XML files because we felt that having both capabilities would get very confusing
- Added flags for specifying that temporary test directories could be saved in the event of a successful test. By default they would be removed, to save space
- Other minor changes meant to get the testing environment up to speed for beginning Work Package 8 included the addition of nested input test cases, as well as general code ehancements
- One of the major problems that was facing our testing environment was that it had a broad collection of test cases - many with poor naming conventions - and we frequently got confused over which cases should be run, or had been run, after majour project milestones. We realized that we wanted the general FLEXPART user to be able to run a single command and have a comprehensive set of tests performed, giving an immediate (within 2-15 minutes, depending on depth of tests) indication of whether the system was running correctly or not. Anything short of this would make it difficult to encourage the discipline of Test Driven Development (TDD) and frequent testing of the system after modifications A description of this enhancement is described in Consolidation of tests
- The existing test environment documentation was reviewed and modified. A number of cosmetic and code-based changes were made, but the primary changes came from the realization that the documentation was very technical, and would be difficult for new users to understand. Therefore, the primary document, Overview.md was modified with two links at the beginning which pointed to more "hands-on" documents, including a quick-start page Quick Start. Note - these two pages are best viewed in a browser with a markdown extension.
- In a credit to the value of the testing environment, while trying to add deposition test cases some problems with the Work Package 4 implementation of met input preprocessing was discovered. They are described in more detail at Discussion on Deposition aspects
2. Completed FLEXPART code with the VTABLES
- The deployment of the Vtable code for NCEP met inputs paralleled that of the ECMWF met inputs from Work Order 6, but several problems were encountered, forcing us to go back and modify some of the code we had implemented for ECMWF (this was actually the reason we chose to split the ECMWF and NCEP deployments into different work orders - we wanted to test the concept first, before putting effort into generalizing it for other types of met inputs). A detailed account of the work is available in the following document
- Deployment of Vtables for NCEP inputs
- A snapshot of the current master branch of the repository has been extracted and is available as flexpart-wo8.tar.gz (about 1 GByte in size - it includes test files), available at http://borealscicomp.com/CTBTO_FLEXPART/snapshot_flexpart_distros/flexpart-wo8.tar.gz
- A Doxygenated representation of the newest code, and test environment code, is available at the links below. It takes a little bit of exploring, but particularly in the FLEXPART code link, it's possible to view every routine, look at the call and caller trees, and even the source code.