wiki:FpCtbtoWo8TestEnvConsolidation

Description of the consolidation activities of the testing environment Flex Test

There have been two tasks to make the test more user-friendly:

  1. Use a consistent naming convention for the test cases and xml files to achieve names that are as much self-explanatory as possible
  2. Generation of a flexible script whereby the user can select what cases to run and have a summary of the failing cases, if any.

Naming convention

Case data

The case data follows the structure:

meteorological case
    flexpart cases
    meteorological data
    fp data (whenever applies)

The meteorological case is named as [type of met data]_[length]_[resolution]_[specification1]_[secification2]_[...]. For example:

  • ecmwf_02hr_1p0deg_grib1-2 -->ECMWF data, 2 hours of meteorological data at 1 degree, global (because the contrary is not stated) and in combination GRIB1 and GRIB2 data
  • ecmwf_02hr_1p0deg_nestedinput --> ECMWF data, 2 hours of meteorological data at GRIB2

Inside the general meteorological case, the user can find the meteorological data and the flexpart case data with the set-up and control files. The meteorological data specifies period and resolution t[start hour]_[end hour - length]h_[met data type abbreviation (ec or nc)][resolution]. For the fp data, there is in addition the [fp] label at the front and whether it is [fwd/bwd]. For example:

  • fp_bwd_12h_ec1p0deg --> fp formatted files, prepared to be run backward, 12 hours of data, based on ECMWF at 1 degree resolution.
  • t1_33h_ec1p0d --> 33 hours of ECMW data at 1 degree horizontal resolution.

Te flexpart cases are named as case_[type of met data]_[simulation lenght in hours]hr_[resolution]_[specification1]_[specification2]_[number of species]species_[number of emissions]emission_[specification3]_[fwd or backward]. Specifications can be GRIB format or whether there is nested input or output. For example:

  • case_ecmwf_02hr_1p0deg_nestedinput_1species_1emission_nestedoutgrid_fwd --> FLEXPART run driven by ECMWF data, with a simulation length of 2 hours and nested input data, 1 species and one emission and nested outgrid in forward.
  • case_ecmwf_02hr_1p0deg_grib1-2_1species_1emission_bwd --> FLEXPART run driven by ECMWF data, with a simulation length of 2 hours, with input data GRIB1 and 2, one species one emission and backward.

XML test files

These files are located in the check directory of the testing environment and define what evaluations the user want to do with and from the aforementioned case data. For example, a user may want to use one forward run with ECMWF data and evaluate the RMSE and ABSOLUTE BIAS. The user may combine case data. Each file includes the description of the test cases to run and the statistical parameters that will be calculated.

Examples:

  • test_ecmwf_02hr_1p0deg_1species_1emission_fwd_bwd.xml --> Basic test with a FLEXPART run using ECMWF data at 1 degree, 2 hours simulation both forward and backward, calculating maximum error and RMSE for the entire domain.
  • test_ecmwf_02hr_1p0deg_nestedinput_1species_1emission_nestedoutgrid_fwd_bwd.xml --> Basic test with a FLEXPART run using ECMWF data at 1 degree, with nested meteorological data, 2 hours simulation both forward and backward, with a nested outrid and calculating maximum error and RMSE for the entire domain.
  • test_ecmwf_06hr_1p0deg_tinymetgrid_fpformat_1species_1emission_fwd.xml--> Basic test with a FLEXPART run using fp formatted data from ECMWF data at 1 degree but using a small meteorological domain, 6 hours simulation forward, calculating maximum error and RMSE for the entire domain.

Inventory of tests

A table with the available case data and xml files is provided here.

Case data

ecmwf_1p0deg_tinymet_multipletests case_ecmwf_03hr_1p0deg_tinymetgrid_2species_6emission_nestedoutgrid_fwd
case_ecmwf_06hr_1p0deg_tinymetgrid_1species_1emission_bwd
case_ecmwf_06hr_1p0deg_tinymetgrid_1species_1emission_fwd
case_ecmwf_06hr_1p0deg_tinymetgrid_fpformat_1species_1emission_bwd
case_ecmwf_06hr_1p0deg_tinymetgrid_fpformat_1species_1emission_fwd
case_ecmwf_33hr_1p0deg_tinymetgrid_1species_1emission_bwd
case_ecmwf_33hr_1p0deg_tinymetgrid_1species_1emission_fwd
fp_bwd_12h_ec1p0deg
fp_fwd_12h_ec1p0deg
t1_33h_ec1p0d
ecmwf_1p0deg_tinymet_nestedinput_multipletests case_ecmwf_02hr_1p0deg_tinymetgrid_nestedinput_1species_1emission_fwd
t1_33h_ec0p5d_nest
t1_33h_ec1p0d
ecmwf_02hr_1p0deg case_ecmwf_02hr_1p0deg_1species_1emission_bwd
case_ecmwf_02hr_1p0deg_1species_1emission_fwd
t1_03h_ec1p0d
ecmwf_02hr_1p0deg_grib1-2 case_ecmwf_02hr_1p0deg_grib1-2_1species_1emission_bwd
t1_03h_ec1p0d_grib1-2
ecmwf_02hr_1p0deg_grib2 case_ecmwf_02hr_1p0deg_grib2_1species_1emission_bwd
t1_03h_ec1p0d_grib2
ecmwf_02hr_1p0deg_nestedinput case_ecmwf_02hr_1p0deg_nestedinput_1species_1emission_nestedoutgrid_bwd
case_ecmwf_02hr_1p0deg_nestedinput_1species_1emission_nestedoutgrid_fwd
t1_03h_ec0p5d_nest
t1_03h_ec1p0d
ncep_02hr_1p0deg case_ncep_02hr_1p0deg_1species_1emission_bwd
case_ncep_02hr_1p0deg_1species_1emission_fwd
t1_03h_nc1p0d
ncep_06hr_1p0deg_grib1 case_ncep_02hr_1p0deg_grib1_1species_1emission_fwd
t1_06h_nc1p0d_grib1

XML test files

generic_distro_only.xml only compilation test
test_ecmwf_06hr_1p0deg_tinymetgrid_2species_6emission_fwd_ONLYRUN.xml only success run test
test_ecmwf_02hr_1p0deg_tinymetgrid_2species_6emission_nestedoutgrid_fwd.xml multiple species/emission small run
test_ecmwf_02hr_1p0deg_tinymetgrid_nestedinput_1species_1emission_fwd.xml single species/emission small run
test_ecmwf_06hr_1p0deg_tinymetgrid_1species_1emission_bwd.xml simple backward small run
test_ecmwf_06hr_1p0deg_tinymetgrid_1species_1emission_fwd.xml simple forward small run
test_ecmwf_06hr_1p0deg_tinymetgrid_fpformat_1species_1emission_bwd.xml simple backward run with fp format file
test_ecmwf_06hr_1p0deg_tinymetgrid_fpformat_1species_1emission_fwd.xml simple forward run with fp format file
test_ecmwf_33hr_1p0deg_tinymetgrid_1species_1emission_bwd.xml simple long backward small run
test_ecmwf_33hr_1p0deg_tinymetgrid_1species_1emission_fwd.xml simple long forward small run
test_ecmwf_02hr_1p0deg_nestedinput_1species_1emission_nestedoutgrid_fwd_bwd.xml global + nested simple run
test_ecmwf_02hr_1p0deg_1species_1emission_bwd.xml simple global backward run
test_ecmwf_02hr_1p0deg_1species_1emission_fwd.xml simple global forward run
test_ecmwf_02hr_1p0deg_1species_1emission_fwd_bwd.xml simple global backward - forward run
test_ecmwf_02hr_1p0deg_grib1-2_1species_1emission_bwd.xml simple global backward run with grib1-2 data
test_ecmwf_02hr_1p0deg_grib2_1species_1emission_bwd.xml simple global backward run with grib2 data
test_ecmwf_ncep_02hr_1p0deg_1species_1emission_fwd_bwd.xml simple global backward - forward run with ncep and ecmwf data
test_ncep_02hr_1p0deg_1species_1emission_bwd.xml simple global backward run
test_ncep_02hr_1p0deg_1species_1emission_fwd.xml simple global forward run
test_ncep_02hr_1p0deg_1species_1emission_fwd_bwd.xml simple global backward - forward run
test_ncep_02hr_1p0deg_grib1_1species_1emission_fwd.xml simple global forward run with grib1 data

Flexible wrapping script

Each of the xml test cases can be run independently using the check_v3.py provided in the testing environment. However, it is convenient to be able to run multiple cases at once when performing an evaluation. In this way, the user may select the cases according to the leel of complexity needed or the areas of code modification that shall be tested. In order to facilitat this, a general script is provided.

run_selected_cases.py : this is a simple script that reads in a file of cases to run and executes them sequentially identifying, and listing, any potential problem whether are compilation, exceptions, running or statistical errors.

Usage:

python run_selected_cases.py -f <text file with list of cases> -m <path to makefile>

where the file with the test cases has the following appearance:

test_ecmwf_06hr_1p0deg_tinymetgrid_1species_1emission_fwd.xml
test_ecmwf_06hr_1p0deg_tinymetgrid_1species_1emission_bwd.xml
test_ncep_02hr_1p0deg_1species_1emission_fwd_bwd.xml

the examplea "list_simple_cases.lst" and "list_complex_cases.lst" are given. The output informs at the start about the cases that will be run and, at the end, the time needed to perform all the tests together with various lists of the failing cases, which will be empty if all the tests are run successfully:

[INFO]      The following cases will be executed:
             -) test_ecmwf_06hr_1p0deg_tinymetgrid_1species_1emission_fwd.xml
             -) test_ecmwf_06hr_1p0deg_tinymetgrid_1species_1emission_bwd.xml
             -) test_ncep_02hr_1p0deg_1species_1emission_fwd_bwd.xml
[INFO]      Start execution of the test cases
             ... the time invested on running
                 these tests is variable, from 2
                 minutes to 30 with all cases
 ----------------------------------


[... information for each of the cases as provided by the testing environment ...]


 ********************************************

[INFO]      -------------- SUMMARY-----------------
[INFO]      Failing compilations:  []
[INFO]      Failing runs:  []
[INFO]      Failing cases:  []
[INFO]      Cases raising exceptions:  []


[INFO]      Time invested in running the tests:
            141.243942022  seconds


Outlook

The most recent complex cases have suggested several improvements that will be tackled in the future:

  1. Automatic downloading of large data. This is needed when long tests are to be done. 2 day of data may be around 2GB and it is not good to have heavy files in the distribution.
  2. In order to test CTBTO patches or other modifications affecting the makefile, allow for modification of the makefile on the fly.
  3. Add test cases for depositing species with dry and wet deposition evaluated independently.
  4. Modify the testing environment to use the Vtables provided by the distribution to ensure last version is used.
  5. Include testing of the GRIB2FLEXPART tool as test cases.
Last modified 3 years ago Last modified on Apr 6, 2016, 7:44:18 AM
hosted by ZAMG