Changes between Initial Version and Version 1 of FpCtbtoWo8TestEnvConsolidation


Ignore:
Timestamp:
Apr 6, 2016, 7:44:18 AM (8 years ago)
Author:
dearn
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • FpCtbtoWo8TestEnvConsolidation

    v1 v1  
     1
     2= Description of the consolidation activities of the testing environment Flex Test =
     3
     4There have been two tasks to make the test more user-friendly:
     5
     61. Use a consistent naming convention for the test cases and xml files to achieve names that are as much self-explanatory as possible
     72. Generation of a flexible script whereby the user can select what cases to run and have a summary of the failing cases, if any.
     8
     9== Naming convention ==
     10
     11'''Case data'''
     12
     13The case data follows the structure:
     14
     15{{{
     16meteorological case
     17    flexpart cases
     18    meteorological data
     19    fp data (whenever applies)
     20}}}
     21
     22The meteorological case is named as [type of met data]_[''length'']_[resolution]_[''specification1'']_[''secification2'']_[...].  For example:
     23
     24* ecmwf_02hr_1p0deg_grib1-2 -->ECMWF data, 2 hours of meteorological data at 1 degree, global (because the contrary is not stated) and in combination GRIB1 and GRIB2 data
     25* ecmwf_02hr_1p0deg_nestedinput --> ECMWF data, 2 hours of meteorological data at GRIB2
     26
     27Inside the general meteorological case, the user can find the meteorological data and the flexpart case data with the set-up and control files. The meteorological data specifies period and resolution t[start hour]_[end hour - length]h_[met data type abbreviation (ec or nc)][resolution]. For the fp data, there is in addition the [fp] label at the front and whether it is [fwd/bwd]. For example:
     28
     29* fp_bwd_12h_ec1p0deg --> fp formatted files, prepared to be run backward, 12 hours of data, based on ECMWF at 1 degree resolution.
     30* t1_33h_ec1p0d --> 33 hours of ECMW data at 1 degree horizontal resolution.
     31
     32Te flexpart cases are named as case_[type of met data]_[simulation lenght in hours]hr_[resolution]_[''specification1'']_[''specification2'']_[number of species]species_[number of emissions]emission_[''specification3'']_[fwd or backward]. Specifications can be GRIB format or whether there is nested input or output. For example:
     33
     34* case_ecmwf_02hr_1p0deg_nestedinput_1species_1emission_nestedoutgrid_fwd --> FLEXPART run driven by ECMWF data, with a simulation length of 2 hours and nested input data, 1 species and one emission and nested outgrid in forward.
     35* case_ecmwf_02hr_1p0deg_grib1-2_1species_1emission_bwd --> FLEXPART run driven by ECMWF data, with a simulation length of 2 hours, with input data GRIB1 and 2, one species one emission and backward.
     36
     37
     38'''XML test files '''
     39
     40These files are located in the '''check''' directory of the testing environment and define what evaluations the user want to do with and from the aforementioned case data. For example, a user may want to use one forward run with ECMWF data and evaluate the RMSE and ABSOLUTE BIAS. The user may combine case data. Each file includes the description of the test cases to run and the statistical parameters that will be calculated.
     41
     42Examples:
     43
     44* test_ecmwf_02hr_1p0deg_1species_1emission_fwd_bwd.xml --> Basic test with a FLEXPART run using ECMWF data at 1 degree, 2 hours simulation both forward and backward, calculating maximum error and RMSE for the entire domain.
     45* test_ecmwf_02hr_1p0deg_nestedinput_1species_1emission_nestedoutgrid_fwd_bwd.xml --> Basic test with a FLEXPART run using ECMWF data at 1 degree, with nested meteorological data,  2 hours simulation both forward and backward, with a nested outrid and calculating maximum error and RMSE for the entire domain.
     46* test_ecmwf_06hr_1p0deg_tinymetgrid_fpformat_1species_1emission_fwd.xml--> Basic test with a FLEXPART run using fp formatted data from ECMWF data at 1 degree but using a small meteorological domain, 6 hours simulation forward, calculating maximum error and RMSE for the entire domain.
     47
     48
     49
     50
     51== Inventory of tests ==
     52
     53A table with the available case data and xml files is provided here.
     54
     55=== Case data ===
     56
     57
     58|| '''ecmwf_1p0deg_tinymet_multipletests''' ||case_ecmwf_03hr_1p0deg_tinymetgrid_2species_6emission_nestedoutgrid_fwd||
     59||                                    ||case_ecmwf_06hr_1p0deg_tinymetgrid_1species_1emission_bwd|| 
     60||                                    ||case_ecmwf_06hr_1p0deg_tinymetgrid_1species_1emission_fwd||
     61||                                    ||case_ecmwf_06hr_1p0deg_tinymetgrid_fpformat_1species_1emission_bwd||
     62||                                    ||case_ecmwf_06hr_1p0deg_tinymetgrid_fpformat_1species_1emission_fwd||
     63||                                    ||case_ecmwf_33hr_1p0deg_tinymetgrid_1species_1emission_bwd||
     64||                                    ||case_ecmwf_33hr_1p0deg_tinymetgrid_1species_1emission_fwd||
     65||                                    || fp_bwd_12h_ec1p0deg||
     66||                                    || fp_fwd_12h_ec1p0deg||
     67||                                    || t1_33h_ec1p0d||
     68|| '''ecmwf_1p0deg_tinymet_nestedinput_multipletests''' ||case_ecmwf_02hr_1p0deg_tinymetgrid_nestedinput_1species_1emission_fwd ||
     69||                                                || t1_33h_ec0p5d_nest||
     70||                                                || t1_33h_ec1p0d||
     71|| '''ecmwf_02hr_1p0deg''' ||case_ecmwf_02hr_1p0deg_1species_1emission_bwd||
     72||                   ||case_ecmwf_02hr_1p0deg_1species_1emission_fwd||
     73||                   || t1_03h_ec1p0d||                 
     74|| '''ecmwf_02hr_1p0deg_grib1-2''' ||case_ecmwf_02hr_1p0deg_grib1-2_1species_1emission_bwd ||
     75||                           ||   t1_03h_ec1p0d_grib1-2||       
     76|| '''ecmwf_02hr_1p0deg_grib2''' ||case_ecmwf_02hr_1p0deg_grib2_1species_1emission_bwd ||
     77||                         ||  t1_03h_ec1p0d_grib2||           
     78|| '''ecmwf_02hr_1p0deg_nestedinput''' ||case_ecmwf_02hr_1p0deg_nestedinput_1species_1emission_nestedoutgrid_bwd ||
     79||                               ||case_ecmwf_02hr_1p0deg_nestedinput_1species_1emission_nestedoutgrid_fwd ||
     80||                               ||  t1_03h_ec0p5d_nest||
     81||                               ||  t1_03h_ec1p0d||           
     82|| '''ncep_02hr_1p0deg''' ||case_ncep_02hr_1p0deg_1species_1emission_bwd ||
     83||                  ||case_ncep_02hr_1p0deg_1species_1emission_fwd ||
     84||                  ||   t1_03h_nc1p0d||               
     85|| '''ncep_06hr_1p0deg_grib1''' ||case_ncep_02hr_1p0deg_grib1_1species_1emission_fwd ||
     86||                        ||    t1_06h_nc1p0d_grib1||
     87
     88=== XML test files ===
     89
     90||generic_distro_only.xml || only compilation test||
     91||test_ecmwf_06hr_1p0deg_tinymetgrid_2species_6emission_fwd_ONLYRUN.xml || only success run test||     
     92||test_ecmwf_02hr_1p0deg_tinymetgrid_2species_6emission_nestedoutgrid_fwd.xml    ||  multiple species/emission small run||
     93||test_ecmwf_02hr_1p0deg_tinymetgrid_nestedinput_1species_1emission_fwd.xml || single species/emission small run||     
     94||test_ecmwf_06hr_1p0deg_tinymetgrid_1species_1emission_bwd.xml || simple backward small run|| 
     95||test_ecmwf_06hr_1p0deg_tinymetgrid_1species_1emission_fwd.xml || simple forward small run||   
     96||test_ecmwf_06hr_1p0deg_tinymetgrid_fpformat_1species_1emission_bwd.xml || simple backward run with fp format file||   
     97||test_ecmwf_06hr_1p0deg_tinymetgrid_fpformat_1species_1emission_fwd.xml || simple forward run with fp format file||   
     98||test_ecmwf_33hr_1p0deg_tinymetgrid_1species_1emission_bwd.xml  || simple long backward small run||   
     99||test_ecmwf_33hr_1p0deg_tinymetgrid_1species_1emission_fwd.xml || simple long forward small run||     
     100||test_ecmwf_02hr_1p0deg_nestedinput_1species_1emission_nestedoutgrid_fwd_bwd.xml || global + nested simple run||
     101||test_ecmwf_02hr_1p0deg_1species_1emission_bwd.xml ||  simple global backward run||   
     102||test_ecmwf_02hr_1p0deg_1species_1emission_fwd.xml ||  simple global forward run||
     103||test_ecmwf_02hr_1p0deg_1species_1emission_fwd_bwd.xml ||  simple global backward - forward run||
     104||test_ecmwf_02hr_1p0deg_grib1-2_1species_1emission_bwd.xml || simple global backward run with grib1-2 data||   
     105||test_ecmwf_02hr_1p0deg_grib2_1species_1emission_bwd.xml || simple global backward run with grib2 data||       
     106||test_ecmwf_ncep_02hr_1p0deg_1species_1emission_fwd_bwd.xml || simple global backward - forward run with ncep and ecmwf data||         
     107||test_ncep_02hr_1p0deg_1species_1emission_bwd.xml  ||  simple global backward run||   
     108||test_ncep_02hr_1p0deg_1species_1emission_fwd.xml ||  simple global forward run||     
     109||test_ncep_02hr_1p0deg_1species_1emission_fwd_bwd.xml   ||  simple global backward - forward run||
     110||test_ncep_02hr_1p0deg_grib1_1species_1emission_fwd.xml ||  simple global forward run with grib1 data||
     111
     112== Flexible wrapping script ==
     113
     114Each of the xml test cases can be run independently using the '''check_v3.py''' provided in the testing environment. However, it is convenient to be able to run multiple cases at once when performing an evaluation. In this way, the user may select the cases according to the leel of complexity needed or the areas of code modification that shall be tested. In order to facilitat this, a general script is provided.
     115
     116'''run_selected_cases.py''' : this is a simple script that reads in a file of cases to run and executes them sequentially identifying, and listing, any potential problem whether are compilation, exceptions, running or statistical errors.
     117
     118Usage:
     119
     120    {{{
     121    python run_selected_cases.py -f <text file with list of cases> -m <path to makefile>
     122    }}}
     123
     124where the file with the test cases has the following appearance:
     125
     126    {{{
     127    test_ecmwf_06hr_1p0deg_tinymetgrid_1species_1emission_fwd.xml
     128    test_ecmwf_06hr_1p0deg_tinymetgrid_1species_1emission_bwd.xml
     129    test_ncep_02hr_1p0deg_1species_1emission_fwd_bwd.xml
     130    }}}
     131
     132the examplea "list_simple_cases.lst"  and "list_complex_cases.lst" are given. The output informs at the start about the cases that will be run and, at the end, the time needed to perform all the tests together with various lists of the failing cases, which will be empty if all the tests are run successfully:
     133
     134{{{
     135
     136[INFO]      The following cases will be executed:
     137             -) test_ecmwf_06hr_1p0deg_tinymetgrid_1species_1emission_fwd.xml
     138             -) test_ecmwf_06hr_1p0deg_tinymetgrid_1species_1emission_bwd.xml
     139             -) test_ncep_02hr_1p0deg_1species_1emission_fwd_bwd.xml
     140[INFO]      Start execution of the test cases
     141             ... the time invested on running
     142                 these tests is variable, from 2
     143                 minutes to 30 with all cases
     144 ----------------------------------
     145
     146
     147[... information for each of the cases as provided by the testing environment ...]
     148
     149
     150 ********************************************
     151
     152[INFO]      -------------- SUMMARY-----------------
     153[INFO]      Failing compilations:  []
     154[INFO]      Failing runs:  []
     155[INFO]      Failing cases:  []
     156[INFO]      Cases raising exceptions:  []
     157
     158
     159[INFO]      Time invested in running the tests:
     160            141.243942022  seconds
     161
     162}}}
     163
     164
     165----
     166
     167== Outlook ==
     168
     169The most recent complex cases have suggested several improvements that will be tackled in the future:
     170
     1711. Automatic downloading of large data. This is needed when long tests are to be done. 2 day of data may be around 2GB and it is not good to have heavy files in the distribution.
     1722. In order to test CTBTO patches or other modifications affecting the makefile, allow for modification of the makefile on the fly.
     1733. Add test cases for depositing species with dry and wet deposition evaluated independently.
     1744. Modify the testing environment to use the Vtables provided by the distribution to ensure last version is used.
     1755. Include testing of the GRIB2FLEXPART tool as test cases.