Changes between Initial Version and Version 1 of FpCtbtoWo4TestEnv


Ignore:
Timestamp:
Nov 5, 2015, 12:20:34 PM (7 years ago)
Author:
dearn
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • FpCtbtoWo4TestEnv

    v1 v1  
     1= FLEXtest =
     2
     3
     4'''FLEXtest''' is a testing environment designed to test multiple aspects of FLEXPART including compilation, execution and comparison (statistics) of runs. It is built in a flexible way so that additional tests and test cases can be built upon it.
     5
     6The primary documentation for FLEXtest is found in the source code distribution in ''flexpart-testing/doc'', in standard (not github) Markdown format.
     7
     8The Overview page of the documentation is presented here, but the user should get the repository and use a good Markdown viewer to best look at the documention.
     9
     10= Overview of Flexpart Testing System
     11
     12
     13The original vision of the testing environment was to make sure that FLEXPART works correctly when initially installed.  Just as many distributions are installed via
     14
     15{{{
     16
     17        make
     18        make test
     19        make install
     20}}}
     21
     22
     23we wanted a way to quickly verify that a new FLEXPART installation was successful and worked correctly on test data.  Further, we wanted to deploy Test Driven Development (TDD) methods, and needed a way to easily test that the FLEXPART system still worked correctly after code changes were made.  The ideal - and this still needs to be achieved - is to have a suite of tests that are automatically performed and validated before any changes can become part of a distribution.
     24
     25From this original vision came a set of requirements
     26
     27* The testing environment requires a core foundation of low-level tools for comparing FLEXPART output in a variety of ways.  The development of these tools will be useful to many who want to analyse FLEXPART runs against each other for whatever reasonThe testing environment requires a core foundation of low-level tools for comparing FLEXPART output in a variety of ways.  The development of these tools will be useful to many who want to analyse FLEXPART runs against each other for whatever reason
     28
     29* The overall testing environment, like the "make; make test" paradigm requires a structured ability to test that code compiles and to test its execution on a variety of problems.
     30
     31  * The original vision was somewhat analogous to unit and system testing, in that tests should be very easy to run and should be quick.  This encourages the use of the testing environment, allowing developers to work more from a Test Driven Development (TDD) or Behaviour Driven Development (BDD) environment.
     32
     33  * It was also recognised early on that many users would be interested in larger scale and longer tests that might run overnight, or even for days.
     34
     35 * The environment we created is based on the concept of testing a FLEXPART distribution with a specific Makefile and configuration of par_mod.f90.  To test this distribution, it would be necessary to make sure it compiles, and then run a number of test cases and make sure they all pass.
     36
     37
     38In the big pict!ure, we consider a hierarchy of a Distribution test containing ''!MetCases'', which in turn contain ''!RunCases'', which in turn contain ''!BasicTests''.  Specific test suites are in user-defined XML files, such as the following simple example
     39
     40
     41{{{
     42        <distribution>
     43
     44            <!-- Sample XML testing namelist -->
     45
     46            <short_descr>
     47                Basic distro
     48            </short_descr>
     49
     50            <distropath>
     51                /home/morton/flexpart
     52            </distropath>
     53
     54            <makefilepath>
     55                /home/morton/makefile.tmp
     56            </makefilepath>
     57
     58            <parmodpath>
     59                /home/morton/par_mod.f90
     60            </parmodpath>
     61
     62            <metcase>
     63
     64                <short_descr>
     65                    ECMWF patch
     66                </short_descr>
     67
     68                <metfiledir>
     69                    /home/morton/metfiles
     70                </metfiledir>
     71
     72                <availablepath>
     73                    /home/morton/metfiles/AVAILABLE
     74                </availablepath>
     75
     76                <runcase>
     77                    <short_descr>
     78                        Run case #1
     79                    </short_descr>
     80
     81                    <casedir>
     82                        /home/morton/casedir/case01
     83                    </casedir>
     84
     85                    <controldatadir>
     86                        /home/morton/casedir/case01/controldata
     87                    </controldatadir>
     88
     89                    <basictest>
     90
     91                        <short_descr>
     92                            Max error over entire temporal and spatial domain
     93                        </short_descr>
     94
     95                        <type>
     96                            max_error
     97                        </type>
     98
     99                        <max_threshold>
     100                           1.0E-3
     101                        </max_threshold>
     102                    </basictest>
     103
     104                    <basictest>
     105                        <short_descr>
     106                            RMS error over entire temporal and spatial domain
     107                        </short_descr>
     108
     109                        <type>
     110                            rmse
     111                        </type>
     112
     113                        <max_threshold>
     114                           1.0E-6
     115                        </max_threshold>
     116                    </basictest>
     117
     118                </runcase>
     119
     120
     121            </metcase>
     122
     123
     124        </distribution>
     125
     126}}}
     127
     128
     129A ''Distribution'' test consists of a collection of one or more ''!MetCases''.  A single ''!MetCase'' is just a timeseries of met files (for now, either NCEP or ECMWF) which could be used as input for a large variety of simulations - regional, global, forward, backward, different number of species, release/receptor points, parameterisations (e.g. convection or not), etc.
     130
     131A single ''!MetCase'' consists of a collection of one or more ''!RunCases''.  A single ''!RunCase'' is a simulation of FLEXPART with a specific configuration.  A ''!RunCase'' contains all of the configuration files that define the simulation, as well as a set of control output which defines expected correct output.  When the ''!RunCase'' is deployed, FLEXPART runs as defined and produces a new set of output.  This output is compared with the control output in user-selected ways through one or more !BasicTests
     132
     133A single ''!BasicTest'' defines the type of test to be performed - type of statistics, what parts of the output will be compared, etc., and a threshold that defines Pass or Fail.
     134
     135
     136
     137
     138
     139### Implementation of the Testing Environment
     140
     141* All Python based (2.7) with !NumPy integration
     142* Set of low-level routines (in ''flexpart-testing/flextest/'')
     143   * '''!FlexpartOutput''' - class that provides low level access to Flexpart output.  Based on modifications of John Burkhart's ''pflexible'' from 2011.  It provides methods to extract portions of output ranging from an entire multidimensional timeseries volume of all variables (levels, species, releases, etc.) or a subset down to a single horizontal slice for a single set of output at a single time, and everything in between.
     144   * '''!FlexpartErrors''' - extensible class that expects two !FlexpartOutput objects (presumably a test and control) at instantiation, and supplies the methods to perform a variety of error calculations on user-defined slices of the FLEXPART output domain.  These methods' roles are to simply return a numerical value, and leave it to calling routines to make decisions based on these values
     145   * '''!FlexpartCase''' - This class sets up a configured FLEXPART simulation in a temporary directory, uses a specified FLEXPART executable, and provides confirmation on whether the simulation completed successfully or not.  If so, then users might use the previously-referenced ''!FlexpartErrors()'' to compare the new output with the control output.
     146   * '''!FlexpartExecutable'''  - This class takes a FLEXPART code distribution with makefile and par_mod.f90 (which may be located outside the distribution for customisation), copies to a temporary directory, attempts to compile, and reports on success and location of the executable
     147
     148* Higher level testing environment (in ''flexpart-testing/distrotest/'') - uses the foundation provided by the low-level routines described above. 
     149    * '''!TestSuite'''  - this is a high-level container that stores all information about a set of tests to perform.  It reads an XML specification file and creates a data hierarchy of Distributions (testing successful compilation and using the executable for tests),
     150    * '''Distribution''' - this is a container that stores properties of a particular FLEXPART distribution test including paths to distribution, Makefile and par_mod.f90, and a collection of !MetCase hierarchical objects
     151    * '''!MetCase''' - this is a container that stores properties of a particular !MetCase including location of met files (which should have an AVAILABLE file) and a collection of !RunCase objects
     152    * '''!RunCase''' - this is a container that stores properties of a particular !RunCase, including the simulation config files (like COMMAND, RELEASE, etc.), location of the control_output data to be compared against representing a single simulation, and a collection of zero or more !BasicTests, each of which describes the test to be performed on the simulation output.  If the simulation is successful, the !RunCase will have information on location of both control and test output data, which can then be examined by !BasicTests
     153    * '''!BasicTest''' - this is a container for testing specifications including the type of test to perform and the threshold value for the test.
     154
     155* Test driver, ''check.py'' (located in ''flexpart-testing/check/'') - This is a Python program that expects an XML file as an argument, then creates a ''!TestSuite'' consisting of all the components described above.  Then it proceeds through the hierarchy of ''!Distribution'', ''!MetCase'', ''!RunCase'' and ''!BasicTest'', performing the compilations, simulations and testing. 
     156* The distribution test cases - set of relatively small cases are part of the current FLEXPART distribution.  They are generally set up so that user only needs to specify location of a custom Makefile
     157
     158----
     159== Distribution testing environment
     160
     161
     162A set of test case XML specification files are available in flexpart-testing/check/.  These tests are set up so that the only parameter in the XML files that needs to be customized for the local environment is the path to the makefile (someday we will have the ability to auto-generate the makefiles, but we are not there yet).
     163
     164
     165{{{
     166        .
     167        ├── basic_ec_nc_testing.xml
     168        ├── check.py
     169        ├── ec_fwd_bwd_testing.xml
     170        ├── generic1.xml
     171        ├── generic_distro_only.xml
     172        ├── generic_no_basictest.xml
     173        ├── nc_fwd_bwd_testing.xml
     174        ├── single_ec_bwd.xml
     175        ├── single_ec_fwd.xml
     176        ├── single_nc_bwd.xml
     177        ├── single_nc_fwd.xml
     178        ├── so_simple_ec_06h_bwd_preproc.xml
     179        ├── so_simple_ec_06h_bwd.xml
     180        ├── so_simple_ec_06h_fwd_preproc.xml
     181        ├── so_simple_ec_06h_fwd.xml
     182        ├── so_simple_ec_33h_bwd.xml
     183        └── so_simple_ec_33h_fwd.xml
     184}}}
     185
     186check.py is the driver we have been using so far, and it expects a single argument - the path to an XML specification file.  A sample run follows:
     187
     188
     189{{{
     190        $ python check.py so_simple_ec_06h_bwd.xml
     191        Testing of FLEXPART for very small ECMWF met file
     192        ../../flexpart_code
     193        /home/morton/makefile.gfortran
     194        ../case_data/ecmwf_tiny/so_simple_ec_06h_bwd/par_mod.F90
     195        distro_destdir_name: /tmp/distrotest_d1104f82-2d99-4fb1-912f-0b9a16ecf850
     196        Executable exists: True
     197
     198        ============================
     199
     200        compile test...
     201        Compile directory: /tmp/distrotest_d1104f82-2d99-4fb1-912f-0b9a16ecf850
     202        compile_success: True
     203        Executable exists: True
     204
     205        ============================
     206
     207        [<distrotest.MetCase.MetCase object at 0x7ff1d578b810>]
     208
     209        ****************************
     210        Running MetCase: ECMWF patch - 11x11 1.0 degrees over Austria
     211        Met file dir: /home/morton/git/ctbto/flexpart-testing/case_data/ecmwf_tiny/met_data
     212        ****************************
     213
     214        Running RunCase: Backward run, 1 species, 1 emissions, 6 hours
     215        Case dir: /home/morton/git/ctbto/flexpart-testing/case_data/ecmwf_tiny/so_simple_ec_06h_bwd/rundir_template
     216
     217        ============================
     218        Case Test Backward run, 1 species, 1 emissions, 6 hours...
     219        Case template directory: /home/morton/git/ctbto/flexpart-testing/case_data/ecmwf_tiny/so_simple_ec_06h_bwd/rundir_template
     220        Case run directory: /tmp/caserun_12e6f6ac-82bf-4969-8777-c2a99be29d49
     221        Control data directory: /home/morton/git/ctbto/flexpart-testing/case_data/ecmwf_tiny/so_simple_ec_06h_bwd/control_output
     222        Met file dir: /home/morton/git/ctbto/flexpart-testing/case_data/ecmwf_tiny/met_data
     223
     224        Executable: /tmp/distrotest_d1104f82-2d99-4fb1-912f-0b9a16ecf850/FLEXPART_GFORTRAN
     225        Number of basic tests: 2
     226        ============================
     227
     228        run_success: True
     229        Execution time: 3.33E+00 seconds
     230
     231        ============================
     232
     233
     234        -----------------------
     235        Basic Test
     236        Description: Max error over entire temporal and spatial mother domain
     237        Test type: mother_all_vars_maxabserr
     238        Threshold: 1.0E-03
     239        Test performed.  Error = 0.0E+00
     240        Test passed
     241        -----------------------
     242
     243
     244        -----------------------
     245        Basic Test
     246        Description: RMS error over entire temporal and spatial mother domain
     247        Test type: mother_all_vars_rmse
     248        Threshold: 1.0E-03
     249        Test performed.  Error = 0.0E+00
     250        Test passed
     251        -----------------------
     252
     253}}}
     254
     255
     256If there is a failure in any part of the test, messages will typically tell the user where to find the test so that they can try to look at error logs and/or run it manually to gain necessary insight.
     257
     258----
     259
     260== Notes on Documentation
     261
     262To help keep the code and documentation better integrated, we chose to keep the documentation with the code, in the ''flexpart-testing/doc'' directory, in Markdown format.  In this way, we hope that developers are more likely to revise the documentation when the code is updated, and keep it in version control.
     263
     264This documentation is written in plain Markdown (not necessarily the same Markdown you see in github).  Specifically, it has been edited with [https://remarkableapp.github.io/linux.html Remarkable] , a Markdown editor for Linux.   In the ''Style'' menu, the ''Markdown'' option is chosen, rather than the default ''Github''.
     265
     266It is best viewed through a web browser with a Markdown Extension
     267
     268* [Chrome Markdown Extension](https://github.com/borismus/markdown-preview) (when installing this extension, be sure to pay attention to the note about selecting ''Allow access to file URL's'')
     269* [Firefox Markdown Viewer](https://addons.mozilla.org/en-us/firefox/addon/markdown-viewer/)