Changes between Version 38 and Version 39 of FpCtbtoOverview
- Timestamp:
- Jun 17, 2016, 12:33:53 PM (8 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
FpCtbtoOverview
v38 v39 63 63 Within work package 4, the following developments have been performed: 64 64 65 * '''CONCEPT OF INTERMEDIATE FORMAT AND IMPLEMENTATION''' [wiki:Fp concept Concept of implementation], [wiki:FpformatdocuDocumentation on .fp format]65 * '''CONCEPT OF INTERMEDIATE FORMAT AND IMPLEMENTATION''' [wiki:FpCtbtoWo4Concept Concept of implementation], [wiki:FpCtbtoWo4FpFormat Documentation on .fp format] 66 66 * Documentation on the existing and new workflows was produced 67 67 * The new .fp format was described in detail 68 68 * A number of surprises were encountered, including the discovery that met data in a forward run is not exactly the same data that will be generated in a backward run with the same GRIB files. 69 * '''UPDATES FLEXtest ENVIRONMENT:''' [wiki: TestEnv Update of the testing environment]69 * '''UPDATES FLEXtest ENVIRONMENT:''' [wiki:FpCtbtoWo4TestEnv Update of the testing environment] 70 70 * For development we need to attack the problem through a Test Driven Development (TDD) paradigm, and chose to invest heavily in the expansion of our earlier work, developing a semi-automated approach to rapidly testing new software modifications for the introduction of bugs 71 71 * The environment consists of small tests embedded in the FLEXPART distribution that allows developers to immediately determine if their changes have had detrimental effects. It is also suitable for "outside the distribution" environments to perform large-scale regression tests. It will be a necessary component in future FLEXPART enhancements. 72 * '''GRIB2FLEXPART UTILITY:''' [wiki: PreProcUtil Pre-processing command-line utility]72 * '''GRIB2FLEXPART UTILITY:''' [wiki:FpCtbtoWo4PreProcUtil Pre-processing command-line utility] 73 73 * A flexible command-line utility, ''grib2flexpart'', was developed to pre-process met files outside of a FLEXPART simulation. 74 74 * Simultaneously, FLEXPART was modified to optionally use these ''.fp'' files as input, providing one to two orders of magnitude greater efficiency in input ingest. 75 75 * An exciting potential of this work is that this approach may be used to define an intermediate ''.fp'' met format, to which any met data may be converted, and then used to drive FLEXPART simulations. 76 * '''VERIFICATION TESTS:''' [wiki: VerifPreProcVerification tests]76 * '''VERIFICATION TESTS:''' [wiki:FpCtbtoWo4PreProcVerification Verification tests] 77 77 * The testing environment was used in various ways to 78 78 * Verify that the results of using ''grib2flexpart'' to generate .fp files, and then use the .fp files to drive a FLEXPART run, were identical to those obtained by using FLEXPART with traditional GRIB met input … … 109 109 * Other minor changes meant to get the testing environment up to speed for beginning Work Package 8 included the addition of nested input test cases, as well as general code ehancements 110 110 111 * One of the major problems that was facing our testing environment was that it had a broad collection of test cases - many with poor naming conventions - and we frequently got confused over which cases should be run, or had been run, after majour project milestones. We realized that we wanted the general FLEXPART user to be able to run a single command and have a comprehensive set of tests performed, giving an immediate (within 2-15 minutes, depending on depth of tests) indication of whether the system was running correctly or not. Anything short of this would make it difficult to encourage the discipline of Test Driven Development (TDD) and frequent testing of the system after modifications A description of this enhancement is described in [wiki:Fp testenvConsolidation Consolidation of tests]111 * One of the major problems that was facing our testing environment was that it had a broad collection of test cases - many with poor naming conventions - and we frequently got confused over which cases should be run, or had been run, after majour project milestones. We realized that we wanted the general FLEXPART user to be able to run a single command and have a comprehensive set of tests performed, giving an immediate (within 2-15 minutes, depending on depth of tests) indication of whether the system was running correctly or not. Anything short of this would make it difficult to encourage the discipline of Test Driven Development (TDD) and frequent testing of the system after modifications A description of this enhancement is described in [wiki:FpCtbtoWo8TestEnvConsolidation Consolidation of tests] 112 112 113 113 * The existing test environment documentation was reviewed and modified. A number of cosmetic and code-based changes were made, but the primary changes came from the realization that the documentation was very technical, and would be difficult for new users to understand. Therefore, the primary document, [http://borealscicomp.com/CTBTO_FLEXPART/temporary_documents/wo8_testing_environment/Overview.md Overview.md] was modified with two links at the beginning which pointed to more "hands-on" documents, including a quick-start page [http://borealscicomp.com/CTBTO_FLEXPART/temporary_documents/wo8_testing_environment/QuickStartDistroTesting.md Quick Start]. Note - these two pages are best viewed in a browser with a markdown extension. 114 * In a credit to the value of the testing environment, while trying to add deposition test cases some problems with the Work Package 4 implementation of met input preprocessing was discovered. They are described in more detail at [wiki:Fp formatdepoDiscussion Discussion on Deposition aspects]114 * In a credit to the value of the testing environment, while trying to add deposition test cases some problems with the Work Package 4 implementation of met input preprocessing was discovered. They are described in more detail at [wiki:FpCtbtoWo8FpFormatDepoDiscussion Discussion on Deposition aspects] 115 115 116 116 === 2. Completed FLEXPART code with the VTABLES === 117 117 * The deployment of the Vtable code for NCEP met inputs paralleled that of the ECMWF met inputs from Work Order 6, but several problems were encountered, forcing us to go back and modify some of the code we had implemented for ECMWF (this was actually the reason we chose to split the ECMWF and NCEP deployments into different work orders - we wanted to test the concept first, before putting effort into generalizing it for other types of met inputs). A detailed account of the work is available in the following document 118 * [wiki: VtableNCEPDeploy Deployment of Vtables for NCEP inputs]118 * [wiki:FpCtbtoWo8VtableNcepDeploy Deployment of Vtables for NCEP inputs] 119 119 * A snapshot of the current master branch of the repository has been extracted and is available as flexpart-wo8.tar.gz (about 1 GByte in size - it includes test files), available at http://borealscicomp.com/CTBTO_FLEXPART/snapshot_flexpart_distros/flexpart-wo8.tar.gz 120 120 * A Doxygenated representation of the newest code, and test environment code, is available at the links below. It takes a little bit of exploring, but particularly in the FLEXPART code link, it's possible to view every routine, look at the call and caller trees, and even the source code.