Changeset f20342a in flex_extract.git for For_developers/Sphinx/source/Documentation/Overview/prog_flow.rst
- Timestamp:
- May 27, 2020, 8:01:54 PM (4 years ago)
- Branches:
- master, ctbto, dev
- Children:
- 550435b
- Parents:
- a14839a
- File:
-
- 1 edited
Legend:
- Unmodified
- Added
- Removed
-
For_developers/Sphinx/source/Documentation/Overview/prog_flow.rst
rb1674ed rf20342a 1 1 ************ 2 Program Flow2 Program flow 3 3 ************ 4 4 … … 16 16 .. figure:: ../../_files/submit.png 17 17 18 Overview of the call of python's ``submit.py`` script and raw sequence of working steps donein ``flex_extract``.18 Overview of the call of the``submit.py`` Python script and raw sequence of work steps in ``flex_extract``. 19 19 20 20 21 The ``submit.py`` Python program is called by the Shell script ``run.sh`` or ``run_local.sh`` and accomplishthe following steps:21 The ``submit.py`` Python script is called by the shell script ``run.sh`` or ``run_local.sh`` and accomplishes the following steps: 22 22 23 1. Setup thecontrol data:24 It gets all command-line and ``CONTROL`` file parameters as well as optionally the ECMWF user credentials. Depending the :doc:`app_modes`, it might also prepare a job script which is then sendto the ECMWF queue.25 2. Retriev esdata from MARS:26 It creates and sends MARS-requests either on the local machine or on ECMWF server, that receives the data and stores them in a specific format in GRIB files. If the parameter ``REQUEST`` was set ``1`` the data are not received but a file ``mars_requests.csv`` is created with a list of MARS requests and their settings. If it is set to ``2`` the file is created in addition to retrieving the data. The requests are created in an optimised way by splitting intime, jobs and parameters.27 3. Post-process data to create final ``FLEXPART`` input files:28 After all data is retrieved, the disaggregation of flux fields (`see here <../disagg.html>`_ ) is done as well as the calculation of vertical velocity (`see here <../vertco.html>`_) by the Fortran program ``calc_etadot``. Eventually, the GRIB fields are merged together such that a single grib file per time step is available with all fields for ``FLEXPART``. Since model level fields are typically in *GRIB2* format whereas surface level fields are still in *GRIB1* format, they can be converted into GRIB2 if parameter ``FORMAT`` is set to *GRIB2*. Please note, however, that older versions of FLEXPART may have difficulties reading pure *GRIB2* files since some parameter IDs change in *GRIB2*. If the retrieval is executed remotely at ECMWF, the resulting files can be communicated to the local gateway server via the ``ECtrans`` utility if the parameter ``ECTRANS`` is set to ``1`` and the parameters ``GATEWAY``, ``DESTINATION`` have been set properly during installation. The status of the transfer can be checked with the command ``ecaccess-ectrans-list`` (on the local gateway server). If the script is executed locallythe progress of the script can be followed with the usual Linux tools.23 1. Setup of control data: 24 Command-line and ``CONTROL``-file parameters are read, as well as (optionally) the ECMWF user credentials. Depending the :doc:`app_modes`, a job script might be prepared which is then sent to the ECMWF queue. 25 2. Retrieval of data from MARS: 26 MARS requests are created either on the local machine or on the ECMWF server and then submitted which retrieve the data and store them in GRIB files. If the parameter ``REQUEST`` was set ``1``, the data are not retrieved and instead a file ``mars_requests.csv`` is created, which contains a list of the MARS requests and their settings. If ``REQEST`` is set to ``2``, the csv file is created in addition to retrieving the data. The requests are created in an optimised way by splitting with respect to time, jobs and parameters. 27 3. Post-processing of data to create final ``FLEXPART`` input files: 28 After all data have been retrieved, flux fields are disaggregated (`see here <../disagg.html>`_ ) and the vertical velocity is calculated (`see here <../vertco.html>`_) by the Fortran program ``calc_etadot``. Finally, the GRIB fields are merged into a single grib file per time step containing all the fields for ``FLEXPART``. Since model-level fields are typically in *GRIB2* format, whereas surface-level fields are still in *GRIB1* format, they will be converted into GRIB2 if parameter ``FORMAT`` is set to *GRIB2*. Please note, however, that older versions of FLEXPART may have difficulties to read these *GRIB2* files since some parameter IDs have been change in *GRIB2*. If the retrieval is executed remotely at ECMWF, the resulting files will be sent to the local gateway server via the ``ECtrans`` utility if the parameter ``ECTRANS`` is set to ``1`` and the parameters ``GATEWAY``, ``DESTINATION`` have been set properly during installation. The status of the transfer can be checked with the command ``ecaccess-ectrans-list`` (on the local gateway server). If the script is executed locally, the progress of the script can be followed with the usual Linux tools. 29 29 30 30 … … 33 33 ======================================== 34 34 35 More details on how different the program flow is for the different :doc:`app_modes` is sketched in the following diagrams:35 The following diagrams show how different the program flow is for the different :doc:`app_modes`: 36 36 37 37 +-------------------------------------------------+------------------------------------------------+
Note: See TracChangeset
for help on using the changeset viewer.