[0b00607] | 1 | ************ |
---|
[274f9ef] | 2 | Program Flow |
---|
[0b00607] | 3 | ************ |
---|
[274f9ef] | 4 | |
---|
[0b00607] | 5 | |
---|
| 6 | |
---|
| 7 | |
---|
| 8 | General program flow |
---|
| 9 | ==================== |
---|
| 10 | |
---|
| 11 | |
---|
| 12 | The following flow diagram shows the general steps performed by ``flex_extract``. |
---|
| 13 | |
---|
| 14 | .. _ref-fig-submit: |
---|
| 15 | |
---|
| 16 | .. figure:: ../../_files/submit.png |
---|
| 17 | |
---|
| 18 | Overview of the call of python's ``submit.py`` script and raw sequence of working steps done in ``flex_extract``. |
---|
| 19 | |
---|
| 20 | |
---|
| 21 | The ``submit.py`` Python program is called by the Shell script ``run.sh`` or ``run_local.sh`` and accomplish the following steps: |
---|
| 22 | |
---|
| 23 | 1. Setup the control data: |
---|
| 24 | It gets all command-line and ``CONTROL`` file parameters as well as optionally the ECMWF user credentials. Depending the :doc:`app_modes`, it might also prepare a job script which is then send to the ECMWF queue. |
---|
| 25 | 2. Retrieves data from MARS: |
---|
| 26 | It creates and sends MARS-requests either on the local machine or on ECMWF server, that receives the data and stores them in a specific format in GRIB files. If the parameter ``REQUEST`` was set ``1`` the data are not received but a file ``mars_requests.csv`` is created with a list of MARS requests and their settings. If it is set to ``2`` the file is created in addition to retrieving the data. The requests are created in an optimised way by splitting in time, jobs and parameters. |
---|
| 27 | 3. Post-process data to create final ``FLEXPART`` input files: |
---|
| 28 | After all data is retrieved, the disaggregation of flux fields (`see here <../disagg.html>`_ ) is done as well as the calculation of vertical velocity (`see here <../vertco.html>`_) by the Fortran program ``COVERT2``. Eventually, the GRIB fields are merged together such that a single grib file per time step is available with all fields for ``FLEXPART``. Since model level fields are typically in *GRIB2* format whereas surface level fields are still in *GRIB1* format, they can be converted into GRIB2 if parameter ``FORMAT`` is set to *GRIB2*. Please note, however, that older versions of FLEXPART may have difficulties reading pure *GRIB2* files since some parameter IDs change in *GRIB2*. If the retrieval is executed remotely at ECMWF, the resulting files can be communicated to the local gateway server via the ``ECtrans`` utility if the parameter ``ECTRANS`` is set to ``1`` and the parameters ``GATEWAY``, ``DESTINATION`` have been set properly during installation. The status of the transfer can be checked with the command ``ecaccess-ectrans-list`` (on the local gateway server). If the script is executed locally the progress of the script can be followed with the usual Linux tools. |
---|
| 29 | |
---|
| 30 | |
---|
| 31 | |
---|
| 32 | Workflows of different application modes |
---|
| 33 | ======================================== |
---|
| 34 | |
---|
| 35 | More details on how different the program flow is for the different :doc:`app_modes` is sketched in the following diagrams: |
---|
| 36 | |
---|
| 37 | +-------------------------------------------------+------------------------------------------------+ |
---|
| 38 | | .. figure:: ../../_files/mode_remote.png | .. figure:: ../../_files/mode_gateway.png | |
---|
| 39 | +-------------------------------------------------+------------------------------------------------+ |
---|
| 40 | |
---|
| 41 | +-------------------------------------------------+------------------------------------------------+ |
---|
| 42 | | .. figure:: ../../_files/mode_local_member.png | .. figure:: ../../_files/mode_local_public.png | |
---|
| 43 | +-------------------------------------------------+------------------------------------------------+ |
---|
| 44 | |
---|
| 45 | |
---|
| 46 | Example application setting for a local member user |
---|
| 47 | =================================================== |
---|
| 48 | |
---|
| 49 | .. figure:: ../../_files/ex_runlocal_en.png |
---|
| 50 | |
---|
| 51 | |
---|
| 52 | |
---|
| 53 | |
---|
| 54 | .. toctree:: |
---|
| 55 | :hidden: |
---|
| 56 | :maxdepth: 2 |
---|