Changeset 0b00607 in flex_extract.git for for_developers


Ignore:
Timestamp:
Jul 29, 2019, 8:23:57 AM (5 years ago)
Author:
Anne Philipp <anne.philipp@…>
Branches:
master, ctbto, dev
Children:
bc27d19
Parents:
41c9dbc
Message:

Documentation status version 0

Location:
for_developers/Sphinx
Files:
33 added
16 deleted
24 edited

Legend:

Unmodified
Added
Removed

  • for_developers/Sphinx/Makefile

    r41c9dbc r0b00607  
    77SPHINXPROJ    = flex_extract
    88SOURCEDIR     = source
    9 BUILDDIR      = ../../documentation
     9BUILDDIR      = ../../Documentation
    1010
    1111# Put it first so that "make" without argument is like "make help".

  • for_developers/Sphinx/source/Developers/gen_docu.rst

    r41c9dbc r0b00607  
    1515- graphviz
    1616- sphinx
     17- pip install sphinxcontrib-exceltable
     18- pip install seqdiag
     19- pip install sphinxcontrib-seqdiag
     20- pip install sphinxcontrib-blockdiag
     21- pip install pycallgraph
     22
     23#- pip install sphinx-fortran
    1724
    1825
    … …  
    2734
    2835
     36Sequence diagramms
     37------------------
     38
     39You might need to adapt the fonts path for the diagrams to a true type font. Currently it is set to:
     40
     41.. code-block:: bash
     42
     43    # Fontpath for seqdiag (truetype font)
     44    seqdiag_fontpath = '/usr/share/fonts/dejavu/DejaVuSerif.ttf'
     45
     46
     47Block diagramms
     48------------------
     49
     50You might need to adapt the fonts path for the diagrams to a true type font. Currently it is set to:
     51
     52.. code-block:: bash
     53
     54     # Fontpath for blockdiag (truetype font)
     55     blockdiag_fontpath = '/usr/share/fonts/dejavu/DejaVuSerif.ttf'
     56
    2957
    3058

  • for_developers/Sphinx/source/Documentation/Input/control.rst

    r41c9dbc r0b00607  
    1818The optional extra indications for the re-analysis datasets mark the files for *public users*
    1919and *global* domain. For the operational datasets (*OD*) the file names contain also information of
    20 the stream, the field type for forecasts, the method for extracting the vertical coordinate and other
    21 things like time or horizontal resolution.
     20the stream, the field type for forecasts, the method for extracting the vertical coordinate and other things like time or horizontal resolution.
    2221
    2322
    2423Format of CONTROL files
    2524----------------------------------
    26 
     25The first string of each line is the parameter name, the following string(s) (separated by spaces) is (are) the parameter values.
    2726The parameters can be sorted in any order with one parameter per line.
    2827Comments are started with a '#' - sign. Some of these parameters can be overruled by the command line
    … …  
    3029All parameters have default values. Only those parameters which have to be changed
    3130must be listed in the :literal:`CONTROL` files.
     31
    3232
    3333Example CONTROL files
    … …  
    3737They can be used as a template for adaptations and understand what's possible to
    3838retrieve from ECMWF's archive.
    39 For each main dataset there is an example and additionally some variances in resolution, type of field
    40 or type of retrieving the vertical coordinate.
     39For each main dataset there is an example and additionally some variances in resolution, type of field or type of retrieving the vertical coordinate.
    4140
    4241
    … …  
    5049the setting of these parameters.
    5150
    52 .. literalinclude:: ../../../../../run/control/CONTROL.documentation
     51.. literalinclude:: ../../../../../Run/Control/CONTROL.documentation
    5352   :language: bash
    5453   :caption: CONTROL.documentation

  • for_developers/Sphinx/source/Documentation/Input/control_params.rst

    r41c9dbc r0b00607  
    22The CONTROL parameters
    33======================
     4     
    45   
    56   
    … …  
    910************
    1011   
    11 .. csv-table::
    12    :file: ../../_files/CONTROLparameter-user.csv
    13    :header-rows: 1
    14    :widths: auto
     12.. exceltable:: User parameter in CONTROL file
     13   :file: ../../_files/CONTROLparameter.xls
     14   :sheet: UserSection
     15   :header: 1
    1516
    1617.. _ref-control-general:
    … …  
    1920***************
    2021
    21 .. csv-table::
    22    :file: ../../_files/CONTROLparameter-general.csv
    23    :header-rows: 1
    24    :widths: auto
     22.. exceltable:: General parameter in CONTROL file
     23   :file: ../../_files/CONTROLparameter.xls
     24   :sheet: GeneralSection
     25   :header: 1
    2526   
    26 .. _ref-control-time:
     27.. _ref-control-time: 
    2728   
    2829Time Section
    2930************
    3031   
    31 .. csv-table::
    32    :file: ../../_files/CONTROLparameter-time.csv
    33    :header-rows: 1
    34    :widths: auto       
     32.. exceltable:: Time parameter in CONTROL file
     33   :file: ../../_files/CONTROLparameter.xls
     34   :sheet: TimeSection 
     35   :header: 1     
    3536   
    3637
    … …  
    3839   
    3940Data Section
    40 ************
     41************ 
    4142   
    42 .. csv-table::
    43    :file: ../../_files/CONTROLparameter-data.csv
    44    :header-rows: 1
    45    :widths: auto   
     43.. exceltable:: Data parameter in CONTROL file
     44   :file: ../../_files/CONTROLparameter.xls
     45   :sheet: DataSection
     46   :header: 1
    4647   
    4748   
    … …  
    5152******************
    5253   
    53 .. csv-table::
    54    :file: ../../_files/CONTROLparameter-datafields.csv
    55    :header-rows: 1
    56    :widths: auto     
     54.. exceltable:: Data field parameter in CONTROL file
     55   :file: ../../_files/CONTROLparameter.xls
     56   :sheet: DatafieldsSection
     57   :header: 1
    5758   
    5859   
    … …  
    6263*****************
    6364
    64 .. csv-table::
    65    :file: ../../_files/CONTROLparameter-fluxdata.csv
    66    :header-rows: 1
    67    :widths: auto
     65.. exceltable:: Flux data parameter in CONTROL file
     66   :file: ../../_files/CONTROLparameter.xls
     67   :sheet: FluxDataSection
     68   :header: 1
    6869
    6970 
    70  .. _ref-control-domain:
     71.. _ref-control-domain:
    7172   
    7273Domain Section
    7374**************
    7475   
    75 .. csv-table::
    76    :file: ../../_files/CONTROLparameter-domain.csv
    77    :header-rows: 1
    78    :widths: auto     
     76.. exceltable:: Domain parameter in CONTROL file
     77   :file: ../../_files/CONTROLparameter.xls
     78   :sheet: DomainSection
     79   :header: 1
     80   :widths: 10,10,10,5,20,20
    7981   
    8082   
    8183         
    82  .. _ref-control-verticalwind:
     84.. _ref-control-verticalwind:
    8385   
    8486Vertical wind Section
    8587*********************
    8688   
    87 .. csv-table::
    88    :file: ../../_files/CONTROLparameter-verticalwind.csv
    89    :header-rows: 1
    90    :widths: auto 
    91  
     89.. exceltable:: Vertical wind parameter in CONTROL file
     90   :file: ../../_files/CONTROLparameter.xls
     91   :sheet: VerticalWindSection
     92   :header: 1
     93    
    9294 
    9395.. _ref-control-adddata:
    … …  
    9698***********************
    9799   
    98 .. csv-table::
    99    :file: ../../_files/CONTROLparameter-adddata.csv
    100    :header-rows: 1
    101    :widths: auto   
     100.. exceltable:: Additional data parameter in CONTROL file
     101   :file: ../../_files/CONTROLparameter.xls
     102   :sheet: AddDataSection
     103   :header: 1
    102104 
    103  
    104 .. _ref-control-flexpart:
    105    
    106 Flexpart Section
    107 ****************
    108    
    109 .. csv-table::
    110    :file: ../../_files/CONTROLparameter-flexpart.csv
    111    :header-rows: 1
    112    :widths: auto 
    113    
    114    
     105   
    115106   
    116107   

  • for_developers/Sphinx/source/Documentation/Input/fortran_makefile.rst

    r41c9dbc r0b00607  
    1 **********************************
    2 The Fortran Program - ``CONVERT2``
    3 **********************************
     1***********************************
     2The Fortran Makefile - ``CONVERT2``
     3***********************************
    44
    55.. _ref-convert:
    66
    7 One of ``flex_extract``'s components is a Fortran program called ``CONVERT2``.
    8 This will be compiled during the installation process to get an executable. ``flex_extract``
    9 has a couple of makefiles prepared which are listed in the following.
     7``Flex_extract``'s Fortran program will be compiled during
     8the installation process to get the executable named ``CONVERT2``.
     9
     10``flex_extract`` has a couple of ``Makefiles`` prepared which can be found in the directory
     11``flex_extract_vX.X/source/fortran``, where ``vX.X`` should be substituted with the current version number.
     12A list of these ``Makefiles`` are shown below:
    1013
    1114
    … …  
    2831    | For the use with ifort compiler.
    2932
    30 They can be found in the path ``flex_extract_vX.X/source/fortran``, where
    31 ``vX.X`` should be substituted with the current version number.
    3233
    33 So starting from the root directory of ``flex_extract``,
    34 go to the ``Fortran`` source directory and open the ``Makefile`` of your
    35 choice to modify with an editor of your choice. We use the ``nedit`` in this case.
    36 So far, we tested ``flex_extract`` with a ``gfortran`` and an ``ifort`` compiler. 
    37 
    38 .. code-block:: bash
    39 
    40    $ cd flex_extract_vX.X/source/fortran
    41    $ nedit Makefile.local.gfortran
     34For instructions on how to adapt the ``Makefiles`` for the local application mode
     35please see :ref:`ref-install-local`.
    4236
    4337
    44 Edit the pathes to the ``eccodes`` library on your local machine.
    45 
    46 .. caution::
    47    This can vary from system to system.
    48    It is suggested to use a command like
    49 
    50    .. code-block:: bash
    51 
    52       # for the ECCODES_INCLUDE_DIR path do:
    53       $ dpkg -L libeccodes-dev | grep eccodes.mod
    54       # for the ECCODES_LIB path do:
    55       $ dpkg -L libeccodes-dev | grep libeccodes.so
    56      
    57    to find out the path to the ``eccodes`` library.
    58    
    59 Substitute these paths in the ``Makefile`` for parameters **ECCODES_INCLUDE_DIR**
    60 and **ECCODES_LIB** and save it.
    61 
    62 .. code-block:: bash
    63 
    64    # these are the paths on a current Debian Testing system (May 2019)
    65    ECCODES_INCLUDE_DIR=/usr/lib/x86_64-linux-gnu/fortran/gfortran-mod-15/
    66    ECCODES_LIB= -L/usr/lib -leccodes_f90 -leccodes -lm 
    67    
    6838   
    6939.. toctree::

  • for_developers/Sphinx/source/Documentation/Input/setup.rst

    r41c9dbc r0b00607  
    33**************************************
    44
    5 .. _ref-setup:
    6 
    7 
    8 The installation script ``setup.sh`` prepares the call of the Python
    9 installation script (``install.py``) with its necessary command line arguments.
    10 The ``setup.sh`` script is located in the root directory of ``flex_extract`` and has a
    11 section labelled with "AVAILABLE COMMANDLINE ARGUMENTS TO SET" which
    12 contains the available command line parameters for ``install.py``. The user has to adapt these
    13 parameters for his personal use. The parameters are listed and described in the following table.
    14 The script also does some checks to guarantee necessary parameters were set.
    15    
    16 .. csv-table::
    17    :file: ../../_files/InstallationParameters.csv
    18    :header-rows: 1
    19    :widths: auto   
    20 
    21 After the installation some tests can be conducted.
    22 They are described in section :ref:`ref-testinstallfe`.
     5
     6The installation of ``flex_extract`` is done by the Shell script ``setup.sh`` which is located in the root directory of ``flex_extract``.
     7It calls the top-level Python script ``install.py`` which does all necessary operations to prepare the selected application environment. This includes:
     8
     9- preparing the file ``ECMWF_ENV`` with the user credentials for member state access to ECMWF servers (in **remote** and **gateway** mode)
     10- preparation of a compilation Korn-shell script (in **remote** and **gateway** mode)
     11- preparation of a job template with user credentials (in **remote** and **gateway** mode)
     12- create a tar-ball of all necessary files
     13- copying tar-ball to target location (depending on application mode and installation path)
     14- submit compilation script to batch queue at ECMWF servers (in **remote** and **gateway** mode) or just untar tar-ball at target location (**local mode**)
     15- compilation of the FORTRAN90 program ``CONVERT2``
     16
     17
     18The Python installation script ``install.py`` has a couple of command line arguments which are defined in ``setup.sh`` in the section labelled with "*AVAILABLE COMMANDLINE ARGUMENTS TO SET*". The user has to adapt these parameters for his personal use. The parameters are listed and described in :ref:`ref-instparams`. The script also does some checks to guarantee necessary parameters were set.
     19   
     20After the installation process, some tests can be conducted. They are described in section :ref:`ref-testinstallfe`.
     21
     22The following diagram sketches the involved files and scripts in the installation process:
     23
     24.. _ref-install-blockdiag:
     25
     26.. blockdiag::
     27
     28   blockdiag {
     29   
     30     default_fontsize = 24;
     31     
     32     // Set node metrix
     33     node_width = 300; // default is 128
     34   
     35     // Set span metrix
     36     span_width = 80;  // default value is 64
     37     
     38     ECMWF_ENV.template [shape = flowchart.input];
     39     compilejob.template [shape = flowchart.input];
     40     job.template [shape = flowchart.input];
     41     
     42     compilejob.ksh [shape = roundedbox];
     43    // tarball
     44    // ECMWF_ENV
     45    // job.temp
     46     
     47     "CONTROL file" [shape = flowchart.input];
     48     
     49     setup.sh [shape = roundedbox];
     50     install.py [shape = roundedbox];
     51     
     52     "ECMWF server"  [shape = flowchart.terminator];
     53     
     54     //beginpoint [shape = beginpoint];
     55     
     56     orientation = landscape;
     57     
     58     //beginpoint -> setup.sh;
     59     setup.sh -> install.py;
     60     
     61     install.py <- "CONTROL file";
     62                 
     63     install.py -> ECMWF_ENV, job.temp, compilejob.ksh, tarball;   
     64
     65     ECMWF_ENV.template, job.template, compilejob.template  -> install.py;
     66     
     67     tarball, compilejob.ksh -> "ECMWF server";
     68     
     69         
     70     group exec {
     71       // set backgound color
     72       color = "#FF6633";
     73       
     74       orientation = portrait;
     75       setup.sh;
     76       install.py;
     77     }
     78     
     79     group out {
     80       color = "#FFFFFF";
     81       group output {
     82         color = "#99FF99";
     83         ECMWF_ENV;
     84         job.temp;
     85         compilejob.ksh;
     86       }
     87       
     88       group ECMWF {
     89         color = "#006600";
     90         tarball;
     91         
     92       }
     93     }
     94 
     95     group input {
     96       
     97       color = "#FFFFFF";
     98       
     99       group temps {
     100         color = "#66CCFF";
     101                 
     102         ECMWF_ENV.template;
     103         job.template;
     104         compilejob.template;
     105       }
     106       
     107       group in {
     108         color = "#3366FF";
     109         "CONTROL file";
     110       }
     111     }
     112     
     113   }
     114
     115
     116.. blockdiag::
     117   :caption: Diagram of data flow during the installation process. The trapezoids are input files with the light blue area being the template files. The edge-rounded, orange boxes are the executable files which start the installation process and reads the input files. The rectangular, green boxes are the output files. The light green files are files which are only needed in the remota and gateway mode.
     118
     119   blockdiag {
     120   
     121     group{
     122       orientation = portrait;
     123       label = "Legend";
     124       fontsize = 28;
     125       color = "#FFFFFF";
     126       'executable scripts' [shape = roundedbox];
     127       'input files' [shape = flowchart.input];
     128       'output files';
     129        server [shape = flowchart.terminator];
     130     }
     131   }
     132
     133.. _ref-instparams:
     134
     135Installation Parameter
     136----------------------
     137   
     138.. exceltable:: Parameter for Installation
     139    :file:  ../../_files/InstallationParameter.xls
     140    :header: 1 
     141   
     142
     143
     144Content of ``setup.sh``
     145-----------------------
     146 
     147.. literalinclude:: ../../../../../setup.sh
     148   :language: bash
     149   :caption: setup.sh
     150       
     151       
     152.. _ref-install-script:
     153       
     154Usage of ``install.py`` (optional)
     155----------------------------------
     156
     157It is also possible to start the installation process of ``flex_extract`` directly from command line by using the ``install.py`` script instead of the wrapping Shell script ``setup.sh``.  This top-level script is located in
     158``flex_extract_vX.X/source/python`` and is executable. With the ``help`` parameter we see again all possible
     159command line parameter.
     160
     161.. code-block:: bash
     162 
     163    install.py --help
     164   
     165    usage: install.py [-h] [--target INSTALL_TARGET] [--makefile MAKEFILE]
     166                  [--ecuid ECUID] [--ecgid ECGID] [--gateway GATEWAY]
     167                  [--destination DESTINATION] [--installdir INSTALLDIR]
     168                  [--job_template JOB_TEMPLATE] [--controlfile CONTROLFILE]
     169
     170    Install flex_extract software locally or on ECMWF machines
     171
     172    optional arguments:
     173      -h, --help            show this help message and exit
     174      --target INSTALL_TARGET
     175                            Valid targets: local | ecgate | cca , the latter two
     176                            are at ECMWF (default: None)
     177      --makefile MAKEFILE   Name of Makefile to use for compiling the Fortran
     178                            program (default: None)
     179      --ecuid ECUID         The user id at ECMWF. (default: None)
     180      --ecgid ECGID         The group id at ECMWF. (default: None)
     181      --gateway GATEWAY     The name of the local gateway server. (default: None)
     182      --destination DESTINATION
     183                            The ecaccess association, e.g. myUser@genericSftp
     184                            (default: None)
     185      --installdir INSTALLDIR
     186                            Root directory where flex_extract will be installed
     187                            to. (default: None)
     188      --job_template JOB_TEMPLATE
     189                            The rudimentary template file to create a batch job
     190                            template for submission to ECMWF servers. (default:
     191                            job.template)
     192      --controlfile CONTROLFILE
     193                            The file with all CONTROL parameters. (default:
     194                            CONTROL_EA5)
     195
     196
     197
    23198
    24199

  • for_developers/Sphinx/source/Documentation/Overview/app_modes.rst

    r41c9dbc r0b00607  
    1414Arising from the two user groups described in :doc:`../../Ecmwf/access`, ``flex_extract`` has 4 different :underline:`user application modes`:
    1515
    16 .. _ref-remote-des:
     16.. _ref-remote-desc:
    1717
    1818  1. Remote (member)
    1919      In the **Remote mode** the user works directly on ECMWF Linux member state server, such as ``ecgate`` or ``cca``. The software will be installed in the ``$HOME`` directory. The user does not need to install any of the additional third-party libraries mentioned in :ref:`ref-requirements` as ECMWF provides everything with environment modules. The module selection will be done automatically in ``flex_extract``.
    2020     
    21 .. _gateway:
     21.. _ref-gateway-desc:
    2222     
    2323  2. Gateway (member)
    24       The **Gateway mode** can be used if a local member state gateway server is in place. Then the job scripts can be submitted to the ECMWF Linux member state server via the ECMWF web access tool ``ecaccess``. The installation script of ``flex_extract`` must be executed at the local gateway server such that the software will be installed in the ``$HOME`` directory at the ECMWF server and some extra setup is done in the local ``flex_extract`` directory at the local gateway server. For more information about establishing a gateway server please see section ???. For the **Gateway mode** the necessary environment has to be established which is described in :ref:`ref-local-prep`.
     24      The **Gateway mode** can be used if a local member state gateway server is in place. Then the job scripts can be submitted to the ECMWF Linux member state server via the ECMWF web access tool ``ecaccess``. The installation script of ``flex_extract`` must be executed at the local gateway server such that the software will be installed in the ``$HOME`` directory at the ECMWF server and some extra setup is done in the local ``flex_extract`` directory at the local gateway server. For more information about establishing a gateway server please see section ???. For the **Gateway mode** the necessary environment has to be established which is described in :ref:`ref-prep-gateway`.
    2525
    26 .. _local:
     26.. _ref-local-desc:
    2727     
    2828  3. Local member
    29       Scripts are installed and executed on a local machine, either in the current ``flex_extract`` directory or in a path given to the installation script. Under this scenario a software environment similar to that at ECMWF is required. Additionally, Web API's have to be installed to access ECMWF server. The complete installation process is described in :ref:`ref-local-prep`.
     29      Scripts are installed and executed on a local machine, either in the current ``flex_extract`` directory or in a path given to the installation script. Under this scenario a software environment similar to that at ECMWF is required. Additionally, Web API's have to be installed to access ECMWF server. The complete installation process is described in :ref:`ref-local-mode`.
    3030     
    3131  4. Local public
    32       Scripts are installed and executed on a local machine, either in the current ``flex_extract`` directory or in a path given to the installation script. Under this scenario a software environment similar to that at ECMWF is required. Additionally, Web API's have to be installed to access ECMWF server. The complete installation process is described in :ref:`ref-local-prep`. In this case a direct registration at ECMWF is necessary and the user has to accept a specific license agreement for each dataset he/she intends to retrieve.
     32      Scripts are installed and executed on a local machine, either in the current ``flex_extract`` directory or in a path given to the installation script. Under this scenario a software environment similar to that at ECMWF is required. Additionally, Web API's have to be installed to access ECMWF server. The complete installation process is described in :ref:`ref-local-mode`. In this case a direct registration at ECMWF is necessary and the user has to accept a specific license agreement for each dataset he/she intends to retrieve.
    3333     
    3434     

  • for_developers/Sphinx/source/Documentation/Overview/prog_flow.rst

    r41c9dbc r0b00607  
     1************
    12Program Flow
    2 ================
     3************
    34
    4  
    5     UNDER CONSTRUCTION
     5
     6
     7
     8General program flow
     9====================
     10
     11
     12The following flow diagram shows the general steps performed by ``flex_extract``.
     13   
     14.. _ref-fig-submit:
     15
     16.. figure:: ../../_files/submit.png   
     17   
     18    Overview of the call of python's ``submit.py`` script and raw sequence of working steps done in ``flex_extract``.
     19
     20   
     21The ``submit.py`` Python program is called by the Shell script ``run.sh`` or ``run_local.sh`` and accomplish the following steps:
     22
     23    1. Setup the control data:
     24        It gets all command-line and ``CONTROL`` file parameters as well as optionally the ECMWF user credentials. Depending the :doc:`app_modes`, it might also prepare a job script which is then send to the ECMWF queue.
     25    2. Retrieves data from MARS:
     26        It creates and sends MARS-requests either on the local machine or on ECMWF server, that receives the data and stores them in a specific format in GRIB files. If the parameter ``REQUEST`` was set ``1`` the data are not received but a file ``mars_requests.csv`` is created with a list of MARS requests and their settings. If it is set to ``2`` the file is created in addition to retrieving the data. The requests are created in an optimised way by splitting in time, jobs  and parameters.   
     27    3. Post-process data to create final ``FLEXPART`` input files:
     28        After all data is retrieved, the disaggregation of flux fields (`see here <../disagg.html>`_ ) is done as well as the calculation of vertical velocity (`see here <../vertco.html>`_) by the Fortran program ``COVERT2``. Eventually, the GRIB fields are merged together such that a single grib file per time step is available with all fields for ``FLEXPART``. Since model level fields are typically in *GRIB2* format whereas surface level fields are still in *GRIB1* format, they can be converted into GRIB2 if parameter ``FORMAT`` is set to *GRIB2*. Please note, however, that older versions of FLEXPART may have difficulties reading pure *GRIB2* files since some parameter IDs change in *GRIB2*. If the retrieval is executed remotely at ECMWF, the resulting files can be communicated to the local gateway server via the ``ECtrans`` utility if the parameter ``ECTRANS`` is set to ``1`` and the parameters ``GATEWAY``, ``DESTINATION`` have been set properly during installation. The status of the transfer can be checked with the command ``ecaccess-ectrans-list`` (on the local gateway server). If the script is executed locally the progress of the script can be followed with the usual Linux tools.
     29
     30
     31
     32Workflows of different application modes
     33========================================
     34
     35More details on how different the program flow is for the different :doc:`app_modes` is sketched in the following diagrams: 
     36
     37+-------------------------------------------------+------------------------------------------------+
     38| .. figure:: ../../_files/mode_remote.png        | .. figure:: ../../_files/mode_gateway.png      |
     39+-------------------------------------------------+------------------------------------------------+   
     40
     41+-------------------------------------------------+------------------------------------------------+
     42| .. figure:: ../../_files/mode_local_member.png  | .. figure:: ../../_files/mode_local_public.png |
     43+-------------------------------------------------+------------------------------------------------+   
     44
     45
     46Example application setting for a local member user
     47===================================================
     48
     49.. figure:: ../../_files/ex_runlocal_en.png 
     50
     51
     52
     53
     54.. toctree::
     55    :hidden:
     56    :maxdepth: 2

  • for_developers/Sphinx/source/Documentation/api.rst

    r41c9dbc r0b00607  
     1****************************
    12Auto Generated Documentation
    2 ============================
     3****************************
     4   
     5   
     6:doc:`Api/api_python`
     7 
     8:doc:`Api/api_fortran`
    39
    4 .. contents::
    5     :local:
     10 
     11       
     12.. toctree::
     13    :hidden:
     14    :maxdepth: 2
    615   
    7 Porgrams
    8 --------
    9 
    10 install
    11 *******
    12 
    13 .. automodule:: install
    14     :members:
    15    
    16 submit
    17 ******
    18 
    19 .. automodule:: submit
    20     :members:
    21    
    22 Classes
    23 -------
    24 
    25 ControlFile
    26 ***********
    27 
    28 .. automodule:: ControlFile
    29     :members:
    30    
    31 EcFlexpart
    32 **********
    33 
    34 .. automodule:: EcFlexpart
    35     :members:
    36    
    37 GribUtil
    38 *********
    39 
    40 .. automodule:: GribUtil
    41     :members:
    42 
    43 MarsRetrieval
    44 *************
    45 
    46 .. automodule:: MarsRetrieval
    47     :members:
    48 
    49 UioFiles
    50 ********
    51 
    52 .. automodule:: UioFiles
    53     :members:
    54 
    55 
    56 
    57 Modules
    58 -------
    59 
    60 get_mars_data
    61 *************
    62 
    63 .. automodule:: get_mars_data
    64     :members:
    65  
    66 prepare_flexpart
    67 ****************
    68 
    69 .. automodule:: prepare_flexpart
    70     :members:   
    71    
    72 tools
    73 *****
    74 
    75 .. automodule:: tools
    76     :members:
    77 
    78 disaggregation
    79 **************   
    80    
    81 .. automodule:: disaggregation
    82     :members:
     16    Api/api_python
     17    Api/api_fortran

  • for_developers/Sphinx/source/Documentation/disagg.rst

    r41c9dbc r0b00607  
    1 ===========================
     1***************************
    22Disaggregation of Flux Data
    3 ===========================
     3***************************
    44   
    5     UNDER CONSTRUCTION   
     5``FLEXPART`` interpolates meteorological input data linearly to the position of computational particles in time and space. This method requires point values in the discrete input fields. However, flux data (as listed in table :ref:`ref-table-fluxpar`) from the ECMWF represent cell averages or integrals and are accumulated over a specific time interval, depending on the dataset. Hence, to conserve the integral quantity with ``FLEXPART``'s linear interpolation a pre-processing scheme has to be applied.
     6
     7.. _ref-table-fluxpar:
     8
     9.. csv-table:: flux fields
     10    :header: "Short Name", "Name", "Units", "Interpolation Type"
     11    :align: center
     12    :widths: 5,15,5,10
     13   
     14    LSP,  "large-scale precipitation",          ":math:`m`",          "modified linear interpolation"
     15    CP,   "convective precipitation",           ":math:`m`",          "modified linear interpolation"
     16    SSHF, "surface sensible heat flux",         ":math:`J m^{-2}`",   "bicubic interpolation"
     17    EWSS, "eastward turbulent surface stress",  ":math:`N m^{-2} s`", "bicubic interpolation"
     18    NSSS, "northward turbulent surface stress", ":math:`N m^{-2} s`", "bicubic interpolation"
     19    SSR,  "surface net solar radiation",        ":math:`J m^{-2}`",   "bicubic interpolation"
     20   
     21
     22The first step is to *de-accumulate* the fields in time so that each value represents an integral in x, y, t space.
     23Afterwards, a *disaggregation* scheme is applied which means to break down the integral value into point values.
     24In order to be able to carry out the disaggregation procedure proposed by Paul James, additional flux data is retrieved automatically for one day at the beginning and one day at the end of the period specified. Thus, data for flux computation will be requested for the period START_DATE-1 to END_DATE+1. Note that these (additional) dates are used only for interpolation within ``flex_extract`` and are not communicated to the final ``FLEXPART`` input files.
     25
     26The flux disaggregation produces files named ``fluxYYYYMMDDHH``, where ``YYYYMMDDHH`` is the date format. Note, that the first two and last two flux files do not contain any data.
     27
     28.. note::
     29
     30    Note also that for operational retrievals (``BASETIME`` set to 00 or 12) forecast fluxes are only available until ``BASETIME``, so that no polynomial interpolation is possible in the last two time intervals. This is the reason why setting ``BASETIME`` is not recommended for on demand scripts.       
     31       
     32
     33Disaggregation for precipitation in older versions
     34--------------------------------------------------
     35
     36In ``flex_extract`` up to version 5 the disaggregation was done with a Fortran program (FLXACC2). In version 6 this part was converted to Python.
     37
     38
     39In the old versions (below 7.1) a relatively simple method processes the precipitation fields in a way that is consistent with the scheme applied in ``FLEXPART`` for all variables: linear interpolation between times where input fields are available.
     40At first the accumulated values are divided by the number of hours (i.e., 3 or 6).
     41The best option for disaggregation, which was realised, is conservation within the interval under consideration plus the two adjacent ones.
     42Unfortunately, this leads to undesired temporal smoothing of the precipitation time series – maxima are damped and minima are raised.
     43It is even possible to produce non-zero precipitation in dry intervals bordering a precipitation period as shown in Fig. 1.
     44This is obviously undesirable as it will affect wet scavenging, a very efficient removal process for many atmospheric trace species.
     45Wet deposition may be produced in grid cells where none should occur, or too little may be produced in others. This could lead to an unrealistic, checkerboard-like deposition fields.
     46Horizontally, the precipitation values are averages for a grid cell around the grid point to which they are ascribed, and ``FLEXPART`` uses bilinear interpolation to obtain precipitation rates at particle positions.
     47However, the supporting points in space are not shifted between precipitation and other variables as is the case for the temporal dimension.
     48
     49
     50.. _ref-fig-olddisagg:
     51
     52.. figure:: ../_files/old_disagg.png
     53    :figclass: align-center
     54
     55    Fig. 1: Example of disaggregation scheme as implemented in older versions for an isolated precipitation event lasting one time interval (thick blue line). The amount of original precipitation after de-accumulation is given by the blue-shaded area. The green circles represent the discrete grid points after disaggregation and linearly interpolate in between them as indicated by the green line and the green-shaded area. Note that supporting points for the interpolation are shifted by a half-time interval compared to the times when other meteorological fields are available (Hittmeir et al. 2018).
     56
     57
     58
     59Disaggregation is done for 4 adjacent timespans (:math:`a_0, a_1, a_2, a_3`) which generates a new, disaggregated value which is output at the central point of the 4 adjacent timespans.
     60
     61.. math::
     62
     63       p_{ac} &= 0.5 * a_1\\
     64            m &= a_0 + a_2 > 0.\\
     65    p_{ac}(m) &= a_1(m) * a_2(m) / (a_0(m) + a_2(m))\\
     66       p_{bd} &= 0.5 * a_2\\
     67            m &= a_1 + a_3 > 0.\\
     68    p_{bd}(m) &= a_1(m) * a_2(m) / (a_1(m) + a_3(m))\\
     69
     70
     71This new point :math:`p` is used for linear interpolation of the complete timeseries afterwards. If one of the 4 original timespans has a value below 0 it is set to 0 prior to the calculation.
     72   
     73.. math::
     74
     75    p = p_{ac} + p_{bd}
     76
     77
     78
     79
     80
     81
     82Disaggregation for precipitation in version 7.1
     83-----------------------------------------------
     84
     85Due to the problems with generating precipitation in originally dry (or lower) intervals and the temporal smoothing a new algorithm was developed. The approach is based on a one dimensional piecewise linear function with two additional supporting grid points within each grid cell, dividing the interval into three pieces. It fulfils the desired requirements by preserving the integral precipitation in each time interval, guaranteeing continuity at interval boundaries, and maintaining non-negativity. An additional monotonicity filter helps to gain monotonicity.
     86The more natural requirements of symmetry, reality, computational efficiency and easy implementation motivates the linear formulation.
     87These requirements on the reconstruction algorithm imply that time intervals with no precipitation remain unchanged, i.e. the reconstructed values vanish throughout this whole time interval, too.
     88In the simplest scenario of an isolated precipitation event, where in the time interval before and after the data values are zero, the reconstruction algorithm therefore has to vanish at the boundaries of the interval, too.
     89The additional conditions of continuity and conservation of the precipitation amount then require us to introduce sub-grid points if we want to keep a linear interpolation (Fig. 2).
     90The height is thereby determined by the condition of conservation of the integral of the function over the time interval.
     91 
     92   
     93.. _ref-fig-newdisagg:                       
     94
     95.. figure:: ../_files/new_disagg.png         
     96    :figclass: align-center
     97   
     98    Fig. 2: Precipitation rate linearly interpolated using a sub-grid with two additional points. Colours as in Fig. 1 (Hittmeir et al. 2018).       
     99   
     100
     101Figure 3 shows an overview of the new algorithm and its components.
     102   
     103.. _ref-fig-IA3:                       
     104
     105.. figure:: ../_files/IA3.png         
     106    :figclass: align-center               
     107   
     108    Fig. 3: Schematic overview of the basic notation in a precipitation interval with the original precipitation rate g (green) as a step function and the interpolated data :math:`f` (dark blue) as the piecewise linear function. The original time interval with fixed grid length :math:`\delta t` is split equidistantly in three subintervals denoted by :math:`I_i^{1,2,3}`, with the slopes in the subintervals as denoted by :math:`k_i^{1,2,3}` . The sub-grid function values :math:`f_i, f_i^{1,2}, f_{i+1}` are marked by red diamonds (Hittmeir et al. 2018).
     109   
     110
     111The following lists the equations of the new algorithm.
     112   
     113.. math::
     114
     115    f_i^{(1)}=&\frac32 g_i -\frac{1}{12}f_{i}-\frac{5}{12}f_{i+1}
     116   
     117    f_i^{(2)}=&\frac32 g_i -\frac{5}{12}f_{i}-\frac{1}{12}f_{i+1}
     118   
     119    f_{i+1}=&\min\{3 g_i,3 g_{i+1},\sqrt{g_ig_{i+1}}\}
     120   
     121.. math::
     122   
     123    \textbf{if}  \quad
     124    \mathrm{sgn}(k_{i}^{(2)})\cdot \mathrm{sgn}(k_{i  }^{(3)})&=-1 \quad \wedge \\
     125    \mathrm{sgn}(k_{i  }^{(3)})\cdot \mathrm{sgn}(k_{i+1}^{(1)})&=-1 \quad \wedge \\
     126    \mathrm{sgn}(k_{i+1}^{(1)})\cdot \mathrm{sgn}(k_{i+1}^{(2)})&=-1  \quad
     127    \textbf{then}
     128
     129.. math::
     130   
     131    f_{i+1}^\diamond=&\frac{18}{13}g_i-\frac{5}{13}f_i
     132   
     133    f_{i+1}^{\diamond\diamond}=&\frac{18}{13}g_{i+1}-\frac{5}{13}f_{i+2}
     134   
     135    f_{i+1} =& \min\left\{3 g_i,\, 3 g_{i+1},\, \sqrt{(f_{i+1}^\diamond\,f_{i+1}^{\diamond\diamond})_+}\right\}   
     136   
     137    f_i^{(1)}=& \frac32 g_i -\frac{1}{12}f_{i}-\frac{5}{12}f_{i+1}^{\textrm{mon}}
     138   
     139    f_i^{(2)}=& \frac32 g_i -\frac{5}{12}f_{i}-\frac{1}{12}f_{i+1}^{\textrm{mon}}
     140   
     141    \textbf{endif}
     142
     143
     144In the case of the new disaggregation method for precipitation, the two new sub grid points are added in the ``flux`` output files. They are identified by the forecast step parameter ``step`` which is 0 for the original time interval and 1 or 2 for the two new sub grid points respectively. The filenames do not change.   
     145
     146   
     147.. note::
     148
     149    The new method for disaggregation was published in the Geoscientific Model Development Journal in 2018:
     150   
     151    Hittmeir, S., Philipp, A., and Seibert, P.: A conservative reconstruction scheme for the interpolation of extensive quantities in the Lagrangian particle dispersion model FLEXPART, Geosci. Model Dev., 11, 2503-2523, https://doi.org/10.5194/gmd-11-2503-2018, 2018.
     152
     153     
     154   
     155
     156 
     157
     158
     159Disaggregation for the rest of the flux fields
     160----------------------------------------------
     161     
     162The accumulated values for the other variables are first divided by the number of hours and
     163then interpolated to the exact times X using a bicubic interpolation which conserves the integrals of the fluxes within each timespan.
     164Disaggregation is done for 4 adjacent timespans (:math:`p_a, p_b, p_c, p_d`) which generates a new, disaggregated value which is output at the central point of the 4 adjacent timespans.
     165
     166.. math::
     167   
     168    p_a &= (a_3 - a_0 + 3. * (a_1 - a_2)) / 6. \\   
     169    p_b &= (a_2 + a_0) / 2. - a_1 - 9. * p_a / 2. \\
     170    p_c &= a_1 - a_0 - 7. * p_a / 2. - 2. * p_b \\   
     171    p_d &= a_0 - p_a / 4. - p_b / 3. - p_c / 2.
     172
     173This new point :math:`p` is used for linear interpolation of the complete timeseries afterwards.
     174
     175.. math::
     176   
     177    p = 8. * p_a + 4. * p_b + 2. * p_c + p_d
     178
     179
     180
     181       
    6182       
    7183.. toctree::

  • for_developers/Sphinx/source/Documentation/input.rst

    r41c9dbc r0b00607  
    1 ====================
     1********************
    22Control & Input Data
    3 ====================
     3********************
     4
     5Input Data
     6    - :doc:`Input/control`
     7          ``Flex_extract`` needs a number of controlling parameters to decide on the behaviour and the actual dataset to be retrieved. They are initialized by ``flex_extract`` with their default values and can be overwritten with definitions set in the so called :doc:`Input/control`.
     8
     9          To be able to successfully retrieve data from the ECMWF Mars archive it is necessary to understand these parameters and set them to proper and consistent values. They are described in :doc:`Input/control_params` section.
     10
     11          We also have some :doc:`Input/examples` and description of :doc:`Input/changes` changes to previous versions and downward compatibilities.
    412       
    5     UNDER CONSTRUCTION
     13    - :doc:`Input/ecmwf_env`
     14         For ``flex_extract`` it is necessary to be able to reach ECMWF servers in the **remote mode** and the **gateway mode**. Therefore a :doc:`Input/ecmwf_env` is created during the installation process.
     15
     16    - :doc:`Input/templates`
     17         A number of files which are created by ``flex_extract`` are taken from templates. This makes it easy to adapt for example the jobscripts regarding its settings for the batch jobs.         
     18
     19
     20
     21
     22
     23.. _setup : Input/setup.html
     24.. _run : Input/run.html
     25.. _install : Input/setup.html#ref-install-script
     26.. _submit : Input/submit.html#ref-submit-script
     27
     28.. _ref-controlling:
     29
     30Controlling
     31    The main tasks and behaviour of ``flex_extract`` are controlled by its Python scripts. There are two top-level scripts, one for installation called install_ and one for execution called submit_.
     32    They can interpret a number of command line arguments which can be seen by typing ``--help`` after the script call. Go to the root directory of ``flex_extract`` to type:
     33
     34    .. code-block:: bash
     35
     36       cd flex_extract_vX.X
     37       python3 source/python/install.py --help
     38       python3 source/python/submit.py --help
     39   
     40    In this new version we provide also the wrapping Shell scripts setup_ and run_, which sets the command line parameters, do some checks and execute the corresponing Python scripts ``install.py`` and ``submit.py`` respectivley.
     41     
     42    It might be faster and easier for beginners. See :doc:`../quick_start` for information on how to use them.
     43
     44    Additionally, ``flex_extract`` creates the Korn Shell scripts :doc:`Input/compilejob` and :doc:`Input/jobscript` which will be send to the ECMWF serves in the **remote mode** and the **gateway mode** for starting batch jobs.
     45
     46    The Fortran program will be compiled during the installation process by the :doc:`Input/fortran_makefile`.
     47   
     48    To sum up, the following scripts controls ``flex_extract``:
     49
     50    Installation
     51       - :doc:`Input/setup`
     52       - :doc:`Input/compilejob`
     53       - :doc:`Input/fortran_makefile`   
     54
     55    Execution
     56      - :doc:`Input/run`
     57      - :doc:`Input/jobscript`
     58             
     59             
     60   
     61
     62
     63                 
     64
     65         
     66       
    667       
    768.. toctree::
    … …  
    970    :maxdepth: 2
    1071   
     72    Input/setup
     73    Input/compilejob
     74    Input/fortran_makefile   
     75    Input/run
     76    Input/jobscript
    1177    Input/control
    12     Input/control_params
     78    Input/control_params 
     79    Input/examples
     80    Input/changes
     81    Input/ecmwf_env
     82    Input/templates

  • for_developers/Sphinx/source/Documentation/output.rst

    r41c9dbc r0b00607  
    1 ===========
     1***********
    22Output Data
    3 ===========
    4    
    5     UNDER CONSTRUCTION
     3***********
     4
     5The output data of ``flex_extract`` are separated mainly into temporary files and the final ``FLEXPART`` input files:
     6
     7+-----------------------------------------------+----------------------------------------------+   
     8|   ``FLEXPART`` input files                    |  Temporary files (saved in debug mode)       |
     9+-----------------------------------------------+----------------------------------------------+
     10| - Standard output filenames                   | - MARS request file (opt)                    |
     11| - Output for pure forecast                    | - flux files                                 |
     12| - Output for ensemble members                 | - VERTICAL.EC                                |
     13| - Output for new precip. disaggregation       | - index file                                 |
     14|                                               | - fort files                                 |
     15|                                               | - MARS grib files                            |
     16+-----------------------------------------------+----------------------------------------------+
     17
     18
     19
     20``FLEXPART`` input files
     21========================
     22
     23The final output files of ``flex_extract`` are also the meteorological ``FLEXPART`` input files.
     24The naming of these files depend on the kind of data extracted by ``flex_extract``.
     25
     26Standard output files
     27---------------------
     28 
     29In general, there is a file for each time step with the filename format:
     30
     31.. code-block:: bash
     32
     33    <prefix>YYMMDDHH
     34   
     35The ``prefix`` is by default defined as ``EN`` and can be re-defined in the ``CONTROL`` file.
     36Each file contains all meteorological fields needed by ``FLEXPART`` for all selected model levels for a specific time step.
     37
     38Here is an example output which lists the meteorological fields in a single file called ``CE00010800`` where we extracted only the lowest model level for demonstration reasons:
     39
     40.. code-block:: bash
     41
     42        $ grib_ls CE00010800
     43       
     44        edition      centre       date         dataType     gridType     stepRange    typeOfLevel  level        shortName    packingType
     45        2            ecmf         20000108     an           regular_ll   0            hybrid       91           u            grid_simple
     46        2            ecmf         20000108     an           regular_ll   0            hybrid       91           v            grid_simple
     47        2            ecmf         20000108     an           regular_ll   0            hybrid       91           etadot       grid_simple
     48        2            ecmf         20000108     an           regular_ll   0            hybrid       91           t            grid_simple
     49        2            ecmf         20000108     an           regular_ll   0            surface      1            sp           grid_simple
     50        2            ecmf         20000108     an           regular_ll   0            hybrid       91           q            grid_simple
     51        2            ecmf         20000108     an           regular_ll   0            hybrid       91           qc           grid_simple
     52        1            ecmf         20000108     fc           regular_ll   0            surface      0            sshf         grid_simple
     53        1            ecmf         20000108     fc           regular_ll   0            surface      0            ewss         grid_simple
     54        1            ecmf         20000108     fc           regular_ll   0            surface      0            nsss         grid_simple
     55        1            ecmf         20000108     fc           regular_ll   0            surface      0            ssr          grid_simple
     56        1            ecmf         20000108     fc           regular_ll   0            surface      0            lsp          grid_simple
     57        1            ecmf         20000108     fc           regular_ll   0            surface      0            cp           grid_simple
     58        1            ecmf         20000108     an           regular_ll   0            surface      0            sd           grid_simple
     59        1            ecmf         20000108     an           regular_ll   0            surface      0            msl          grid_simple
     60        1            ecmf         20000108     an           regular_ll   0            surface      0            tcc          grid_simple
     61        1            ecmf         20000108     an           regular_ll   0            surface      0            10u          grid_simple
     62        1            ecmf         20000108     an           regular_ll   0            surface      0            10v          grid_simple
     63        1            ecmf         20000108     an           regular_ll   0            surface      0            2t           grid_simple
     64        1            ecmf         20000108     an           regular_ll   0            surface      0            2d           grid_simple
     65        1            ecmf         20000108     an           regular_ll   0            surface      0            z            grid_simple
     66        1            ecmf         20000108     an           regular_ll   0            surface      0            lsm          grid_simple
     67        1            ecmf         20000108     an           regular_ll   0            surface      0            cvl          grid_simple
     68        1            ecmf         20000108     an           regular_ll   0            surface      0            cvh          grid_simple
     69        1            ecmf         20000108     an           regular_ll   0            surface      0            lcc          grid_simple
     70        1            ecmf         20000108     an           regular_ll   0            surface      0            mcc          grid_simple
     71        1            ecmf         20000108     an           regular_ll   0            surface      0            hcc          grid_simple
     72        1            ecmf         20000108     an           regular_ll   0            surface      0            skt          grid_simple
     73        1            ecmf         20000108     an           regular_ll   0            depthBelowLandLayer  0            stl1         grid_simple
     74        1            ecmf         20000108     an           regular_ll   0            depthBelowLandLayer  0            swvl1        grid_simple
     75        1            ecmf         20000108     an           regular_ll   0            surface      0            sr           grid_simple
     76        1            ecmf         20000108     an           regular_ll   0            surface      0            sdor         grid_simple
     77        1            ecmf         20000108     an           regular_ll   0            surface      0            cvl          grid_simple
     78        1            ecmf         20000108     an           regular_ll   0            surface      0            cvh          grid_simple
     79        1            ecmf         20000108     an           regular_ll   0            surface      0            fsr          grid_simple
     80        35 of 35 messages in CE00010800
     81
     82
     83Output files for pure forecast
     84------------------------------
     85
     86``Flex_extract`` can retrieve forecasts which can be longer than 23 hours. To avoid collisions of time steps for forecasts of more than one day a new scheme for filenames in pure forecast mode is introduced:
     87
     88.. code-block:: bash
     89
     90    <prefix>YYMMDD.HH.<FORECAST_STEP>
     91
     92The ``<prefix>`` is, as in the standard output, by default ``EN`` and can be re-defined in the ``CONTROL`` file. ``YYMMDD`` is the date format and ``HH`` the forecast time which is the starting time for the forecasts. The ``FORECAST_STEP`` is a 3 digit number which represents the forecast step in hours.
     93   
     94
     95Output files for ensemble predictions
     96-------------------------------------
     97
     98Ensembles can be retrieved and are addressed by the grib message parameter ``number``. The ensembles are saved per file and standard filenames are supplemented by the letter ``N`` and the ensemble member number in a 3 digit format.
     99
     100.. code-block:: bash
     101
     102    <prefix>YYMMDDHH.N<ENSEMBLE_MEMBER>
     103
     104
     105Additional fields with new precipitation disaggregation
     106-------------------------------------------------------
     107
     108The new disaggregation method for precipitation fields produces two additional precipitation fields for each time step and precipitation type. They serve as sub-grid points in the original time interval. For details of the method see :doc:`disagg` ??????????????????.
     109The two additional fields are marked with the ``step`` parameter in the Grib messages and are set to "1" and "2" for sub-grid point 1 and 2 respectively.
     110The output filenames do not change in this case. 
     111Below is an example list of precipitation fields in an output file generated with the new disaggregation method:
     112
     113.. code-block:: bash
     114
     115    $ grib_ls
     116
     117    edition      centre       date         dataType     gridType     stepRange    typeOfLevel  level        shortName    packingType
     118    1            ecmf         20000108     fc           regular_ll   0            surface      0            lsp          grid_simple
     119    1            ecmf         20000108     fc           regular_ll   1            surface      0            lsp          grid_simple
     120    1            ecmf         20000108     fc           regular_ll   2            surface      0            lsp          grid_simple
     121    1            ecmf         20000108     fc           regular_ll   0            surface      0            cp           grid_simple
     122    1            ecmf         20000108     fc           regular_ll   1            surface      0            cp           grid_simple
     123    1            ecmf         20000108     fc           regular_ll   2            surface      0            cp           grid_simple
     124
     125
     126
     127
     128Temporary files
     129===============
     130
     131``Flex_extract`` works with a number of temporary data files which are usually deleted after a successful data extraction. They are only stored if the ``DEBUG`` mode is switched on (see :doc:`Input/control_params`.
     132
     133MARS grib files
     134---------------
     135
     136``Flex_extract`` retrieves all meteorological fields from MARS and stores them in files ending with ``.grb``.
     137Since the request times and data transfer of MARS access are limited and ECMWF asks for efficiency in requesting data from MARS, ``flex_extract`` splits the overall data request in several smaller requests. Each request is stored in an extra ``.grb`` file and the file names are put together by several pieces of information:
     138
     139    .. code-block:: bash
     140   
     141       <field_type><grid_type><temporal_property><level_type>.<date>.<ppid>.<pid>.grb
     142
     143Description:
     144       
     145Field type:
     146    ``AN`` - Analysis, ``FC`` - Forecast, ``4V`` - 4d variational analysis, ``CV`` - Validation forecast, ``CF`` - Control forecast, ``PF`` - Perturbed forecast
     147Grid type:
     148   ``SH`` - Spherical Harmonics, ``GG`` - Gaussian Grid, ``OG`` - Output Grid (typically lat/lon), ``_OROLSM`` - Orography parameter
     149Temporal property:
     150    ``__`` - instantaneous fields, ``_acc`` - accumulated fields
     151Level type:
     152    ``ML`` - Model Level, ``SL`` - Surface Level
     153ppid:
     154    The process number of the parent process of submitted script.
     155pid:
     156    The process number of the submitted script.
     157
     158The process ids should avoid mixing of fields if several ``flex_extract`` jobs are performed in parallel (which is, however, not recommended). The date format is YYYYMMDDHH.
     159
     160Example ``.grb`` files for a day of CERA-20C data:
     161
     162    .. code-block:: bash
     163
     164        ANOG__ML.20000908.71851.71852.grb 
     165        FCOG_acc_SL.20000907.71851.71852.grb
     166        ANOG__SL.20000908.71851.71852.grb 
     167        OG_OROLSM__SL.20000908.71851.71852.grb
     168        ANSH__SL.20000908.71851.71852.grb
     169
     170
     171MARS request file
     172-----------------
     173
     174This file is a ``csv`` file called ``mars_requests.csv`` with a list of the actual settings of MARS request parameters (one request per line) in a flex_extract job. It is used for documenting the data which were retrieved and for testing reasons.
     175
     176Each request consist of the following parameters, whose meaning mainly can be taken from :doc:`Input/control_params` or :doc:`Input/run`:
     177request_number, accuracy, area, dataset, date, expver, gaussian, grid, levelist, levtype, marsclass, number, param, repres, resol, step, stream, target, time, type
     178 
     179Example output of a one day retrieval of CERA-20c data:
     180
     181.. code-block:: bash
     182
     183    request_number, accuracy, area, dataset, date, expver, gaussian, grid, levelist, levtype, marsclass, number, param, repres, resol, step, stream, target, time, type
     184    1, 24, 40.0/-5.0/30.0/5.0, None, 20000107/to/20000109, 1, , 1.0/1.0, 1, SFC, EP, 000, 142.128/143.128/146.128/180.128/181.128/176.128, , 159, 3/to/24/by/3, ENDA, /mnt/nas/Anne/Interpolation/flexextract/flex_extract_v7.1/run/./workspace/CERA_testgrid_local_cds/FCOG_acc_SL.20000107.23903.23904.grb, 18, FC
     185    1, 24, 40.0/-5.0/30.0/5.0, None, 20000108/to/20000108, 1, , 1.0/1.0, 85/to/91, ML, EP, 000, 130.128/133.128/131.128/132.128/077.128/246.128/247.128, , 159, 00, ENDA, /mnt/nas/Anne/Interpolation/flexextract/flex_extract_v7.1/run/./workspace/CERA_testgrid_local_cds/ANOG__ML.20000108.23903.23904.grb, 00/03/06/09/12/15/18/21, AN
     186    2, 24, 40.0/-5.0/30.0/5.0, None, 20000108/to/20000108, 1, , OFF, 1, ML, EP, 000, 152.128, , 159, 00, ENDA, /mnt/nas/Anne/Interpolation/flexextract/flex_extract_v7.1/run/./workspace/CERA_testgrid_local_cds/ANSH__SL.20000108.23903.23904.grb, 00/03/06/09/12/15/18/21, AN
     187    3, 24, 40.0/-5.0/30.0/5.0, None, 20000108, 1, , 1.0/1.0, 1, SFC, EP, 000, 160.128/027.128/028.128/244.128, , 159, 000, ENDA, /mnt/nas/Anne/Interpolation/flexextract/flex_extract_v7.1/run/./workspace/CERA_testgrid_local_cds/OG_OROLSM__SL.20000108.23903.23904.grb, 00, AN
     188    4, 24, 40.0/-5.0/30.0/5.0, None, 20000108/to/20000108, 1, , 1.0/1.0, 1, SFC, EP, 000, 141.128/151.128/164.128/165.128/166.128/167.128/168.128/129.128/172.128/027.128/028.128/186.128/187.128/188.128/235.128/139.128/039.128/173.128, , 159, 00, ENDA, /mnt/nas/Anne/Interpolation/flexextract/flex_extract_v7.1/run/./workspace/CERA_testgrid_local_cds/ANOG__SL.20000108.23903.23904.grb, 00/03/06/09/12/15/18/21, AN
     189
     190
     191VERTICAL.EC
     192-----------
     193
     194The vertical discretization of model levels. This file contains the ``A`` and ``B`` parameters to calculate the model level height in meters.
     195
     196
     197Index file
     198----------
     199
     200This file is usually called ``date_time_stepRange.idx``. It contains indices pointing to specific grib messages from one or more grib files. The messages are selected with a composition of grib message keywords.
     201
     202
     203flux files
     204----------
     205
     206The flux files contain the de-accumulated and dis-aggregated flux fields of large scale and convective precipitation, eastward turbulent surface stress, northward turbulent surface stress, surface sensible heat flux and the surface net solar radiation.
     207
     208.. code-block:: bash
     209
     210    flux<date>[.N<xxx>][.<xxx>]
     211
     212The date format is YYYYMMDDHH. The optional block ``[.N<xxx>]`` marks the ensemble forecast number, where ``<xxx>`` is the ensemble member number. The optional block ``[.<xxx>]`` marks a pure forecast with ``<xxx>`` being the forecast step.
     213
     214.. note::
     215
     216    In the case of the new dis-aggregation method for precipitation, two new sub-intervals are added in between each time interval. They are identified by the forecast step parameter which is ``0`` for the original time interval and ``1`` or ``2`` for the two new intervals respectively.
     217
     218   
     219fort files
     220----------
     221
     222There are a number of input files for the ``CONVERT2`` Fortran program named
     223
     224.. code-block:: bash
     225
     226    fort.xx
     227   
     228where ``xx`` is the number which defines the meteorological fields stored in these files.
     229They are generated by the Python part of ``flex_extract`` by just splitting the meteorological fields for a unique time stamp from the ``*.grb`` files into the ``fort`` files.
     230The following table defines the numbers with their corresponding content.   
     231
     232.. csv-table:: Content of fort - files
     233    :header: "Number", "Content"
     234    :widths: 5, 20
     235 
     236    "10", "U and V wind components"
     237    "11", "temperature"
     238    "12", "logarithm of surface pressure"
     239    "13", "divergence (optional)"
     240    "16", "surface fields"
     241    "17", "specific humidity"
     242    "18", "surface specific humidity (reduced gaussian)"
     243    "19", "vertical velocity (pressure) (optional)"
     244    "21", "eta-coordinate vertical velocity (optional)"
     245    "22", "total cloud water content (optional)"
     246
     247Some of the fields are solely retrieved with specific settings, e.g. the eta-coordinate vertical velocity is not available in ERA-Interim datasets and the total cloud water content is an optional field for ``FLEXPART v10`` and newer. Please see section ????????? for more information.
     248
     249The ``CONVERT2`` program saves its results in file ``fort.15`` which typically contains:
     250
     251.. csv-table:: Output file of the Fortran program ``CONVERT2``
     252    :header: "Number", "Content"
     253    :widths: 5, 20
     254 
     255    "15", "U and V wind components, eta-coordinate vertical velocity, temperature, surface pressure, specific humidity "
     256   
     257More details about the content of ``CONVERT2`` can be found in :doc:`vertco`.   
     258   
     259.. note::
     260 
     261    The ``fort.4`` file is the namelist file to drive the Fortran program ``CONVERT2``. It is therefore also an input file and is described in ???????????????
     262   
     263    Example of a namelist:
     264   
     265    .. code-block:: bash
     266   
     267        &NAMGEN
     268          maxl = 11,
     269          maxb = 11,
     270          mlevel = 91,
     271          mlevelist = "85/to/91",
     272          mnauf = 159,
     273          metapar = 77,
     274          rlo0 = -5.0,
     275          rlo1 = 5.0,
     276          rla0 = 30.0,
     277          rla1 = 40.0,
     278          momega = 0,
     279          momegadiff = 0,
     280          mgauss = 0,
     281          msmooth = 0,
     282          meta = 1,
     283          metadiff = 0,
     284          mdpdeta = 1
     285        /
     286       
    6287       
    7288.. toctree::

  • for_developers/Sphinx/source/Documentation/overview.rst

    r41c9dbc r0b00607  
    22Overview
    33========
    4        
    5     UNDER CONSTRUCTION   
    6    
    7    
    8    
    9 The system retrieves the data from the European Centre for
    10 Medium-Range Weather Forecasts (ECMWF) situated in Reading, United Kingdom.
     4
     5``Flex_extract`` is an open-source software to retrieve meteorological fields from the European Centre for Medium-Range Weather Forecasts (ECMWF) Mars archive to serve as input files for the ``FLEXTRA``/``FLEXPART`` Atmospheric Transport Modelling system.
     6``Flex_extract`` was created explicitly for ``FLEXPART`` users who wants to use meteorological data from ECMWF to drive the ``FLEXPART`` model.
     7The software retrieves the minimal number of parameters ``FLEXPART`` needs to work and provides the data in the explicity format ``FLEXPART`` understands.
     8
     9``Flex_extract`` consists of 2 main parts:
     10    1. a Python part, where the reading of parameter settings, retrieving data from MARS and preparing the data for ``FLEXPART`` is done and
     11    2. a Fortran part, where the calculation of the vertical velocity is done and if necessary the conversion from spectral to regular latitude/longitude grids.
     12
     13Additionally, it has some Korn shell scripts which are used to set the environment and batch job features on ECMWF servers for the *gateway* and *remote* mode. See :doc:`Overview/app_modes` for information of application modes.   
     14
     15A number of Shell scripts are wrapped around the software package for easy installation and fast job submission.
     16
     17The software depends on a number of third-party libraries which can be found in :ref:`ref-requirements`.
     18
     19Some details on the tasks and program worksteps are described in :doc:`Overview/prog_flow`.
     20
     21
     22..  - directory structure (new diagramm!)
     23           
     24    - Software components - complete component structure (table or diagram)
     25           
     26       - Python package
     27           
     28           - Package diagram
     29           - Files and modules as table with information about unit tests
     30           - Api
     31             
     32       - Fortran program - CONVERT2
     33           
     34           - Package diagram
     35           - Api
     36             
     37
     38
     39
    1140
    1241   
    … …  
    1746    Overview/app_modes
    1847    Overview/prog_flow
    19     Overview/convert

  • for_developers/Sphinx/source/Documentation/vertco.rst

    r41c9dbc r0b00607  
    1 ===================
     1*******************
    22Vertical Coordinate
    3 ===================
     3*******************
     4       
     5Calculation of vertical velocity and preparation of Output-files
     6================================================================
     7
     8``flex_extract`` has two ways to calculate the vertical velocity for ``FLEXTRA``/``FLEXPART``:
     9    (i) from the horizontal wind field,
     10    (ii) from the MARS parameter 77, which is available for operational forecasts and analyses since September 2008 and for reanalysis datasets **ERA5** and **CERA-20C**.
     11
     12Especially for high resolution data, use of the ``MARS`` parameter 77 is recommended,
     13since the computational cost (measured in ECMWF HPC units) is reduced by 90-95% at
     14T799. The extraction time, which depends heavily also on the performance of ``MARS``, is
     15generally reduced by 50% as well. The ``MARS`` parameter 77 is then multiplied by ``dp/deta`` to
     16give a vertical velocity in Pa/s as needed by ``FLEXPART``.
     17
     18Calculation from the horizontal wind field is still required for historical case studies using
     19**ERA-40**, **ERA-Interim** or operational data prior to September 2008.   
    420   
    5     UNDER CONSTRUCTION
     21   
     22Calculation of vertical velocity from horizontal wind using the continuity equation
     23===================================================================================
     24
     25The vertical velocity is computed by the FORTRAN90 program ``CONVERT2`` in the ECMWF
     26vertical coordinate system by applying the equation of continuity and thereby ensuring mass consistent 3D wind fields. A detailed description of ``CONVERT2`` can be found in the
     27documents v20_update_protocol.pdf, V30_update_protocol.pdf and
     28V40_update_protocol.pdf. The computational demand and accuracy of ``CONVERT2`` is highly
     29dependent on the specification of parameters ``GAUSS``, ``RESOL`` and ``SMOOTH``. The
     30following guidance can be given for choosing the right parameters:
     31
     32    * For very fine output grids (0.25 degree or finer) the full resolution T799 or even T1279 of the operational model is required (``RESOL=799``, ``SMOOTH=0``). The highest available resolution (and the calculation of vertical velocity on the Gaussian grid (``GAUSS=1``) is, however, rather demanding and feasible only for resolutions up to T799. Higher resolutions are achievable on the HPC. If data retrieval at T1279  needs to be performed on *ecgate*, the computation of the vertical velocity is feasible only on the lat/lon grid (``GAUSS=0``), which also yields very good results. Please read document v20_update_protocol.pdf-v60_update_protocol.pdf to see if the errors incurred are acceptable for the planned application.
     33    * For lower resolution (often global) output grids, calculation of vertical velocities with lower than operational spectral resolution is recommended. For global grids the following settings appear optimal:
     34        - For 1.0 degree grids: ``GAUSS=1``, ``RESOL=255``, ``SMOOTH=179``
     35        - For 0.5 degree grids: ``GAUSS=1``, ``RESOL=399``, ``SMOOTH=359``
     36        - Calculation on the lat/lon grid is not recommended for less than the operational (T1279) resolution.   
     37        - If ``GAUSS`` is set to 1, only the following choices are possible for ``RESOL`` on *ecgate*: 159,255,319,399,511,799, (on the HPC also 1279, 2047 in future models). This choice is restricted because a reduced Gaussian grid is defined in then ECMWF EMOSLIB only for these spectral resolutions. For ``GAUSS=0``, ``RESOL`` can be any value below the operational resolution.
     38        - For ``SMOOTH`` any resolution lower than ``RESOL`` is possible. If no smoothing is desired, ``SMOOTH=0`` should be chosen. ``SMOOTH`` has no effect if vertical velocity is calculated on lat\/lon grid (``GAUSS=0``).
     39    * The on demand scripts send an error message for settings where ``SMOOTH`` (if set) and ``RESOL`` are larger than 360./``GRID``/2, since in this case, the output grid cannot resolve the highest wave numbers. The scripts continue operations, however.
     40    * Regional grids are not cyclic in zonal directions, but global grids are. The software assumes a cyclic grid if ``RIGHT``-``LEFT`` is equal to ``GRID`` or is equal to ``GRID``-360.
     41    * Finally, model and flux data as well as the vertical velocity computed are written to files ``<prefix>yymmddhh`` for application in ATM modelling. If the parameters ``OMEGA`` or ``OMEGADIFF`` are set, also files ``OMEGAyymmddhh`` are created, containing the pressure vertical velocity (omega) and the difference between omega from ``MARS`` and the surface pressure tendency. ``OMEGADIFF`` should be zero except for debugging, since it triggers expensive calculations on the Gaussian grid.
     42   
     43   
     44Calculation of vertical velocity from pre-calculated MARS parameter 77
     45======================================================================
     46
     47Since November 2008, the parameter 77 (deta/dt) is stored in ``MARS`` on full model levels. ``FLEXTRA``/``FLEXPART`` in its current version requires ``deta/dt`` on model half levels, multiplied by ``dp/deta``. In ``flex_extract``, the program ``CONVERT2`` assumes that this parameter is available if the ``CONTROL`` parameter ``ETA`` is set to 1.
     48
     49It is recommended to use the pre-calculated parameter 77 by setting ``ETA`` to 1 whenever possible.
     50
     51Setting parameter ``ETA`` to 1 normally disables calculation of vertical velocity from the horizontal wind field, which saves a lot of computational time.
     52
     53.. note::
     54   However, the calculation on the Gaussian grid are avoided only if both ``GAUSS`` and ``ETADIFF`` are set to 0. Please set ``ETADIFF`` to 1 only if you are really need it for debugging since this is a very expensive option. In this case ``ETAyymmddhh`` files are produced that contain the vertical velocity from horizontal winds and the difference to the pre-calculated vertical velocity.
     55
     56The parameters ``RESOL``, ``GRID``, ``UPPER``, ``LOWER``, ``LEFT``, ``RIGHT`` still apply. As for calculations on the Gaussian grid, the spectral resolution parameter ``RESOL`` should be compatible with the grid resolution (see previous subsection).
     57   
    658       
    759.. toctree::

  • for_developers/Sphinx/source/Ecmwf/access.rst

    r41c9dbc r0b00607  
    55.. _public datasets: https://confluence.ecmwf.int/display/WEBAPI/Available+ECMWF+Public+Datasets
    66.. _Computing Representative: https://www.ecmwf.int/en/about/contact-us/computing-representatives
     7.. _Climate Data Store: https://cds.climate.copernicus.eu
     8.. _CDS API: https://cds.climate.copernicus.eu/api-how-to
    79
    810Access to the ECMWF Mars archive is divided into two groups: **member state** users and **public** users.
    911
    1012**Member state user**:
    11     This access mode allows the user to work directly on the ECMWF Linux Member State Servers or via a Web Access Toolkit ``ecaccess`` through a local Member State Gateway Server. This enables the user to have direct and full access to the Mars archive. There might be some limitations in user rights such as the declined access to the latest forecasts. This has to be discussed with the `Computing Representative`_. This user group is also able to work from their local facilities without a gateway server in the same way a **public** user would. The only difference is the method of Web API which is automatically selected by ``flex_extract``.
     13    This access mode allows the user to work directly on the ECMWF Linux Member State Servers or via a Web Access Toolkit ``ecaccess`` through a local Member State Gateway Server. This enables the user to have direct and full access to the Mars archive. There might be some limitations in user rights such as the declined access to the latest forecasts. This has to be discussed with the `Computing Representative`_. This user group is also able to work from their local facilities without a gateway server in the same way a **public** user would. The only difference is the connection with the Web API. However, this is automatically selected by ``flex_extract``.
    1214   
    1315
    1416**Public user**:
    15     This access mode allows every user to access the ECMWF `public datasets`_ from their local facilities. ``Flex_extract`` is able (tested for the use with ``FLEXPART``) to extract the re-analysis datasets such as ERA5, ERA-Interim and CERA-20C. The main difference to the **member state user** is the method of access with the Web API and the availability of data. For example, in ERA-Interim there is only a 6-hourly temporal resolution instead of 3 hours. The access method is selected by providing the command line argument "public=1" and providing the Mars keyword "dataset" in the ``CONTROL`` file. Also, the user has to explicitly accept the license of the dataset to be retrieved. This can be done as described in the installation process at section :ref:`ref-licence`.   
     17    This access mode allows every user to access the ECMWF `public datasets`_ from their local facilities. ``Flex_extract`` is able (tested for the use with ``FLEXPART``) to extract the re-analysis datasets such as ERA-Interim and CERA-20C. The main difference to the **member state user** is the method of access with the Web API and the availability of data. For example, in ERA-Interim there is only a 6-hourly temporal resolution instead of 3 hours. The access method is selected by providing the command line argument "public=1" and providing the MARS keyword "dataset" in the ``CONTROL`` file. Also, the user has to explicitly accept the license of the dataset to be retrieved. This can be done as described in the installation process at section :ref:`ref-licence`.   
     18     
     19.. note::
     20   
     21   The availability of the public dataset *ERA5* with the ECMWF Web API was cancelled in March 2019. The oportunity of local retrieval of this dataset was moved to the `Climate Data Store`_ which uses another Web API named `CDS API`_. This Data Store stores the data on explicit webservers for faster and easier access. Unfortunately, for *ERA5* there are only surface level and pressure level data available for *public users*. In the case of a *member user* it is possible to bypass the request to the MARS archive from ECMWF to retrieve the data. ``Flex_extract`` is already modified to use this API so *member user* can already retrieve *ERA5* data while *public users* have to wait until model level are available.
    1622       
    1723For information on how to register see :ref:`ref-registration`.

  • for_developers/Sphinx/source/Ecmwf/ec-links.rst

    r41c9dbc r0b00607  
    2121   
    2222    `MARS Actions <https://confluence.ecmwf.int/display/UDOC/MARS+actions>`_
     23   
     24    `Parameter Database <https://apps.ecmwf.int/codes/grib/param-db>`_
    2325 
    2426Registration
    … …  
    4648Datasets
    4749    Overview
     50        `Complete list of datasets <https://www.ecmwf.int/en/forecasts/datasets>`_
     51               
    4852        `What is climate reanalysis <https://www.ecmwf.int/en/research/climate-reanalysis>`_
     53       
     54        `Forecast user guide <https://confluence.ecmwf.int/display/FUG/1+Introduction>`_
     55
     56    Real-time (Operational)       
     57        `List of real_time datasets <https://www.ecmwf.int/en/forecasts/datasets/catalogue-ecmwf-real-time-products>`_
     58
     59        `Atmospheric model - HRES (our typical operational dataset) <https://www.ecmwf.int/en/forecasts/datasets/set-i>`_
     60       
     61        `Atmospheric model - ENS (15-day ensemble forecast) <https://www.ecmwf.int/en/forecasts/datasets/set-iii>`_
    4962
    5063    ERA-Interim
    … …  
    5871        `What is CERA-20C <https://software.ecmwf.int/wiki/display/CKB/What+is+CERA-20C>`_
    5972       
    60         `CERA-20C documentation <https://www.ecmwf.int/en/forecasts/datasets/archive-datasets/reanalysis-datasets/cera-20c>`_
    61    
     73        `CERA-20C dataset <https://www.ecmwf.int/en/forecasts/datasets/archive-datasets/reanalysis-datasets/cera-20c>`_
     74           
    6275    ERA5
    6376        `What is ERA5 <https://software.ecmwf.int/wiki/display/CKB/What+is+ERA5>`_
    … …  
    6982        `How to migrate from ECMWF Web API to CDS API <https://confluence.ecmwf.int/display/CKB/How+to+migrate+from+ECMWF+Web+API+to+CDS+API>`_
    7083       
    71         `ERA5 Documentation <https://software.ecmwf.int/wiki/display/CKB/ERA5+data+documentation>`_
    72        
     84        `ERA5 Documentation <https://software.ecmwf.int/wiki/display/CKB/ERA5+data+documentation>`_       
    7385
    7486Third Party Libraries
    … …  
    97109
    98110
     111Technical Information of ECMWF serves
     112
     113    `Introduction presentation to SLURM  <https://confluence.ecmwf.int/download/attachments/73008494/intro-slurm-2017.pdf?version=1&modificationDate=1488574096323&api=v2>`_
     114
    99115Troubleshooting
    100116    `ECMWF Web API Troubleshooting <https://confluence.ecmwf.int/display/WEBAPI/Web-API+Troubleshooting>`_

  • for_developers/Sphinx/source/Evaluation/metrics.rst

    r41c9dbc r0b00607  
    11Metrics
    22=======
     3
     4   UNDER CONSTRUCTION
    35
    46.. csv-table: TTEST
    … …  
    79..   :header-rows: 1
    810
     11
     12
     13.. toctree::
     14    :hidden:
     15    :maxdepth: 2

  • for_developers/Sphinx/source/_files/packages.png

    r41c9dbc r0b00607  
    1 ../../../../for_developers/packages.png
     1../../../packages.png

  • for_developers/Sphinx/source/authors.rst

    r41c9dbc r0b00607  
    33**************
    44
    5 | MSc. Anne Philipp
     5| Anne Philipp
    66| Department of Meteorology and Geophysics /
    77| Aerosol Physics and Environmental physics
    … …  
    1414
    1515| Leopold Haimberger
    16 | University of Vienna
     16| Department of Meteorology and Geophysics
     17| University of Vienna
     18| Althanstraße 14 / UZA II
     19| 1090 Vienna, Austria
     20| mail: leopold.haimberger [at] univie.ac.at
    1721
    1822   

  • for_developers/Sphinx/source/conf.py

    r41c9dbc r0b00607  
    1414#
    1515import os
    16 import sys
    17 sys.path.insert(0, os.path.abspath('../../../source/python'))
    18 sys.path.insert(0, os.path.abspath('../../../source/python/mods'))
    19 sys.path.insert(0, os.path.abspath('../../../source/python/classes'))
    20 
    21 
     16import sys, glob
     17sys.path.insert(0, os.path.abspath('../../../Source/Python'))
     18sys.path.insert(0, os.path.abspath('../../../Source/Python/Mods'))
     19sys.path.insert(0, os.path.abspath('../../../Source/Python/Classes'))
     20
     21sys.path.insert(0, os.path.abspath('_static/python'))
     22
     23#fortran_src = [f for f in os.listdir('../../../source/fortran') if '.f90' in f or '.f' in f]
     24#print(fortran_src)
    2225
    2326# -- Project information -----------------------------------------------------
    … …  
    5255    'sphinx.ext.viewcode',
    5356    'sphinx.ext.githubpages',
    54 #    'sphinx_pyreverse',
    5557    'sphinx-jsonschema',
    56     'sphinx.ext.intersphinx'
    57 ]
     58    'sphinx.ext.intersphinx',
     59    'hidden_code_block',
     60    'sphinxcontrib.exceltable',
     61    'sphinxcontrib.seqdiag',
     62    'sphinxcontrib.blockdiag',
     63    'sphinx.ext.todo',
     64#    'sphinxfortran.fortran_autodoc',
     65#    'sphinxfortran.fortran_domain'
     66]
     67
     68
     69# Fontpath for seqdiag (truetype font)
     70seqdiag_fontpath = '/usr/share/fonts/dejavu/DejaVuSerif.ttf'
     71
     72# Fontpath for blockdiag (truetype font)
     73blockdiag_fontpath = '/usr/share/fonts/dejavu/DejaVuSerif.ttf'
     74
    5875
    5976# Add any paths that contain templates here, relative to this directory.
    … …  
    186203# If true, `todo` and `todoList` produce output, else they produce nothing.
    187204todo_include_todos = True
     205#todo_link_only = True
    188206
    189207
    … …  
    194212
    195213
    196 

  • for_developers/Sphinx/source/documentation.rst

    r41c9dbc r0b00607  
    1 =============
     1*************
    22Documentation
    3 =============
     3*************
    44       
    5     Overview
    6       - What is ``flex_extract``?
     5    Overview (Under construction)
    76     
    8         - general overview
    9         - motivation, aim
    10         - directory structure
    11        
    12       - Software components
    13         complete component structure (table or diagram)
    14        
    15         - Python package
    16        
    17           - Package diagram
    18           - Files and modules as table with information about unit tests
    19           - Api
    20          
    21         - Fortran program - CONVERT2
    22        
    23           - Package diagram
    24           - Api
    25          
    26         - Wrapping shell-scripts
    27        
    28       - Dependencies
    29       - General context (for who?, Usecases, work tree)
    30       - Access to ECMWF
    31       - Application modes
    32       - Call tree
    33       - Sequence diagram
     7    Control & Input Data
    348   
    35     Control & Input Data
    36       - Controlling files
    37      
    38         - setup.sh (explain input parameters)           
    39         - job scripts
    40         - compile job
    41         - run[_local].sh
    42        
    43       - Input Data
    44      
    45         - CONTROL files
    46        
    47           - The CONTROL file
    48           - CONTROL parameters
    49           - Examples
    50           - Changes and its downward compatibilities (eg M _ /grid)
    51          
    52         - ECMWF_ENV file
    53         - Templates
    54        
    55     Output Data
    56       - mars request file
    57       - Mars grib files
    58       - flux files
    59       - index.idx
    60       - CONTROL (neu)
    61       - fort files
    62       - EN* files
    63      
    64         - Usual  output
    65         - Output for pure forecast
    66         - Output for ensemble members
    67         - Output for new precipitation disaggregation
     9    Output Data (Under construction)
    6810   
    69     Disaggregation
    70       - What is disaggregation?
    71       - Why do we need it/ which fields?
    72       - Method for precipitation
    73      
    74         - Old method
    75         - New method
    76         - Differences in resulting Gribfiles
    77        
    78       - Method for rest fields
     11    Disaggregation of Flux Data (Under construction)
    7912   
    80     Vertical Coordinate
     13    Vertical Coordinate (Under construction)
    8114      - Methods (GAUSS, ETA, OMEGA)
    82       - CONVERT (Petra)
     15      - CONVERT
    8316   
    8417    Auto Generated Documentation
    8518      - Python
    86       - Fortran
     19      - Fortran (Under construction)
    8720
    8821   

  • for_developers/Sphinx/source/evaluation.rst

    r41c9dbc r0b00607  
    44    UNDER CONSTRUCTION
    55   
     6 
     7 
     8 
    69   
    710   
    8 Tests
    9 - unit tests
    10   - table with all files and their functions and which already have unit tests
    11 - regression tests
    12 - performance
    13 - test runs with control files
     11.. Tests
     12    - unit tests
     13      - table with all files and their functions and which already have unit tests
     14    - regression tests
     15    - performance
     16    - test runs with control files
    1417
    15 Metriken
    16 - list all of them
     18    Metriken
     19    - list all of them
    1720
    1821
    19 Coverage
    20 - test coverage?!
    21 - documentation coverage
     22    Coverage
     23    - test coverage?!
     24    - documentation coverage
    2225
    23 Static Analysis
    24 - pylint
    25 - pyreverse
    26 - radon
    27    
    28 Performance analysis:
    29 -       Whole software
    30 -       Disaggregation
    31 -       Convert2
    32 -   prepare_flexpart
    33 -       Andere komponenten
     26    Static Analysis
     27    - pylint
     28    - pyreverse
     29    - radon
     30       
     31    Performance analysis:
     32    -   Whole software
     33    -   Disaggregation
     34    -   Convert2
     35    -   prepare_flexpart
     36    -   Andere komponenten
    3437
    3538   
    … …  
    3942    :maxdepth: 2
    4043   
    41 ..    evaluation/code
     44    Evaluation/staticcode
     45    Evaluation/testcases
     46    Evaluation/metrics
    4247   

  • for_developers/Sphinx/source/index.rst

    r41c9dbc r0b00607  
    44   contain the root `toctree` directive.
    55
    6 ========================================================
    7 Welcome to :literal:`flex_extract's` user documentation!
    8 ========================================================
     6=================================================
     7Welcome to ``flex_extract``'s user documentation!
     8=================================================
    99
    10 :literal:`flex_extract` is an open-source software to retrieve meteorological fields from the European Centre for Medium-Range Weather Forecasts (ECMWF) Mars archive to serve as input files for the FLEXTRA/FLEXPART Atmospheric Transport Modelling system.
    11 
    12 
    13 .. The scripts can be 1) executed locally using the ECMWF WebMARS interface, or 2) on the ECMWF member
    14 .. state gateway server or HPC facility using the ecaccess software package. As another option the scripts can 3)
    15 .. also be triggered using ecflow scheduling software used by ECMWF for data dissemination.
    16 
    17 .. All third-party software and libraries required by :literal:`flex_extract` are open source and free of charge.
     10``Flex_extract`` is an open-source software to retrieve meteorological fields from the European Centre for Medium-Range Weather Forecasts (ECMWF) Mars archive to serve as input files for the FLEXTRA/FLEXPART Atmospheric Transport Modelling system.
    1811
    1912.. raw:: html
    … …  
    6053            <img style="position:absolute;height:90%;vertical-align:bottom;" src="_static/guide_icon.png">
    6154          </div>
    62           <h2><a href="first_steps.html">First Steps</a></h2>
     55          <h2><a href="quick_start.html">Quick Start</a></h2>
    6356          <ul>
    6457            <li><a href="">...</a></li>
    … …  
    140133
    141134   installation
    142    first_steps
     135   quick_start
    143136   ecmwf_data
    144137   documentation

  • for_developers/Sphinx/source/installation.rst

    r41c9dbc r0b00607  
    6767
    6868**Public user**:
    69     To be able to download all intended public datasets with ``flex_extract`` such as **ERA-Interim**, **CERA-20C** and **ERA5**, the public user has to create 2 accounts.
    70     At first, registration at the ECMWF website by filling out this `registration form`_ is needed to be able to retrieve *ERA-Interim* and *CERA-20C* data. For the retrievement of *ERA5* a registration at the `Copernicus Climate Data Store <https://cds.climate.copernicus.eu/user/register>`_ is needed in addition.
     69    To be able to download public datasets with ``flex_extract`` such as **ERA-Interim** and **CERA-20C** (**ERA5** is not supported via ECMWF Web API anymore), the public user has to create an account at ECMWF.
     70    Use the registration at the ECMWF website by filling out this `registration form`.
     71   
     72    .. note::
     73
     74        In the future retrievement of *ERA5* will be possible via the CDS API for public users also. Then a registration at the `Copernicus Climate Data Store <https://cds.climate.copernicus.eu/user/register>`_ is needed in addition.
    7175   
    7276   
    … …  
    7781.. _ref-licence:
    7882   
    79 Agree on Licences for Public Datasets
     83Agree on licences for public datasets
    8084=====================================
    8185
    … …  
    123127.. _ref-requirements:
    124128 
    125 Environment Requirements
     129Environment requirements
    126130========================
    127131
    … …  
    173177.. _ref-remote-mode:
    174178
    175 Remote Mode
     179Remote mode
    176180-----------
    177181
    178182.. _ref-req-remote:
    179183 
    180 Remote Environment Requirements
     184Remote environment requirements
    181185^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    182186
    … …  
    196200.. _ref-prep-remote:
    197201
    198 Prepare Remote Environment
     202Prepare remote environment
    199203^^^^^^^^^^^^^^^^^^^^^^^^^^
    200204 
    … …  
    206210.. _ref-install-remote:
    207211
    208 Remote Installation
     212Remote installation
    209213^^^^^^^^^^^^^^^^^^^
    210214
    … …  
    266270``flex_extract`` root directory for installation.
    267271Before executing it, it is necessary to adapt some parameters from ``setup.sh``
    268 described in :ref:`ref-setup`.
     272described in :doc:`Documentation/Input/setup`.
    269273
    270274Open ``setup.sh`` with your editor and adapt the values: 
    … …  
    275279| .. code-block:: bash                         | .. code-block:: bash                         |
    276280|                                              |                                              |
    277 |   ...                                        |   ...                                        |    
     281|   ...                                        |   ...                                        |   
    278282|   # -----------------------------------------|   # -----------------------------------------|
    279283|   # AVAILABLE COMMANDLINE ARGUMENTS TO SET   |   # AVAILABLE COMMANDLINE ARGUMENTS TO SET   |
    … …  
    290294|   JOB_TEMPLATE='job.template'                |   JOB_TEMPLATE='job.template'                |
    291295|   CONTROLFILE='CONTROL_EA5'                  |   CONTROLFILE='CONTROL_EA5'                  |
    292 |   ...                                        |   ...                                        |    
     296|   ...                                        |   ...                                        |   
    293297+----------------------------------------------+----------------------------------------------+
    294298
    … …  
    359363.. _ref-gateway-mode:
    360364
    361 Gateway Mode
     365Gateway mode
    362366------------
    363367
    … …  
    365369.. _ref-req-gateway:
    366370 
    367 Gateway Environment Requirements
     371Gateway environment requirements
    368372^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    369373
    … …  
    379383.. _ref-prep-gateway:
    380384
    381 Prepare Gateway Environment
     385Prepare gateway environment
    382386^^^^^^^^^^^^^^^^^^^^^^^^^^^
    383387
    384388The easiest way to install all required packages is to use the package management system of your Linux distribution. To do so, it is necessary to use a user with admin rights.
    385 The installation was tested on a *Debian GNU/Linux buster/sid* and an *Ubuntu 18.04 Bionic Beaver* system.
     389The installation was tested on a *GNU/Linux Debian buster* and an *Ubuntu 18.04 Bionic Beaver* system.
    386390
    387391.. code-block:: sh
    … …  
    389393   # On a Linux Debian or Ubuntu system do
    390394   # (if not already available):
    391    sudo apt -PV install python3
    392    sudo apt -PV install pip
    393    pip install genshi
    394    pip install numpy
     395   apt-get install python3
     396   apt-get install pip
     397   apt-get install genshi
     398   apt-get install numpy
    395399
    396400
    397401.. _ref-test-gateway:
    398402
    399 Test Gateway Environment
     403Test gateway environment
    400404^^^^^^^^^^^^^^^^^^^^^^^^
    401405 
    … …  
    414418.. _ref-install-gateway:
    415419
    416 Gateway Installation
     420Gateway installation
    417421^^^^^^^^^^^^^^^^^^^^
    418422
    … …  
    431435   Your passcode: ***
    432436   
    433 ``Flex_extract`` will be run on an ECMWF server which makes the setup the same as for the **remote mode**. In the ``setup.sh`` script :ref:`[ref] <ref-setup>`, select the ``Makefile.gfortran`` for the ``CONVERT2`` Fortran program and the ECMWF server (*target*) you would like to use.
     437``Flex_extract`` will be run on an ECMWF server which makes the setup the same as for the **remote mode**. In the ``setup.sh`` script `[ref] <Documentation/Input/setup.html>`_, select the ``Makefile.gfortran`` for the ``CONVERT2`` Fortran program and the ECMWF server (*target*) you would like to use.
    434438The job script, send to the job queue via the ECaccess software, selects again automatically the correct libraries from the module system. For enableing the file transfer you have to set the *ECUID*, *ECGID*, *GATEWAY* and *DESTINATION* parameter values.
    435439 
    … …  
    506510.. _ref-local-mode:
    507511
    508 Local Mode
     512Local mode
    509513----------
    510514
    … …  
    513517.. _ref-req-local:
    514518 
    515 Local Environment Requirements
     519Local environment requirements
    516520^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    517521
    … …  
    536540.. _ref-prep-local:
    537541
    538 Prepare Local Environment
     542Prepare local environment
    539543^^^^^^^^^^^^^^^^^^^^^^^^^
    540544
    … …  
    546550   # On a Linux Debian or Ubuntu system do
    547551   # (if not already available):
    548    sudo apt -PV install python3
    549    sudo apt -PV install pip
    550    sudo apt -PV install gfortran
    551    sudo apt -PV install fftw3-dev
    552    sudo apt -PV install libeccodes-dev
    553    sudo apt -PV install libemos-dev
    554    sudo apt -PV install python3-eccodes
    555    pip install genshi
    556    pip install numpy
     552   apt-get install python3 (usually available on normal Linux systems)
     553   apt-get install pip
     554   apt-get install gfortran
     555   apt-get install fftw3-dev
     556   apt-get install libeccodes-dev
     557   apt-get install libemos-dev
     558   apt-get install python3-eccodes
     559   apt-get install genshi
     560   apt-get install numpy
    557561   pip install cdsapi
    558562   pip install ecmwf-api-client
    … …  
    571575The CDS API (cdsapi) and the ECMWF Web API (ecmwf-api-client) have both to be installed since ERA5 can only be retrieved with the ``CDS API`` and all other datasets with the ``ECMWF Web API``.     
    572576       
     577.. note::
     578
     579    Since **public users** currently don't have access to the full *ERA5* dataset they can skip the installation of the ``CDS API``.
     580   
    573581Both user groups have to provide key's with their credentials for the Web API's in their home directory. Therefore, follow these instructions:
    574582   
    … …  
    582590.. _ref-test-local:
    583591   
    584 Test Local Environment
     592Test local environment
    585593^^^^^^^^^^^^^^^^^^^^^^
    586594
    … …  
    615623"""""""""""""
    616624
    617 **Member user**
    618     Please use this piece of python code:
    619 
    620     .. code-block:: python
    621 
    622         from ecmwfapi import ECMWFService
     625
     626+----------------------------------------------------------+----------------------------------------------------------+
     627|Please use this piece of python code for **Member user**: |Please use this piece of python code for **Public user**: |
     628+----------------------------------------------------------+----------------------------------------------------------+
     629|.. code-block:: python                                    |.. code-block:: python                                    |
     630|                                                          |                                                          |
     631|    from ecmwfapi import ECMWFService                     |    from ecmwfapi import ECMWFDataServer                  |
     632|                                                          |                                                          |
     633|    server = ECMWFService('mars')                         |    server = ECMWFDataServer()                            |
     634|                                                          |                                                          |
     635|    server.retrieve({                                     |    server.retrieve({                                     |
     636|        'stream'    : "oper",                             |        'stream'    : "enda",                             |
     637|        'levtype'   : "sfc",                              |        'levtype'   : "sfc",                              |
     638|        'param'     : "165.128/166.128/167.128",          |        'param'     : "165.128/166.128/167.128",          |
     639|        'dataset'   : "interim",                          |        'dataset'   : "cera20c",                          |
     640|        'step'      : "0",                                |        'step'      : "0",                                |
     641|        'grid'      : "0.75/0.75",                        |        'grid'      : "1./1.",                            |
     642|        'time'      : "00/06/12/18",                      |        'time'      : "00/06/12/18",                      |
     643|        'date'      : "2014-07-01/to/2014-07-31",         |        'date'      : "2000-07-01/to/2000-07-31",         |
     644|        'type'      : "an",                               |        'type'      : "an",                               |
     645|        'class'     : "ei",                               |        'class'     : "ep",                               |
     646|        'target'    : "download_erainterim_ecmwfapi.grib" |        'target'    : "download_cera20c_ecmwfapi.grib"    |
     647|    })                                                    |    })                                                    |
     648+----------------------------------------------------------+----------------------------------------------------------+
     649
    623650           
    624         server = ECMWFService('mars')
    625            
    626         server.retrieve({
    627             'stream'    : "oper",
    628             'levtype'   : "sfc",
    629             'param'     : "165.128/166.128/167.128",
    630             'dataset'   : "interim",
    631             'step'      : "0",
    632             'grid'      : "0.75/0.75",
    633             'time'      : "00/06/12/18",
    634             'date'      : "2014-07-01/to/2014-07-31",
    635             'type'      : "an",
    636             'class'     : "ei",
    637             'target'    : "download_erainterim_ecmwfapi.grib"
    638         })
    639 
    640 
    641 **Public user**
    642     Please use this piece of python code:
    643 
    644     .. code-block:: python
    645 
    646         from ecmwfapi import ECMWFDataServer
    647            
    648         server = ECMWFDataServer()
    649            
    650         server.retrieve({
    651             'stream'    : "enda",
    652             'levtype'   : "sfc",
    653             'param'     : "165.128/166.128/167.128",
    654             'dataset'   : "cera20c",
    655             'step'      : "0",
    656             'grid'      : "1./1.",
    657             'time'      : "00/06/12/18",
    658             'date'      : "2000-07-01/to/2000-07-31",
    659             'type'      : "an",
    660             'class'     : "ep",
    661             'target'    : "download_cera20c_ecmwfapi.grib"
    662         })
    663651   
    664652CDS API
    … …  
    667655Since ERA5 extraction with CDS API might take some time due to the very high number of requests, you can start by retrieving some online stored pressure levels (not from MARS). This is usually much faster and gives a quick result to find out if the web API works:
    668656
    669 **All users**
    670     Please use this piece of python code:
    671    
    672     .. code-block:: python
    673 
    674         import cdsapi
    675        
    676         c = cdsapi.Client()
    677        
    678         c.retrieve("reanalysis-era5-pressure-levels",
    679         {
    680         "variable": "temperature",
    681         "pressure_level": "1000",
    682         "product_type": "reanalysis",
    683         "year": "2008",
    684         "month": "01",
    685         "day": "01",
    686         "time": "12:00",
    687         "format": "grib"
    688         },
    689         "download_cdsapi.grib")
     657Please use this piece of python code to retrieve a small sample of *ERA5* pressure levels:
     658
     659.. code-block:: python
     660
     661    import cdsapi
     662   
     663    c = cdsapi.Client()
     664   
     665    c.retrieve("reanalysis-era5-pressure-levels",
     666    {
     667    "variable": "temperature",
     668    "pressure_level": "1000",
     669    "product_type": "reanalysis",
     670    "year": "2008",
     671    "month": "01",
     672    "day": "01",
     673    "time": "12:00",
     674    "format": "grib"
     675    },
     676    "download_cdsapi.grib")
    690677
    691678
    692679Afterwards, when you know that the CDS API generally works, you can try to extract some
    693 data from the MARS archive. From the latest experience this can take a while.   
    694 
    695 **Member user**
    696     Please use this piece of python code:
    697    
    698     .. code-block:: python
    699 
    700        import cdsapi
    701        
    702        c = cdsapi.Client()
    703        
    704        c.retrieve('reanalysis-era5-complete',
    705        {
    706            'class'   : 'ea',
    707            'expver'  : '1',
    708            'stream'  : 'oper',
    709            'type'    : 'fc',
    710            'step'    : '3/to/12/by/3',
    711            'param'   : '130.128',
    712            'levtype' : 'ml',
    713            'levelist': '135/to/137',
    714            'date'    : '2013-01-01',
    715            'time'    : '06/18',
    716            'area'    : '50/-5/40/5',
    717            'grid'    : '1.0/1.0',
    718            'format'  : 'grib',
    719        }, 'download_era5_cdsapi.grib')
    720 
    721 
    722 **Public user**
     680data from the MARS archive. From the latest experience we know that this can take a while.   
     681
     682.. **Member user**
     683
     684Please use this piece of python code to retrieve a small *ERA5* data sample as a **member user**! The **public user** doesn't have access to the full *ERA5* dataset!
     685
     686.. code-block:: python
     687
     688   import cdsapi
     689   
     690   c = cdsapi.Client()
     691   
     692   c.retrieve('reanalysis-era5-complete',
     693   {
     694       'class'   : 'ea',
     695       'expver'  : '1',
     696       'stream'  : 'oper',
     697       'type'    : 'fc',
     698       'step'    : '3/to/12/by/3',
     699       'param'   : '130.128',
     700       'levtype' : 'ml',
     701       'levelist': '135/to/137',
     702       'date'    : '2013-01-01',
     703       'time'    : '06/18',
     704       'area'    : '50/-5/40/5',
     705       'grid'    : '1.0/1.0',
     706       'format'  : 'grib',
     707   }, 'download_era5_cdsapi.grib')
     708
     709
     710..  ********************** COMMENTED OUT FOR FUTURE
     711    ********************** PUBLIC RETRIEVAL IS CURRENTLY NOT ACCESSIBLE
     712   
     713    **Public user**
    723714    Please use this piece of python code:
    724715
    … …  
    754745.. _ref-install-local:
    755746
    756 Local Installation
     747Local installation
    757748^^^^^^^^^^^^^^^^^^
    758749
    … …  
    763754the ``ifort`` compiler:
    764755
    765  *Makefile.local.gfortran
    766  *Makefile.local.ifort
     756 * Makefile.local.gfortran
     757 * Makefile.local.ifort
    767758
    768759They can be found in the path ``flex_extract_vX.X/source/fortran``, where
    … …  
    928919   
    929920
    930 Mars request test
    931 -----------------
    932 
    933     IN PREPARATION ...
    934921
    935922Full test
    936923---------
    937924
    938     IN PREPARATION ...
    939 
    940    
     925    see :doc:`quick_start`
     926
     927   
Note: See TracChangeset for help on using the changeset viewer.
hosted by ZAMG