Changeset b1674ed in flex_extract.git for For_developers


Ignore:
Timestamp:
Apr 1, 2020, 5:03:13 PM (4 years ago)
Author:
Anne Philipp <anne.philipp@…>
Branches:
master, ctbto, dev
Children:
d78851d, e0005c9
Parents:
50f9ca6
Message:

updated online documentation FIX for tickets #265 and #262

Location:
For_developers
Files:
2 added
31 edited

Legend:

Unmodified
Added
Removed

  • For_developers/Sphinx/source/Developers/gen_docu.rst

    rba99230 rb1674ed  
    1010Developers working on ``flex_extract`` should make extensive use of the prepared test cases, unit tests and update the documentation. For this, some additional software is necessary:
    1111
    12 - pylint
    13 - pytest
    14 - mock
    15 - graphviz
    16 - sphinx
    17 - pip install sphinxcontrib-exceltable
    18 - pip install seqdiag
    19 - pip install sphinxcontrib-seqdiag
    20 - pip install sphinxcontrib-blockdiag
    21 - pip install pycallgraph
     12.. code-block:: sh
    2213
    23 #- pip install sphinx-fortran
     14  pip install pylint
     15  pip install pytest
     16  pip install mock
     17  pip install graphviz
     18  pip install sphinx
     19  pip install sphinxcontrib-exceltable
     20  pip install seqdiag
     21  pip install sphinxcontrib-seqdiag
     22  pip install sphinxcontrib-blockdiag
     23  pip install pycallgraph
     24
     25
    2426
    2527
    … …  
    3133
    3234It uses ``pyreverse`` to generate a class and a package diagram with graphviz and overwrites the old files in the developers directory.
    33 ``pyreverse`` creates ``dot`` files and with the ``dot`` program of the ``graphviz`` software the ``png`` files are created. Everything happens in the Python source directory before moving them finally to the ``for_developers`` directory. The Sphinx source code has a ``_files`` directory which contains links to these ``png`` files and therefore they should not be renamed. 
     35``pyreverse`` creates ``dot`` files and with the ``dot`` program of the ``graphviz`` software the ``png`` files are created. Everything happens in the Python source directory before moving them finally to the ``For_developers`` directory. The Sphinx source code has a ``_files`` directory which contains links to these ``png`` files and therefore they should not be renamed. 
    3436
    3537

  • For_developers/Sphinx/source/Documentation/Input/compilejob.rst

    rba99230 rb1674ed  
    7474      module load emos/455-r64
    7575      export FLEXPART_ROOT_SCRIPTS=${HOME}
    76       export MAKEFILE=Makefile.gfortran
     76      export MAKEFILE=makefile_ecgate
    7777      ;;
    7878      *cca*)
    … …  
    8888      export SCRATCH=/scratch/ms/${GROUP}/${USER}
    8989      export FLEXPART_ROOT_SCRIPTS=${HOME}
    90       export MAKEFILE=Makefile.gfortran
     90      export MAKEFILE=makefile_ecgate
    9191      ;;
    9292    esac
    … …  
    9595    cd ${FLEXPART_ROOT_SCRIPTS}/flex_extract_v${VERSION}   # if FLEXPART_ROOT is not set this means cd to the home directory
    9696    tar -xvf ${HOME}/flex_extract_v${VERSION}.tar
    97     cd source/fortran
    98     \rm *.o *.mod CONVERT2
     97    cd Source/Fortran
     98    \rm *.o *.mod calc_etadot
    9999    make -f ${MAKEFILE} >flexcompile 2>flexcompile
    100100
    101     ls -l CONVERT2 >>flexcompile
     101    ls -l calc_etadot >>flexcompile
    102102    if [ $? -eq 0 ]; then
    103103      echo 'SUCCESS!' >>flexcompile

  • For_developers/Sphinx/source/Documentation/Input/control.rst

    rd9abaac rb1674ed  
    3434--------------------------------
    3535
    36 A number of example files can be found in the directory :literal:`flex_extract_vX.X/run/control/`.
     36A number of example files can be found in the directory :literal:`flex_extract_vX.X/Run/Control/`.
    3737They can be used as a template for adaptations and understand what's possible to
    3838retrieve from ECMWF's archive.

  • For_developers/Sphinx/source/Documentation/Input/control_params.rst

    rba99230 rb1674ed  
    44     
    55   
     6
    67   
    78.. _ref-control-user:

  • For_developers/Sphinx/source/Documentation/Input/ecmwf_env.rst

    r30f7911 rb1674ed  
    33****************************************
    44
    5 This file contains the user credentials for working on ECMWF servers and transferring files between the ECMWF servers and the local gateway server. It is located in the ``flex_extract_vX.X/run`` directory and will be created in the installation process for the application modes **remote** and **gateway**.
     5This file contains the user credentials for working on ECMWF servers and transferring files between the ECMWF servers and the local gateway server. It is located in the ``flex_extract_vX.X/Run`` directory and will be created in the installation process for the application modes **remote** and **gateway**.
    66
    77This file is based on the template ``ECMWF_ENV.template`` which is located in the ``templates`` directory.
    … …  
    99.. note::
    1010 
    11    In the **local** mode it is not present.
     11   In the **local** mode this file is not present.
    1212
    1313

  • For_developers/Sphinx/source/Documentation/Input/examples.rst

    rba99230 rb1674ed  
    33**********************
    44
    5 ``Flex_extract`` has a couple of example ``CONTROL`` files for a number of different data set constellations in the directory path ``flex_extract_vX.X/run/control``.
     5``Flex_extract`` has a couple of example ``CONTROL`` files for a number of different data set constellations in the directory path ``flex_extract_vX.X/Run/Control``.
    66
    77Here is a list of the example files and a description of the data set:
    … …  
    1010   This file is not intended to be used with ``flex_extract``. It has a list of all possible parameters and their default values for a quick overview.
    1111   
    12 CONTROL_CERA
    13 CONTROL_CERA.global
    14 CONTROL_CERA.public
    15 CONTROL_EA5
    16 CONTROL_EA5.global
    17 CONTROL_EA5.public
    18 CONTROL_EI
    19 CONTROL_EI.global
    20 CONTROL_EI.public
    21 CONTROL_OD.ELDA.FC.eta.ens.double
    22 CONTROL_OD.ENFO.CF
    23 CONTROL_OD.ENFO.CV
    24 CONTROL_OD.ENFO.PF
    25 CONTROL_OD.ENFO.PF.36hours
    26 CONTROL_OD.ENFO.PF.ens
    27 CONTROL_OD.OPER.4V.operational
    28 CONTROL_OD.OPER.FC.36hours
    29 CONTROL_OD.OPER.FC.eta.global
    30 CONTROL_OD.OPER.FC.eta.highres
    31 CONTROL_OD.OPER.FC.gauss.highres
    32 CONTROL_OD.OPER.FC.operational
    33 CONTROL_OD.OPER.FC.twiceaday.1hourly
    34 CONTROL_OD.OPER.FC.twiceaday.3hourly
     12.. code-block:: bash
     13
     14        CONTROL_CERA
     15        CONTROL_CERA.global
     16        CONTROL_CERA.public
     17        CONTROL_EA5
     18        CONTROL_EA5.global
     19        CONTROL_EA5.public
     20        CONTROL_EI
     21        CONTROL_EI.global
     22        CONTROL_EI.public
     23        CONTROL_OD.ELDA.FC.eta.ens.double
     24        CONTROL_OD.ENFO.CF
     25        CONTROL_OD.ENFO.CV
     26        CONTROL_OD.ENFO.PF
     27        CONTROL_OD.ENFO.PF.36hours
     28        CONTROL_OD.ENFO.PF.ens
     29        CONTROL_OD.OPER.4V.operational
     30        CONTROL_OD.OPER.FC.36hours
     31        CONTROL_OD.OPER.FC.eta.global
     32        CONTROL_OD.OPER.FC.eta.highres
     33        CONTROL_OD.OPER.FC.gauss.highres
     34        CONTROL_OD.OPER.FC.operational
     35        CONTROL_OD.OPER.FC.twiceaday.1hourly
     36        CONTROL_OD.OPER.FC.twiceaday.3hourly
    3537
    3638   

  • For_developers/Sphinx/source/Documentation/Input/fortran_makefile.rst

    rba99230 rb1674ed  
    1 ***********************************
    2 The Fortran Makefile - ``CONVERT2``
    3 ***********************************
     1**************************************
     2The Fortran Makefile - ``calc_etadot``
     3**************************************
    44
    55.. _ref-convert:
    66
    77``Flex_extract``'s Fortran program will be compiled during
    8 the installation process to get the executable named ``CONVERT2``.
     8the installation process to get the executable named ``calc_etadot``.
    99
    10 ``flex_extract`` has a couple of ``Makefiles`` prepared which can be found in the directory
    11 ``flex_extract_vX.X/source/fortran``, where ``vX.X`` should be substituted with the current version number.
    12 A list of these ``Makefiles`` are shown below:
     10``Flex_extract`` has a couple of ``makefiles`` prepared which can be found in the directory
     11``flex_extract_vX.X/Source/Fortran``, where ``vX.X`` should be substituted with the current version number.
     12A list of these ``makefiles`` are shown below:
    1313
    1414
    … …  
    1616| Files to be used as they are!
    1717   
    18     | **Makefile.gfortran**
     18    | **makefile_ecgate**
    1919    | For the use on ECMWF's server **ecgate**.
    2020
    21     | **Makefile.CRAY**
     21    | **makefile_cray**
    2222    | For the use on ECMWF's server **cca/ccb**.
    2323   
    … …  
    2525| It is necessary to adapt **ECCODES_INCLUDE_DIR** and **ECCODES_LIB**
    2626 
    27     | **Makefile.local.gfortran**
    28     | For the use with gfortran compiler.
     27    | **makefile_fast**
     28    | For the use with gfortran compiler and optimization mode.
    2929
    30     | **Makefile.local.ifort**
    31     | For the use with ifort compiler.
     30    | **makefile_debug**
     31    | For the use with gfortran compiler in debugging mode.
    3232
    3333
    34 For instructions on how to adapt the ``Makefiles`` for the local application mode
     34For instructions on how to adapt the ``makefiles`` for the local application mode
    3535please see :ref:`ref-install-local`.
    3636

  • For_developers/Sphinx/source/Documentation/Input/jobscript.rst

    rba99230 rb1674ed  
    55The jobscript is a Korn-shell script which will be created at runtime for each ``flex_extract`` execution in the application modes **remote** and **gateway**.
    66
    7 It is based on the ``job.temp`` template file which is stored in the ``templates`` directory.
    8 This template is by itself generated in the installation process from a ``jobscript.template`` template file.
     7It is based on the ``job.temp`` template file which is stored in the ``Templates`` directory.
     8This template is by itself generated in the installation process from a ``job.template`` template file.
    99
    1010``Flex_extract`` uses the python package `genshi <https://genshi.edgewall.org/>`_ to generate
    … …  
    7474      module load eccodes/2.12.0
    7575      module load emos/455-r64
    76       export PATH=${PATH}:${HOME}/flex_extract_v7.1/source/python
     76      export PATH=${PATH}:${HOME}/flex_extract_v7.1/Source/Python
    7777      ;;
    7878      *cca*)
    … …  
    8383      module load emos
    8484      export SCRATCH=${TMPDIR}
    85       export PATH=${PATH}:${HOME}/flex_extract_v7.1/source/python
     85      export PATH=${PATH}:${HOME}/flex_extract_v7.1/Source/Python
    8686      ;;
    8787    esac

  • For_developers/Sphinx/source/Documentation/Input/run.rst

    rba99230 rb1674ed  
    1111Based on these input information ``flex_extract`` applies one of the application modes to either retrieve the ECMWF data via a Web API on a local maschine or submit a jobscript to ECMWF servers and retrieve the data there with sending the files to the local system eventually.
    1212
    13 
    14 
    15 PUT IN HERE A BLOCKDIAGRAM OF RAW PROGRAM FLOW
    1613
    1714
    … …  
    4340
    4441It is also possible to start ``flex_extract`` directly from command line by using the ``submit.py`` script instead of the wrapping Shell script ``run.sh``.  This top-level script is located in
    45 ``flex_extract_vX.X/source/python`` and is executable. With the ``help`` parameter we see again all possible
     42``flex_extract_vX.X/Source/Python`` and is executable. With the ``help`` parameter we see again all possible
    4643command line parameter.
    4744

  • For_developers/Sphinx/source/Documentation/Input/setup.rst

    rba99230 rb1674ed  
    1313- copying tar-ball to target location (depending on application mode and installation path)
    1414- submit compilation script to batch queue at ECMWF servers (in **remote** and **gateway** mode) or just untar tar-ball at target location (**local mode**)
    15 - compilation of the FORTRAN90 program ``CONVERT2``
     15- compilation of the FORTRAN90 program ``calc_etadot``
    1616
    1717
    … …  
    156156
    157157It is also possible to start the installation process of ``flex_extract`` directly from command line by using the ``install.py`` script instead of the wrapping Shell script ``setup.sh``.  This top-level script is located in
    158 ``flex_extract_vX.X/source/python`` and is executable. With the ``help`` parameter we see again all possible
     158``flex_extract_vX.X/Source/Python`` and is executable. With the ``help`` parameter we see again all possible
    159159command line parameter.
    160160

  • For_developers/Sphinx/source/Documentation/Input/templates.rst

    rba99230 rb1674ed  
    33*********
    44
    5 In ``flex_extract`` we use the Python package ``genshi`` (`ref <https://genshi.edgewall.org/>`_) to create specific files from templates. It is the most efficient way to be able to quickly adapt e.g. the job scripts send to the ECMWF batch queue system or the namelist file für the Fortran program without the need to change the program code.
     5In ``flex_extract`` we use the Python package `genshi <https://genshi.edgewall.org/>`_ to create specific files from templates. It is the most efficient way to be able to quickly adapt e.g. the job scripts send to the ECMWF batch queue system or the namelist file für the Fortran program without the need to change the program code.
    66
    77.. note::
    … …  
    1515----------
    1616
    17     This is the template for a Fortran namelist file called ``fort.4`` which will be read by ``CONVERT2``.
    18     It contains all the parameters ``CONVERT2`` needs.
     17    This is the template for a Fortran namelist file called ``fort.4`` which will be read by ``calc_etadot``.
     18    It contains all the parameters ``calc_etadot`` needs.
    1919   
    2020    .. code-block:: fortran
    … …  
    125125        cd $${FLEXPART_ROOT_SCRIPTS}/flex_extract_v$${VERSION}   # if FLEXPART_ROOT is not set this means cd to the home directory
    126126        tar -xvf $${HOME}/flex_extract_v$${VERSION}.tar
    127         cd source/fortran
     127        cd Source/Fortran
    128128        \rm *.o *.mod $fortran_program
    129129        make -f $${MAKEFILE} >flexcompile 2>flexcompile
    … …  
    191191          module load eccodes/2.12.0
    192192          module load emos/455-r64
    193           export PATH=$${PATH}:$${HOME}/flex_extract_v7.1/source/python
     193          export PATH=$${PATH}:$${HOME}/flex_extract_v7.1/Source/Python
    194194          ;;
    195195          *cca*)
    … …  
    200200          module load emos
    201201          export SCRATCH=$${TMPDIR}
    202           export PATH=$${PATH}:$${HOME}/flex_extract_v7.1/source/python
     202          export PATH=$${PATH}:$${HOME}/flex_extract_v7.1/Source/Python
    203203          ;;
    204204        esac

  • For_developers/Sphinx/source/Documentation/Overview/app_modes.rst

    rba99230 rb1674ed  
    88.. _member state: https://www.ecmwf.int/en/about/who-we-are/member-states
    99.. _instructions: https://apps.ecmwf.int/registration/
     10.. _ECMWF's instructions on gateway server: https://confluence.ecmwf.int/display/ECAC/ECaccess+Home
    1011
    1112   
    … …  
    1718
    1819  1. Remote (member)
    19       In the **Remote mode** the user works directly on ECMWF Linux member state server, such as ``ecgate`` or ``cca``. The software will be installed in the ``$HOME`` directory. The user does not need to install any of the additional third-party libraries mentioned in :ref:`ref-requirements` as ECMWF provides everything with environment modules. The module selection will be done automatically in ``flex_extract``.
     20      In the **Remote mode** the user works directly on ECMWF Linux member state server, such as ``ecgate`` or ``cca/ccb``. The software will be installed in the ``$HOME`` directory. The user does not need to install any of the additional third-party libraries mentioned in :ref:`ref-requirements` as ECMWF provides everything with environment modules. The module selection will be done automatically in ``flex_extract``.
    2021     
    2122.. _ref-gateway-desc:
    2223     
    2324  2. Gateway (member)
    24       The **Gateway mode** can be used if a local member state gateway server is in place. Then the job scripts can be submitted to the ECMWF Linux member state server via the ECMWF web access tool ``ecaccess``. The installation script of ``flex_extract`` must be executed at the local gateway server such that the software will be installed in the ``$HOME`` directory at the ECMWF server and some extra setup is done in the local ``flex_extract`` directory at the local gateway server. For more information about establishing a gateway server please see section ???. For the **Gateway mode** the necessary environment has to be established which is described in :ref:`ref-prep-gateway`.
     25      The **Gateway mode** can be used if a local member state gateway server is in place. Then the job scripts can be submitted to the ECMWF Linux member state server via the ECMWF web access tool ``ecaccess``. The installation script of ``flex_extract`` must be executed at the local gateway server such that the software will be installed in the ``$HOME`` directory at the ECMWF server and some extra setup is done in the local ``flex_extract`` directory at the local gateway server. For more information about establishing a gateway server please see `ECMWF's instructions on gateway server`_. For the **Gateway mode** the necessary environment has to be established which is described in :ref:`ref-prep-gateway`.
    2526
    2627.. _ref-local-desc:

  • For_developers/Sphinx/source/Documentation/Overview/prog_flow.rst

    rba99230 rb1674ed  
    2626        It creates and sends MARS-requests either on the local machine or on ECMWF server, that receives the data and stores them in a specific format in GRIB files. If the parameter ``REQUEST`` was set ``1`` the data are not received but a file ``mars_requests.csv`` is created with a list of MARS requests and their settings. If it is set to ``2`` the file is created in addition to retrieving the data. The requests are created in an optimised way by splitting in time, jobs  and parameters.   
    2727    3. Post-process data to create final ``FLEXPART`` input files:
    28         After all data is retrieved, the disaggregation of flux fields (`see here <../disagg.html>`_ ) is done as well as the calculation of vertical velocity (`see here <../vertco.html>`_) by the Fortran program ``COVERT2``. Eventually, the GRIB fields are merged together such that a single grib file per time step is available with all fields for ``FLEXPART``. Since model level fields are typically in *GRIB2* format whereas surface level fields are still in *GRIB1* format, they can be converted into GRIB2 if parameter ``FORMAT`` is set to *GRIB2*. Please note, however, that older versions of FLEXPART may have difficulties reading pure *GRIB2* files since some parameter IDs change in *GRIB2*. If the retrieval is executed remotely at ECMWF, the resulting files can be communicated to the local gateway server via the ``ECtrans`` utility if the parameter ``ECTRANS`` is set to ``1`` and the parameters ``GATEWAY``, ``DESTINATION`` have been set properly during installation. The status of the transfer can be checked with the command ``ecaccess-ectrans-list`` (on the local gateway server). If the script is executed locally the progress of the script can be followed with the usual Linux tools.
     28        After all data is retrieved, the disaggregation of flux fields (`see here <../disagg.html>`_ ) is done as well as the calculation of vertical velocity (`see here <../vertco.html>`_) by the Fortran program ``calc_etadot``. Eventually, the GRIB fields are merged together such that a single grib file per time step is available with all fields for ``FLEXPART``. Since model level fields are typically in *GRIB2* format whereas surface level fields are still in *GRIB1* format, they can be converted into GRIB2 if parameter ``FORMAT`` is set to *GRIB2*. Please note, however, that older versions of FLEXPART may have difficulties reading pure *GRIB2* files since some parameter IDs change in *GRIB2*. If the retrieval is executed remotely at ECMWF, the resulting files can be communicated to the local gateway server via the ``ECtrans`` utility if the parameter ``ECTRANS`` is set to ``1`` and the parameters ``GATEWAY``, ``DESTINATION`` have been set properly during installation. The status of the transfer can be checked with the command ``ecaccess-ectrans-list`` (on the local gateway server). If the script is executed locally the progress of the script can be followed with the usual Linux tools.
    2929
    3030

  • For_developers/Sphinx/source/Documentation/input.rst

    rba99230 rb1674ed  
    3535
    3636       cd flex_extract_vX.X
    37        python3 source/python/install.py --help
    38        python3 source/python/submit.py --help
     37       python3 Source/Python/install.py --help
     38       python3 Source/Python/submit.py --help
    3939   
    4040    In this new version we provide also the wrapping Shell scripts setup_ and run_, which sets the command line parameters, do some checks and execute the corresponing Python scripts ``install.py`` and ``submit.py`` respectivley.

  • For_developers/Sphinx/source/Documentation/output.rst

    rba99230 rb1674ed  
    106106-------------------------------------------------------
    107107
    108 The new disaggregation method for precipitation fields produces two additional precipitation fields for each time step and precipitation type. They serve as sub-grid points in the original time interval. For details of the method see :doc:`disagg` ??????????????????.
     108The new disaggregation method for precipitation fields produces two additional precipitation fields for each time step and precipitation type. They serve as sub-grid points in the original time interval. For details of the method see :doc:`disagg`.
    109109The two additional fields are marked with the ``step`` parameter in the Grib messages and are set to "1" and "2" for sub-grid point 1 and 2 respectively.
    110110The output filenames do not change in this case. 
    … …  
    129129===============
    130130
    131 ``Flex_extract`` works with a number of temporary data files which are usually deleted after a successful data extraction. They are only stored if the ``DEBUG`` mode is switched on (see :doc:`Input/control_params`.
     131``Flex_extract`` works with a number of temporary data files which are usually deleted after a successful data extraction. They are only stored if the ``DEBUG`` mode is switched on (see :doc:`Input/control_params`).
    132132
    133133MARS grib files
    … …  
    220220----------
    221221
    222 There are a number of input files for the ``CONVERT2`` Fortran program named
     222There are a number of input files for the ``calc_etadot`` Fortran program named
    223223
    224224.. code-block:: bash
    … …  
    245245    "22", "total cloud water content (optional)"
    246246
    247 Some of the fields are solely retrieved with specific settings, e.g. the eta-coordinate vertical velocity is not available in ERA-Interim datasets and the total cloud water content is an optional field for ``FLEXPART v10`` and newer. Please see section ????????? for more information.
    248 
    249 The ``CONVERT2`` program saves its results in file ``fort.15`` which typically contains:
    250 
    251 .. csv-table:: Output file of the Fortran program ``CONVERT2``
     247Some of the fields are solely retrieved with specific settings, e.g. the eta-coordinate vertical velocity is not available in ERA-Interim datasets and the total cloud water content is an optional field for ``FLEXPART v10`` and newer.
     248
     249The ``calc_etadot`` program saves its results in file ``fort.15`` which typically contains:
     250
     251.. csv-table:: Output file of the Fortran program ``calc_etadot``
    252252    :header: "Number", "Content"
    253253    :widths: 5, 20
    … …  
    255255    "15", "U and V wind components, eta-coordinate vertical velocity, temperature, surface pressure, specific humidity "
    256256   
    257 More details about the content of ``CONVERT2`` can be found in :doc:`vertco`.   
     257More details about the content of ``calc_etadot`` can be found in :doc:`vertco`.   
    258258   
    259259.. note::
    260260 
    261     The ``fort.4`` file is the namelist file to drive the Fortran program ``CONVERT2``. It is therefore also an input file and is described in ???????????????
     261    The ``fort.4`` file is the namelist file to drive the Fortran program ``calc_etadot``. It is therefore also an input file.
    262262   
    263263    Example of a namelist:

  • For_developers/Sphinx/source/Documentation/overview.rst

    rba99230 rb1674ed  
    3030           - Api
    3131             
    32        - Fortran program - CONVERT2
     32       - Fortran program - calc_etadot
    3333           
    3434           - Package diagram

  • For_developers/Sphinx/source/Documentation/vertco.rst

    rba99230 rb1674ed  
    2323===================================================================================
    2424
    25 The vertical velocity is computed by the FORTRAN90 program ``CONVERT2`` in the ECMWF
    26 vertical coordinate system by applying the equation of continuity and thereby ensuring mass consistent 3D wind fields. A detailed description of ``CONVERT2`` can be found in the
     25The vertical velocity is computed by the FORTRAN90 program ``calc_etadot`` in the ECMWF
     26vertical coordinate system by applying the equation of continuity and thereby ensuring mass consistent 3D wind fields. A detailed description of ``calc_etadot`` can be found in the
    2727documents v20_update_protocol.pdf, V30_update_protocol.pdf and
    28 V40_update_protocol.pdf. The computational demand and accuracy of ``CONVERT2`` is highly
     28V40_update_protocol.pdf. The computational demand and accuracy of ``calc_etadot`` is highly
    2929dependent on the specification of parameters ``GAUSS``, ``RESOL`` and ``SMOOTH``. The
    3030following guidance can be given for choosing the right parameters:
    … …  
    4545======================================================================
    4646
    47 Since November 2008, the parameter 77 (deta/dt) is stored in ``MARS`` on full model levels. ``FLEXTRA``/``FLEXPART`` in its current version requires ``deta/dt`` on model half levels, multiplied by ``dp/deta``. In ``flex_extract``, the program ``CONVERT2`` assumes that this parameter is available if the ``CONTROL`` parameter ``ETA`` is set to 1.
     47Since November 2008, the parameter 77 (deta/dt) is stored in ``MARS`` on full model levels. ``FLEXTRA``/``FLEXPART`` in its current version requires ``deta/dt`` on model half levels, multiplied by ``dp/deta``. In ``flex_extract``, the program ``calc_etadot`` assumes that this parameter is available if the ``CONTROL`` parameter ``ETA`` is set to 1.
    4848
    4949It is recommended to use the pre-calculated parameter 77 by setting ``ETA`` to 1 whenever possible.

  • For_developers/Sphinx/source/Evaluation/testcases.rst

    rba99230 rb1674ed  
    77===========================
    88
    9 
     9  UNDER CONSTRUCTION
    1010
    1111
    … …  
    1414========================
    1515
    16 
     16  UNDER CONSTRUCTION
    1717
    1818

  • For_developers/Sphinx/source/Installation/gateway.rst

    r30f7911 rb1674ed  
    6666   
    6767   $ ecaccess-certificate-create
    68    Please enter your user-id: example_username
     68   Please enter your user-id: <username>
    6969   Your passcode: ***
    7070   
    7171   
    7272   
    73 The easiest way to install all required packages is to use the package management system of your Linux distribution which required admin rights.
     73The easiest way to install all required packages is to use the package management system of your Linux distribution which requires admin rights.
    7474The installation was tested under *GNU/Linux Debian buster* and *Ubuntu 18.04 Bionic Beaver*.
    7575
    … …  
    106106====================
    107107
    108 ``Flex_extract`` will be run on an ECMWF server which makes the setup the same as for the **remote mode**. In the ``setup.sh`` script `[ref] <Documentation/Input/setup.html>`_, select the ``makefile.gfortran`` for the ``CONVERT2`` Fortran program and the ECMWF server (*target*) you would like to use.
     108``Flex_extract`` will be run on an ECMWF server which makes the setup the same as for the **remote mode**. In the ``setup.sh`` script :doc:`../Documentation/Input/setup`, select the ``makefile_ecgate`` or ``makefile_cray`` for the ``calc_etadot`` Fortran program and the ECMWF server (*target*) you would like to use.
    109109The job script, send to the job queue via the ECaccess software, loads the required modules from the module system. For enabling the file transfer via the gateway, you have to set the *ECUID*, *ECGID*, *GATEWAY* and *DESTINATION* parameter values.
     110
     111Open ``setup.sh`` with your preferred editor (e.g., nano) and adapt the values: 
    110112   
     113+----------------------------------------------+----------------------------------------------+   
     114|   Use this for target = **ectrans**          |   Use this for target = **cca** or **ccb**   |
     115+----------------------------------------------+----------------------------------------------+
     116| .. code-block:: bash                         | .. code-block:: bash                         |
     117|                                              |                                              |
     118|   ...                                        |   ...                                        |   
     119|   # -----------------------------------------|   # -----------------------------------------|
     120|   # AVAILABLE COMMANDLINE ARGUMENTS TO SET   |   # AVAILABLE COMMANDLINE ARGUMENTS TO SET   |
     121|   #                                          |   #                                          | 
     122|   # THE USER HAS TO SPECIFY THESE PARAMETER  |   # THE USER HAS TO SPECIFY THESE PARAMETER  |
     123|   #                                          |   #                                          |
     124|   TARGET='ecgate'                            |   TARGET='cca'                               |
     125|   MAKEFILE='makefile_ecgate'                 |   MAKEFILE='makefile_cray'                   | 
     126|   ECUID='<username>'                         |   ECUID='<username>'                         | 
     127|   ECGID='<groupID>'                          |   ECGID='<groupID>'                          |
     128|   GATEWAY='<gatewayname>'                    |   GATEWAY='<gatewayname>'                    |
     129|   DESTINATION='<assoc_name>@genericSftp'     |   DESTINATION='<assoc_name>@genericSftp'     |
     130|   INSTALLDIR=None                            |   INSTALLDIR=None                            |
     131|   JOB_TEMPLATE='job.template'                |   JOB_TEMPLATE='job.template'                |
     132|   CONTROLFILE='CONTROL_EA5'                  |   CONTROLFILE='CONTROL_EA5'                  |
     133|   ...                                        |   ...                                        |   
     134+----------------------------------------------+----------------------------------------------+
    111135
    112 .. code-block:: bash
    113     :caption: 'Example settings for a gateway installation.'
    114     :name: setup.sh
    115    
    116     # -----------------------------------------------------------------
    117     # AVAILABLE COMMANDLINE ARGUMENTS TO SET
    118     #
    119     # THE USER HAS TO SPECIFY THESE PARAMETER
    120     #
    121     TARGET='ecgate'
    122     MAKEFILE='makefile.gfortran'
    123     ECUID='uid'
    124     ECGID='gid'
    125     GATEWAY='localserver.tld'
    126     DESTINATION='association_name@genericSftp'
    127     INSTALLDIR=None
    128     JOB_TEMPLATE='job.template'
    129     CONTROLFILE='CONTROL_EA5'
     136
    130137
    131138
    … …  
    146153   You should get an email with subject "flexcompile" within the next few minutes!
    147154
    148 ``Flex_extract`` uses the email address connectd to the user account on ECMWF servers. The email content should look like this with a "SUCCESS" statement in the last line:
     155``Flex_extract`` automatically uses the email address connected to the user account on ECMWF servers. The email content should look like this with a "SUCCESS" statement in the last line:
    149156
    150157.. code-block:: bash
    151158
    152     gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp phgrreal.f
    153     gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp grphreal.f
    154     gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp ftrafo.f
    155     gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I. -I/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/include -g rwGRIB2.f90
    156     gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I. -I/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/include -g posnam.f
    157     gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I. -I/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/include -g preconvert.f90
    158     gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -g -O3 -o ./CONVERT2 ftrafo.o phgrreal.o grphreal.o rwGRIB2.o posnam.o preconvert.o -L/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/lib -leccodes_f90 -leccodes -ljasper -lpthread -L/usr/local/apps/jasper/1.900.1/LP64/lib -ljasper -lm -L/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -Wl,-rpath,/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -lemos.R64.D64.I32 -L/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -lfftw3   
    159     -rwxr-x---. 1 USER at 353134 May 23 12:27 CONVERT2
     159    gfortran    -O3 -march=native -L/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -Wl,-rpath,/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -leccodes_f90 -leccodes -ljasper -lpthread -L/usr/local/apps/jasper/1.900.1/LP64/lib -ljasper -lm -L/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -Wl,-rpath,/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -lemos.R64.D64.I32 -L/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -lfftw3    -I. -I/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c     ./rwgrib2.f90
     160    gfortran    -O3 -march=native -L/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -Wl,-rpath,/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -leccodes_f90 -leccodes -ljasper -lpthread -L/usr/local/apps/jasper/1.900.1/LP64/lib -ljasper -lm -L/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -Wl,-rpath,/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -lemos.R64.D64.I32 -L/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -lfftw3    -I. -I/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c     ./phgrreal.f90
     161    gfortran    -O3 -march=native -L/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -Wl,-rpath,/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -leccodes_f90 -leccodes -ljasper -lpthread -L/usr/local/apps/jasper/1.900.1/LP64/lib -ljasper -lm -L/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -Wl,-rpath,/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -lemos.R64.D64.I32 -L/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -lfftw3    -I. -I/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c     ./grphreal.f90
     162    gfortran    -O3 -march=native -L/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -Wl,-rpath,/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -leccodes_f90 -leccodes -ljasper -lpthread -L/usr/local/apps/jasper/1.900.1/LP64/lib -ljasper -lm -L/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -Wl,-rpath,/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -lemos.R64.D64.I32 -L/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -lfftw3    -I. -I/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c     ./ftrafo.f90
     163    gfortran    -O3 -march=native -L/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -Wl,-rpath,/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -leccodes_f90 -leccodes -ljasper -lpthread -L/usr/local/apps/jasper/1.900.1/LP64/lib -ljasper -lm -L/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -Wl,-rpath,/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -lemos.R64.D64.I32 -L/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -lfftw3    -I. -I/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c     ./calc_etadot.f90
     164    gfortran    -O3 -march=native -L/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -Wl,-rpath,/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -leccodes_f90 -leccodes -ljasper -lpthread -L/usr/local/apps/jasper/1.900.1/LP64/lib -ljasper -lm -L/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -Wl,-rpath,/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -lemos.R64.D64.I32 -L/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -lfftw3    -I. -I/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c     ./posnam.f90
     165    gfortran   rwgrib2.o calc_etadot.o ftrafo.o grphreal.o posnam.o phgrreal.o -o calc_etadot_fast.out  -O3 -march=native -L/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -Wl,-rpath,/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -leccodes_f90 -leccodes -ljasper -lpthread -L/usr/local/apps/jasper/1.900.1/LP64/lib -ljasper -lm -L/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -Wl,-rpath,/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -lemos.R64.D64.I32 -L/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -lfftw3    -fopenmp
     166    ln -sf calc_etadot_fast.out calc_etadot
     167    lrwxrwxrwx. 1 <username> at 20 Mar  8 14:11 calc_etadot -> calc_etadot_fast.out
    160168    SUCCESS!   
    161    
    162    
    163    
    164    
    165169
    166    
    167    
    168          

  • For_developers/Sphinx/source/Installation/local.rst

    r30f7911 rb1674ed  
    6262| * `eccodes for python`_                        | * `emoslib`_    |
    6363| * `ecmwf-api-client`_ (everything except ERA5) |                 |
    64 | * `cdsapi`_ (just for ERA5)                    |                 |
     64| * `cdsapi`_ (just for ERA5 and member user)    |                 |
    6565+------------------------------------------------+-----------------+
    6666
    … …  
    116116     
    117117CDS API:
    118    Go to 'CDS API registration'_ and register there too. Log in at the `cdsapi`_ website and follow the instructions at section "Install the CDS API key" to save your credentials in a `.cdsapirc` file.
     118   Go to `CDS API registration`_ and register there too. Log in at the `cdsapi`_ website and follow the instructions at section "Install the CDS API key" to save your credentials in a `.cdsapirc` file.
    119119
    120120   
    … …  
    263263
    264264
    265 
    266 
    267265.. _ref-install-local:
    268266
    … …  
    270268==================
    271269
    272 First prepare the Fortran ``makefile`` for your environment and set it in the ``setup.sh`` script. (See section :ref:`ref-convert` for information on the Fortran program.)
    273 ``flex_extract`` comes with two ``makefiles`` prepared for the ``gfortran`` and
    274 the ``ifort`` compiler. The ``gfortran`` version assumes that ``eccodes`` and ``emoslib`` are installed as distribution packages.
    275 
    276  * makefile.local.gfortran
    277  * makefile.local.ifort
     270First prepare the Fortran ``makefile`` for your environment and set it in the ``setup.sh`` script. (See section :ref:`Fortran Makefile <ref-convert>` for more information.)
     271``flex_extract`` comes with two ``makefiles`` prepared for the ``gfortran`` compiler. One for the normal use ``makefile_fast`` and one for debugging ``makefile_debug`` which is usually only resonable for developers.
    278272 
    279 They can be found at ``flex_extract_vX.X/Source/Fortran``, where
    280 ``vX.X`` should be substituted with the current version number.
     273They assume that ``eccodes`` and ``emoslib`` are installed as distribution packages and can be found at ``flex_extract_vX.X/Source/Fortran``, where ``vX.X`` should be substituted with the current version number.
    281274
    282275.. caution::   
    … …  
    291284
    292285   cd flex_extract_vX.X/Source/Fortran
    293    nedit makefile.local.gfortran
     286   nedit makefile_fast
    294287 
    295288Edit the paths to the ``eccodes`` library on your local machine.
    … …  
    319312   
    320313   
    321 The Fortran program called ``CONVERT2`` will be compiled during the
     314The Fortran program called ``calc_etadot`` will be compiled during the
    322315installation process.Therefore the name of the ``makefile`` to be used needs to be given in  ``setup.sh``.
    323316
    … …  
    338331   #
    339332   TARGET='local'
    340    MAKEFILE='makefile.local.gfortran'
     333   MAKEFILE='makefile_fast'
    341334   ECUID=None
    342335   ECGID=None
    … …  
    360353.. code-block:: bash
    361354
    362    # Output of setup.sh   
    363    WARNING: installdir has not been specified
    364    flex_extract will be installed in here by compiling the Fortran source in /raid60/nas/tmc/Anne/Interpolation/flexextract/flex_extract_v7.1/source/fortran
    365    Install flex_extract_v7.1 software at local in directory /raid60/nas/tmc/Anne/Interpolation/flexextract/flex_extract_v7.1
    366 
    367    Using makefile: Makefile.local.gfortran
    368    gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp phgrreal.f
    369    gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp grphreal.f
    370    gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp ftrafo.f
    371    gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -fopenmp -I. -I/usr/local/gcc-4.9.3/grib_api-1.14.3/include -O3 rwGRIB2.f90
    372    gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -fopenmp -I. -I/usr/local/gcc-4.9.3/grib_api-1.14.3/include -O3 posnam.f
    373    gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -fopenmp -I. -I/usr/local/gcc-4.9.3/grib_api-1.14.3/include -O3 preconvert.f90
    374    gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -O3 -O3 -fopenmp -o ./CONVERT2 ftrafo.o phgrreal.o grphreal.o rwGRIB2.o posnam.o preconvert.o -L/usr/local/gcc-4.9.3/grib_api-1.14.3/lib -Bstatic  -lgrib_api_f77 -lgrib_api_f90 -lgrib_api -Bdynamic  -lm  -ljasper -lemosR64
    375 
    376    -rwxrwxr-x. 1 philipa8 tmc 282992 May 23 22:27 ./CONVERT2
    377 
    378 
    379 
    380 
    381 
     355    # Output of setup.sh   
     356        WARNING: installdir has not been specified
     357        flex_extract will be installed in here by compiling the Fortran source in <path-to-flex_extract>/flex_extract_v7.1/Source/Fortran
     358        Install flex_extract_v7.1 software at local in directory <path-to-flex_extract>/flex_extract_v7.1
     359
     360        Using makefile: makefile_fast
     361        gfortran   -O3 -march=native -Bstatic -leccodes_f90 -leccodes -Bdynamic -lm -ljasper -lemosR64 -I. -I/usr/local/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c     ./rwgrib2.f90
     362        gfortran   -O3 -march=native -Bstatic -leccodes_f90 -leccodes -Bdynamic -lm -ljasper -lemosR64 -I. -I/usr/local/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c     ./phgrreal.f90
     363        gfortran   -O3 -march=native -Bstatic -leccodes_f90 -leccodes -Bdynamic -lm -ljasper -lemosR64 -I. -I/usr/local/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c     ./grphreal.f90
     364        gfortran   -O3 -march=native -Bstatic -leccodes_f90 -leccodes -Bdynamic -lm -ljasper -lemosR64 -I. -I/usr/local/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c     ./ftrafo.f90
     365        gfortran   -O3 -march=native -Bstatic -leccodes_f90 -leccodes -Bdynamic -lm -ljasper -lemosR64 -I. -I/usr/local/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c     ./calc_etadot.f90
     366        gfortran   -O3 -march=native -Bstatic -leccodes_f90 -leccodes -Bdynamic -lm -ljasper -lemosR64 -I. -I/usr/local/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c     ./posnam.f90
     367        gfortran  rwgrib2.o calc_etadot.o ftrafo.o grphreal.o posnam.o phgrreal.o -o calc_etadot_fast.out  -O3 -march=native -Bstatic -leccodes_f90 -leccodes -Bdynamic -lm -ljasper -lemosR64 -fopenmp
     368        ln -sf calc_etadot_fast.out calc_etadot
     369
     370        lrwxrwxrwx. 1 <username> tmc 20 15. Mär 13:31 ./calc_etadot -> calc_etadot_fast.out
     371

  • For_developers/Sphinx/source/Installation/remote.rst

    r30f7911 rb1674ed  
    7777   ssh -X <ecuid>@ecaccess.ecmwf.int
    7878
    79 This will lead to the following output on the command line, asking for your
    80 password:
     79This will lead to the following output on the command line, asking for your password:
    8180   
    8281.. code-block:: bash
    … …  
    117116Execute the ``setup.sh`` script from the ``flex_extract``'s root directory.
    118117Before executing it, it is necessary to adapt some parameters from ``setup.sh``
    119 described in :doc:`Documentation/Input/setup`.
     118described in :doc:`../Documentation/Input/setup`.
    120119
    121 Open ``setup.sh`` with your preferred editor (e.g., ``nano'') and adapt the values: 
     120Open ``setup.sh`` with your preferred editor (e.g., nano) and adapt the values: 
    122121   
    123122+----------------------------------------------+----------------------------------------------+   
    124 |   Use this for target = **ectrans**          |   Use this for target = **cca**              |
     123|   Use this for target = **ectrans**          |   Use this for target = **cca** or **ccb**   |
    125124+----------------------------------------------+----------------------------------------------+
    126125| .. code-block:: bash                         | .. code-block:: bash                         |
    … …  
    133132|   #                                          |   #                                          |
    134133|   TARGET='ecgate'                            |   TARGET='cca'                               |
    135 |   MAKEFILE='Makefile.gfortran'               |   MAKEFILE='Makefile.CRAY'                   | 
    136 |   ECUID='uid'                                |   ECUID='uid'                                | 
    137 |   ECGID='gid'                                |   ECGID='gid'                                |
    138 |   GATEWAY=None                               |   GATEWAY=None                               |
    139 |   DESTINATION=None                           |   DESTINATION=None                           |
     134|   MAKEFILE='makefile_ecgate'                 |   MAKEFILE='makefile_cray'                   | 
     135|   ECUID='<username>'                         |   ECUID='<username>'                         | 
     136|   ECGID='<groupID>'                          |   ECGID='<groupID>'                          |
     137|   GATEWAY='<gatewayname>'                    |   GATEWAY='<gatewayname>'                    |
     138|   DESTINATION='<username>@genericSftp'       |   DESTINATION='<username>@genericSftp'       |
    140139|   INSTALLDIR=None                            |   INSTALLDIR=''                              |
    141140|   JOB_TEMPLATE='job.template'                |   JOB_TEMPLATE='job.template'                |
    … …  
    169168
    170169   
    171 ``Flex_extract`` uses the email address connectd to the user account on ECMWF servers. The email content should look like this with a "SUCCESS" statement in the last line:
     170``Flex_extract`` automatically uses the email address connected to the user account on ECMWF servers. The email content should look like this with a "SUCCESS" statement in the last line:
    172171
    173172.. code-block:: bash
    174173
    175     gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp phgrreal.f
    176     gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp grphreal.f
    177     gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp ftrafo.f
    178     gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I. -I/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/include -g rwGRIB2.f90
    179     gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I. -I/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/include -g posnam.f
    180     gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I. -I/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/include -g preconvert.f90
    181     gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -g -O3 -o ./CONVERT2 ftrafo.o phgrreal.o grphreal.o rwGRIB2.o posnam.o preconvert.o -L/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/lib -leccodes_f90 -leccodes -ljasper -lpthread -L/usr/local/apps/jasper/1.900.1/LP64/lib -ljasper -lm -L/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -Wl,-rpath,/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -lemos.R64.D64.I32 -L/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -lfftw3   
    182     -rwxr-x---. 1 USER at 353134 May 23 12:27 CONVERT2
    183     SUCCESS!   
    184 
    185 
     174    gfortran    -O3 -march=native -L/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -Wl,-rpath,/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -leccodes_f90 -leccodes -ljasper -lpthread -L/usr/local/apps/jasper/1.900.1/LP64/lib -ljasper -lm -L/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -Wl,-rpath,/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -lemos.R64.D64.I32 -L/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -lfftw3    -I. -I/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c     ./rwgrib2.f90
     175    gfortran    -O3 -march=native -L/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -Wl,-rpath,/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -leccodes_f90 -leccodes -ljasper -lpthread -L/usr/local/apps/jasper/1.900.1/LP64/lib -ljasper -lm -L/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -Wl,-rpath,/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -lemos.R64.D64.I32 -L/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -lfftw3    -I. -I/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c     ./phgrreal.f90
     176    gfortran    -O3 -march=native -L/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -Wl,-rpath,/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -leccodes_f90 -leccodes -ljasper -lpthread -L/usr/local/apps/jasper/1.900.1/LP64/lib -ljasper -lm -L/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -Wl,-rpath,/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -lemos.R64.D64.I32 -L/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -lfftw3    -I. -I/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c     ./grphreal.f90
     177    gfortran    -O3 -march=native -L/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -Wl,-rpath,/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -leccodes_f90 -leccodes -ljasper -lpthread -L/usr/local/apps/jasper/1.900.1/LP64/lib -ljasper -lm -L/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -Wl,-rpath,/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -lemos.R64.D64.I32 -L/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -lfftw3    -I. -I/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c     ./ftrafo.f90
     178    gfortran    -O3 -march=native -L/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -Wl,-rpath,/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -leccodes_f90 -leccodes -ljasper -lpthread -L/usr/local/apps/jasper/1.900.1/LP64/lib -ljasper -lm -L/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -Wl,-rpath,/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -lemos.R64.D64.I32 -L/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -lfftw3    -I. -I/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c     ./calc_etadot.f90
     179    gfortran    -O3 -march=native -L/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -Wl,-rpath,/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -leccodes_f90 -leccodes -ljasper -lpthread -L/usr/local/apps/jasper/1.900.1/LP64/lib -ljasper -lm -L/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -Wl,-rpath,/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -lemos.R64.D64.I32 -L/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -lfftw3    -I. -I/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c     ./posnam.f90
     180    gfortran   rwgrib2.o calc_etadot.o ftrafo.o grphreal.o posnam.o phgrreal.o -o calc_etadot_fast.out  -O3 -march=native -L/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -Wl,-rpath,/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -leccodes_f90 -leccodes -ljasper -lpthread -L/usr/local/apps/jasper/1.900.1/LP64/lib -ljasper -lm -L/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -Wl,-rpath,/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -lemos.R64.D64.I32 -L/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -lfftw3    -fopenmp
     181    ln -sf calc_etadot_fast.out calc_etadot
     182    lrwxrwxrwx. 1 <username> at 20 Mar  8 14:11 calc_etadot -> calc_etadot_fast.out
     183    SUCCESS!
    186184   
    187 
    188    

  • For_developers/Sphinx/source/Support/known_bugs_issues.rst

    rba99230 rb1674ed  
    11Known Bugs and Issues
    2 =====================
     2*********************
    33
    4  
     4Release v7.1
     5============
     6
     7
     8CDS API and ERA5 data
     9---------------------
     10
     11  See ticket `#230 <https://www.flexpart.eu/ticket/230>`_  on flexpart.eu for information.
     12
     13
     14Installation problems with ``GATEWAY`` and ``DESTINATION`` parameters
     15---------------------------------------------------------------------
     16
     17  See ticket `#263 <https://www.flexpart.eu/ticket/263>`_  on flexpart.eu for information.
     18
     19
     20Installation problems with the Fortran program ``calc_etadot``
     21--------------------------------------------------------------
     22
     23   See ticket `#264 <https://www.flexpart.eu/ticket/264>`_  on flexpart.eu for information.
     24
     25

  • For_developers/Sphinx/source/changelog.rst

    rba99230 rb1674ed  
    99.. _ECMWF Web API: https://confluence.ecmwf.int/display/WEBAPI/ECMWF+Web+API+Home
    1010.. _new algorithm: https://www.geosci-model-dev.net/11/2503/2018/
     11
     12
     13.. _ref-v711:
     14
     15Release v7.1.1
     16--------------
     17
     18Bug fixes
     19#########
     20   * corrected naming of makefiles and Fortran program executable in online documentation
     21   * adapted checking for ``GATEWAY`` and ``DESTINATION`` parameter
     22   * fixed makefiles to use f90 Fortran files for ECMWF server
     23
    1124   
    1225.. _ref-v71:
    … …  
    3346   * completely revised/refactored python section
    3447   * restructured program directories
     48   * Fortran program name changed from ``CONVERT2`` to ``calc_etadot``
    3549
    3650.. _ref-v704:

  • For_developers/Sphinx/source/conf.py

    r30f7911 rb1674ed  
    2828
    2929project = u'flex_extract'
    30 copyright = u'2019, Anne Philipp and Leopold Haimberger'
     30copyright = u'2020, Anne Philipp and Leopold Haimberger'
    3131author = u'Anne Philipp and Leopold Haimberger'
    3232

  • For_developers/Sphinx/source/dev_guide.rst

    rba99230 rb1674ed  
    44    UNDER CONSTRUCTION
    55   
     6.. note::
     7
     8  This section still needs to be done.
    69   
    710.. repository (how /who manages the code, where to get)

  • For_developers/Sphinx/source/documentation.rst

    rba99230 rb1674ed  
    1313    Vertical Coordinate (Under construction)
    1414      - Methods (GAUSS, ETA, OMEGA)
    15       - CONVERT
     15      - calc_etadot
    1616   
    1717    Auto Generated Documentation

  • For_developers/Sphinx/source/evaluation.rst

    rd9abaac rb1674ed  
    44    UNDER CONSTRUCTION
    55   
    6  
     6.. note::
     7
     8  This section in the online documentation still needs to be done.
     9  Currently, evaluation methods and information can be found in the `flex_extract discussion paper <https://www.geosci-model-dev-discuss.net/gmd-2019-358/>`_ of the Geoscientific Model Development journal.
    710 
    811 

  • For_developers/Sphinx/source/index.rst

    reca358d rb1674ed  
    2727        <td style="width:30%;height:150px;">
    2828          <div style="width:100%;height:60px;margin-left:20%;position:relative;">
    29             <img style="position:absolute;height:90%;vertical-align:bottom;" src="">
     29            <img style="position:absolute;height:90%;vertical-align:bottom;" src="_static/ecmwf_icon.png">
    3030          </div>
    3131          <h2><a href="ecmwf_data.html">ECMWF data</a></h2>
    … …  
    6262        <td style="width:30%;height:150px;">
    6363          <div style="width:100%;height:60px;margin-left:20%;position:relative;">
    64             <img style="position:absolute;height:90%;vertical-align:bottom;" src="">
     64            <img style="position:absolute;height:90%;vertical-align:bottom;" src="_static/check_icon.png">
    6565          </div>
    6666          <h2><a href="evaluation.html">Evaluation</a></h2>
    … …  
    7373        <td style="width:30%;height:150px;">
    7474          <div style="width:100%;height:60px;margin-left:20%;position:relative;">
    75             <img style="position:absolute;height:90%;vertical-align:bottom;" src="_static/guide_icon.png">
     75            <img style="position:absolute;height:90%;vertical-align:bottom;" src="_static/dev_icon.png">
    7676          </div>
    7777          <h2><a href="dev_guide.html">Developer</a></h2>
    … …  
    9494          <h2><a href="changelog.html">Changelog</a></h2>
    9595          <ul>
     96            <li><a href="changelog.html#ref-v711">v7.1.1</a></li>
    9697            <li><a href="changelog.html#ref-v71">v7.1</a></li>
    9798            <li><a href="changelog.html#ref-v704">v7.0.4</a></li>
    9899            <li><a href="changelog.html#ref-v703">v7.0.3</a></li>
    99             <li><a href="changelog.html#ref-v60">v6</a></li>
    100100          </ul>
    101101        </td>       

  • For_developers/Sphinx/source/installation.rst

    r30f7911 rb1674ed  
    124124    .. code-block:: bash
    125125
    126        $ git clone --single-branch --branch master https://www.flexpart.eu/gitmob/flexpart
     126       $ git clone --single-branch --branch master https://www.flexpart.eu/gitmob/flex_extract
    127127
    128128
    … …  
    223223--------------------
    224224
    225 To check whether the Fortran program ``CONVERT2`` has been compiled and runs properly, it can be applied to a prepared minimal dataset.
     225To check whether the Fortran program ``calc_etadot`` has been compiled and runs properly, it can be applied to a prepared minimal dataset.
    226226
    227227For this, go from the ``flex_extract`` root directory to the ``Testing/Installation/Convert/`` directory and execute the Fortran program.
    … …  
    234234   cd Testing/Installation/Convert
    235235   # execute the Fortran progam without arguments
    236    ../../../Source/Fortran/CONVERT2
     236   ../../../Source/Fortran/calc_etadot
    237237
    238238The installation was successfull if you obtain on standard output:
    239239
    240240.. code-block:: bash
    241 
    242     readspectral:            1  records read
    243     readlatlon:            8  records read
    244    STATISTICS:  98842.4598 98709.7359  5120.5385
    245     readlatlon:            4  records read
    246     readlatlon:            4  records read
    247     readlatlon:            4  records read
    248    SUCCESSFULLY FINISHED CONVERT_PRE: CONGRATULATIONS
     241   
     242    STATISTICS:          98842.4598  98709.7359   5120.5385
     243    STOP SUCCESSFULLY FINISHED calc_etadot: CONGRATULATIONS
     244
    249245
    250246Now go back to the root directory:

  • For_developers/Sphinx/source/quick_start.rst

    r30f7911 rb1674ed  
    3131The ``Jobscripts`` directory is used to store the Korn shell job scripts generated by a ``flex_extract`` run in the **Remote** or **Gateway** mode. They are used to submit the setup information to the ECMWF server and start the jobs in ECMWF's batch mode. The typical user must not touch these files. They will be generated from template files which are stored in the ``Templates`` directory under ``flex_extract_vX.X``. Usually there will be a ``compilejob.ksh`` and a ``job.ksh`` script which are explained in the section :doc:`Documentation/input`. In the rare case of operational data extraction there will be a ``joboper.ksh`` which is designed to get the time parameters from environment variables at the ECMWF servers.
    3232
    33 The ``Controls`` directory contains a number of example ``CONTROL`` files. These``CONTROL`` files represent the current range of possible dataset retrievals with ``flex_extract``. Some parameters in the ``CONTROL`` files can be adapted and some others should not be changed. In this :doc:`quick_start` guide we explain how an extraction with ``flex_extract`` can be started in the different :doc:`Documentation/Overview/app_modes` and point out some specifics of each dataset and ``CONTROL`` file.
     33The ``Controls`` directory contains a number of example ``CONTROL`` files. These ``CONTROL`` files represent the current range of possible dataset retrievals with ``flex_extract``. Some parameters in the ``CONTROL`` files can be adapted and some others should not be changed. In this :doc:`quick_start` guide we explain how an extraction with ``flex_extract`` can be started in the different :doc:`Documentation/Overview/app_modes` and point out some specifics of each dataset and ``CONTROL`` file.
    3434
    3535Directly under ``Run`` you find the files ``run.sh`` and ``run_local.sh`` and according to your selected :doc:`Documentation/Overview/app_modes` there might also be a file named ``ECMWF_ENV`` for the user credentials to quickly and automatically access ECMWF servers. 
    … …  
    150150    │   ├── CE000908* (resulting files)
    151151
    152 If the job was submitted to the HPC ( ``queue=cca`` ) you may login to the HPC and look into the directory ``/scratch/ms/ECGID/ECUID/.ecaccess_do_not_remove`` for job logs. The working directories are deleted after job failure and thus normally cannot be accessed.
     152If the job was submitted to the HPC ( ``queue=cca`` or  ``queue=ccb`` ) you may login to the HPC and look into the directory ``/scratch/ms/ECGID/ECUID/.ecaccess_do_not_remove`` for job logs. The working directories are deleted after job failure and thus normally cannot be accessed.
    153153
    154154To check if the resulting files are still transferred to local gateway server you can use the command ``ecaccess-ectrans-list`` or check the destination path for resulting files on your local gateway server.
    … …  
    231231
    232232   
    233 !!!!!!!! ADD HERE ANOTHER SCREENSHOT OF THE PARAMETER SELECTION AREA ( HAS TO BE DONE AT HOME )
    234233
    235234Additionally, you can find a lot of helpful links to dataset documentations, direct links to specific dataset web catalogues or further general information at the `link collection <Ecmwf/ec-links.html>`_ in the ECMWF data section.   
    … …  
    276275.. figure:: _files/dataset_cmp_table.png
    277276
    278    DO THIS TABLE AGAIN BY HAND!
     277
    279278                   
    280279A common problem for beginners in retrieving ECMWF datasets is the mismatch in the definition of these parameters. For example, if you would like to retrieve operational data before ``June 25th 2013`` and set the maximum level to ``137`` you will get an error because this number of levels was first introduced at this effective day. So, be cautious in the combination of space and time resolution as well as the field types which are not available all the time.
    … …  
    332331
    333332This re-analysis dataset will exceed its end of production at 31st August 2019!
    334 It is then available from 1st January 1979 to 31st August 2019. The ``etadot`` is not available in this dataset. Therefore ``flex_extract`` must select the ``GAUSS`` parameter to retrieve the divergence field in addition. The vertical velocity is the calculated with the continuity equation in the Fortran program ``CONVERT2``. Since the analysis fields are only available for every 6th hour, the dataset can be made 3 hourly by adding forecast fields in between. No ensemble members are available.
    335 
    336 
    337 .. todo::
    338 
    339     @LEO: please check the complete description and functionality of the CONTROL FILEs
     333It is then available from 1st January 1979 to 31st August 2019. The ``etadot`` is not available in this dataset. Therefore ``flex_extract`` must select the ``GAUSS`` parameter to retrieve the divergence field in addition. The vertical velocity is the calculated with the continuity equation in the Fortran program ``calc_etadot``. Since the analysis fields are only available for every 6th hour, the dataset can be made 3 hourly by adding forecast fields in between. No ensemble members are available.
     334
     335
    340336   
    341337Operational data
    … …  
    357353These files defines the minimum number of parameters necessary to retrieve a daily subset. The setup of field types is optimal and should only be changed if the user understands what he does. The grid, domain and temporal resolution can be changed according to availability.     
    358354   
    359 .. todo::
    360    
    361     @LEO - explain the setup with 4V fields! Who can extract it, when would this be useful?
    362355
    363356
    … …  
    404397        CONTROL_OD.ENFO.PF.ens
    405398
    406 .. todo::
    407 
    408     @LEO: Please tell me why perturbed forecast are possible? Is it because of some user rights? I have no opportunity of retrieve PF data.       
    409    
     399
    410400   
    411401   
Note: See TracChangeset for help on using the changeset viewer.
hosted by ZAMG