Changeset 30f7911 in flex_extract.git for For_developers


Ignore:
Timestamp:
Aug 20, 2019, 12:49:39 PM (5 years ago)
Author:
Anne Philipp <anne.philipp@…>
Branches:
master, ctbto, dev
Children:
2255082
Parents:
0690a6c
Message:

reviewed installation section of online documentation; minor corrections

Location:
For_developers/Sphinx/source
Files:
3 added
5 edited

Legend:

Unmodified
Added
Removed

  • For_developers/Sphinx/source/Documentation/Input/ecmwf_env.rst

    rba99230 r30f7911  
    33****************************************
    44
    5 This file contains the user credentials for working on ECMWF servers and transfering files between the ECMWF servers and the local gateway server. It is located in the ``flex_extract_vX.X/run`` directory and will be created in the installation process for the application modes **remote** and **gateway**.
     5This file contains the user credentials for working on ECMWF servers and transferring files between the ECMWF servers and the local gateway server. It is located in the ``flex_extract_vX.X/run`` directory and will be created in the installation process for the application modes **remote** and **gateway**.
    66
    77This file is based on the template ``ECMWF_ENV.template`` which is located in the ``templates`` directory.

  • For_developers/Sphinx/source/conf.py

    rba99230 r30f7911  
    2020
    2121sys.path.insert(0, os.path.abspath('_static/python'))
     22sys.setrecursionlimit(1500)
    2223
    2324#fortran_src = [f for f in os.listdir('../../../source/fortran') if '.f90' in f or '.f' in f]

  • For_developers/Sphinx/source/index.rst

    rba99230 r30f7911  
    129129
    130130.. toctree::
    131    :maxdepth: 3
     131   :maxdepth: 4
    132132   :caption: Table of Contents:
    133133

  • For_developers/Sphinx/source/installation.rst

    rba99230 r30f7911  
    99    :hidden:
    1010    :maxdepth: 2
     11   
    1112     
    1213.. _Python 3: https://docs.python.org/3/
    … …  
    3738
    3839
    39 ``flex_extract`` is a software package which contains a mix of Python and Shell scripts as well as a Fortran program. These components rely on a couple of third party libraries which need to be prepared first before starting the installation process. For now, the software is only tested for a Linux/Unix environment. Feel free to try it out on other platforms.
    40 
    41 Start with the decision of which `user group <Ecmwf/access>`_ you belong to and follow the instructions at :ref:`ref-registration` to get an account at ECMWF. Considering your user group and the method of applying ``flex_extract`` there appear to be 4 application modes: 
    42 
    43 - Remote (for member state users only) :ref:`[installation]<ref-remote-mode>`
    44 - Gateway (for member state users only) :ref:`[installation]<ref-gateway-mode>`
    45 - Local member :ref:`[installation]<ref-local-mode>`
    46 - Local public :ref:`[installation]<ref-local-mode>`
    47 
    48 More information can be found in :doc:`Documentation/Overview/app_modes`.
     40The ``flex_extract`` software package contains python and shell scripts as well as a Fortran program. These components rely on several libraries which need to be available before starting the installation process. Currently, the software is only tested for a GNU/Linux environment. Feel free to try it out on other platforms.
     41
     42At first, go to the `user group <Ecmwf/access.html>`_ section and decide which group you belong to and follow the instructions at :ref:`ref-registration` to obtain an account at ECMWF (if you don't have it already). Depending on the user group and the way to access the ECWMF MARS archive, there are four possible :doc:`Documentation/Overview/app_modes`: 
     43
     44- Remote (member-state users only) :ref:`[installation]<ref-remote-mode>`
     45- Gateway (member-state users only) :ref:`[installation]<ref-gateway-mode>`
     46- Local, member-state user :ref:`[installation]<ref-local-mode>`
     47- Local, public user :ref:`[installation]<ref-local-mode>`
     48
     49Please follow the link :doc:`Documentation/Overview/app_modes` to get a description of the modes.
    4950
    5051.. note::
    5152
    5253   If you encounter any problems in the installation process, you can ask for :doc:`support`.
    53 
    54 
    55 
    56 
     54   
     55   
     56   
     57   
    5758
    5859.. _ref-registration:
    … …  
    6162=====================
    6263
    63 Decide which user group you belong to and follow the instructions for registration:
    64 
    65 **Member state user**:
    66     To get a member-state user account, users have to be a resident of a `member state`_. In that case, you can contact your `Computing Representative`_ for granting access. If you would like to use the local application mode to retrieve **ERA5** data you'd have to register at the `Copernicus Climate Data Store <https://cds.climate.copernicus.eu/user/register>`_ also.
    67 
    68 **Public user**:
    69     To be able to download public datasets with ``flex_extract`` such as **ERA-Interim** and **CERA-20C** (**ERA5** is not supported via ECMWF Web API anymore), the public user has to create an account at ECMWF.
    70     Use the registration at the ECMWF website by filling out this `registration form`.
    71    
    72     .. note::
    73 
    74         In the future retrievement of *ERA5* will be possible via the CDS API for public users also. Then a registration at the `Copernicus Climate Data Store <https://cds.climate.copernicus.eu/user/register>`_ is needed in addition.
    75    
    76    
    77    
    78    
     64The registration depends on the :doc:`Documentation/Overview/app_modes` and in case of the local mode also on the data set you'd like to retrieve. The following table gives an overview where you need to register.
     65
     66+--------------+------------------------------------+--------------+
     67|              |  Member-state user                 | Public user  |
     68|              +---------+----------+---------------+--------------+
     69|    Data sets |Remote   |Gateway   |Local          | Local        |
     70+--------------+---------+----------+---------------+--------------+
     71| Operational  | 1       | 1        | 1,2           | -            |
     72+--------------+---------+----------+---------------+--------------+
     73| ERA-Interim  | 1       | 1        | 1,2           | 2            |
     74+--------------+---------+----------+---------------+--------------+
     75| CERA-20C     | 1       | 1        | 1,2           | 2            |
     76+--------------+---------+----------+---------------+--------------+
     77| ERA5         | 1       | 1        | 3             | -            |
     78+--------------+---------+----------+---------------+--------------+
     79
     80
     81
     82Registration options:
     83
     84    1.)  Access through a member-state user account granted by the `Computing Representative`_. The credentials have to be provided during installation.
     85   
     86    2.)  Access through the ECMWF Web API. Need to sign in at `ECMWF Web API <https://confluence.ecmwf.int/display/WEBAPI/ECMWF+Web+API+Home>`_ and configure the ECMWF key as described. Member-state users can sign in with their credentials from the `Computing Representative`_ and public users have to fill out the `registration form`_ to get an account.
     87   
     88    3.) Access through the `CDS API <https://cds.climate.copernicus.eu/api-how-to>`_. Extra registration for member-state users is required at `Copernicus Climate Data Store <https://cds.climate.copernicus.eu/user/register>`_ including the configurations of the CDS key as described. This mode is currently not available for public users. See the note at `user group <Ecmwf/access.html>`_ for information.
     89
    7990   
    8091   
    8192.. _ref-licence:
    8293   
    83 Agree on licences for public datasets
     94Licence agreement for public datasets
    8495=====================================
    8596
    86 Each public dataset which is intended to be downloaded by ``flex_extract`` has its own licence which has to be accepted, regardless of the user group.
     97Each ECMWF public dataset has its own licence which has to be accepted, regardless of the user group.
    8798
    8899For the *ERA-Interim* and *CERA-20C* datasets this can be done at the ECMWF website `Available ECMWF Public Datasets <https://confluence.ecmwf.int/display/WEBAPI/Available+ECMWF+Public+Datasets>`_. Log in and follow the licence links on the right side for each dataset and accept it.
    89100   
    90 For the *ERA5* dataset this has to be done at the `Climate Data Store (CDS) website <https://cds.climate.copernicus.eu/cdsapp#!/search?type=dataset>`_. Log in with your credentials and then select on the left panel the product type "Reanalysis" for finding *ERA5* datasets. Then follow the link of a title with *ERA5* (anyone) to the full dataset record, click on tab "Download data" and scroll down. There is a section "Terms of use" where you have to click the :underline:`Accept terms` button.   
    91 
     101For the *ERA5* dataset this has to be done at the `Climate Data Store (CDS) website <https://cds.climate.copernicus.eu/cdsapp#!/search?type=dataset>`_. Log in and select, on the left panel, product type "Reanalysis" for finding *ERA5* datasets. Then follow any link with *ERA5* to the full dataset record, click on tab "Download data" and scroll down. There is a section "Terms of use" where you have to click the :underline:`Accept terms` button.
    92102   
    93103
    … …  
    100110=========================
    101111
    102 There are 2 options to download ``flex_extract``:
     112There are two options to download ``flex_extract``:
    103113
    104114tar ball
    105     You can download the latest prepared release tar ball from the `download section`_
    106     of our ``FLEXPART`` community website and then untar the file. Substitute
    107     the **<ID>** in the ``wget`` command with the ID-number of the ``flex_extract``
    108     release tar ball in the list of downloads at the community website.
    109    
    110    
     115    You can download a tar ball with the latest release from the `flex_extract page <https://www.flexpart.eu/wiki/FpInputMetEcmwf>`_ from our ``FLEXPART`` community website and then untar the file.
     116 
    111117    .. code-block:: bash
    112118       
    113        wget https://www.flexpart.eu/downloads/<ID>
    114119       tar -xvf <flex_extract_vX.X.tar>
    115120
    116 git repo   
    117     Or you can clone the current release version from our git repository master branch.
     121git repo 
     122    Alternatively, if you have ``git`` installed on your machine, and if you are interested to keep the code in a version control system, you may clone the latest version from our git repository master branch. 
    118123
    119124    .. code-block:: bash
    120125
    121        $ git clone https://www.flexpart.eu/gitmob/flexpart
     126       $ git clone --single-branch --branch master https://www.flexpart.eu/gitmob/flexpart
    122127
    123128
    … …  
    127132.. _ref-requirements:
    128133 
    129 Environment requirements
    130 ========================
    131 
    132 This is a list of the general environment requirements for ``flex_extract``.
    133 What is required exactly for each application mode will be described in the specifc installation section.
    134 
    135    
    136 To run the python part of ``flex_extract`` a `Python 3`_ environment is needed.
    137 We tested ``flex_extract`` with a normal Linux Python package distribution and Anaconda Python.
    138 Except for `Python3`_ all are python packages which can be installed via ``pip``.
    139 
    140 * `Python3`_ or `Anaconda Python3`_
    141 * `numpy`_
    142 * `ecmwf-api-client`_ (Web Interface to ECMWF servers for datasets except ERA5)
    143 * `cdsapi`_ (Web Interface to `C3S <https://climate.copernicus.eu/>`_ servers for ERA5)
    144 * `genshi`_
    145 * `eccodes for python`_ 
    146  
    147 For the Fortran part of ``flex_extract`` we need the following distribution packages:
    148  
    149 * `gfortran`_
    150 * `fftw3`_
    151 * `eccodes`_
    152 * `emoslib`_
     134Dependencies
     135============
     136
     137The software required for running ``flex_extract`` depends on the :doc:`Documentation/Overview/app_modes` and therefore is described in the respective specific installation sections.
     138   
     139Generally speaking, ``flex_extract`` requires `Python 3`_ and Fortran together with certain modules / libraries.
     140We tested ``flex_extract`` with the python3 package from the the GNU/Linux distribution and Anaconda Python. The required python3 modules should prefarably be installed as distribution packages, or alternatively using Python's own package manager ``pip`` (this may mess up some aspects of your python installation, especially if you use ``pip`` as root. Think about using virtual environments.).
     141
     142Before installing the system packages check the availability with ``dpkg -s <package-name> |  grep Status`` or ``rpm -q <package_name>``, depending on your system. For example:
     143
     144.. code-block:: sh
     145
     146   $ dpkg -s libeccodes-dev |  grep Status
     147   # or
     148   $ rpm -q libeccodes-dev
    153149
    154150
    … …  
    162158================================
    163159
    164 The actual installation of ``flex_extract`` will be done by executing a `Shell` script called ``setup.sh``.
    165 It defines some parameters and calls a Python script by giving the parameters as command line arguments.
    166 More information on the script and its parameters can be found at :doc:`Documentation/Input/setup`.
     160The actual installation of ``flex_extract`` is done by executing a shell script called ``setup.sh``.
     161It defines some parameters and calls a Python script passing the parameters as command line arguments. For details, see :doc:`Documentation/Input/setup`.
    167162
    168163For each application mode installation section we describe the requirements for the explicit
    … …  
    171166
    172167
    173 
    174 
    175 
    176 
    177 .. _ref-remote-mode:
    178 
    179 Remote mode
    180 -----------
    181 
    182 .. _ref-req-remote:
    183  
    184 Remote environment requirements
    185 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    186 
    187 The environment for ``flex_extract`` on ECMWF servers has to provide these
    188 software packages and libraries:
    189    
    190 +---------------------------+-----------------+
    191 |  Python part              | Fortran part    |
    192 +---------------------------+-----------------+
    193 | * `Python3`_              | * `gfortran`_   |
    194 | * `numpy`_                | * `fftw3`_      |
    195 | * `genshi`_               | * `eccodes`_    |
    196 | * `eccodes for python`_   | * `emoslib`_    |
    197 +---------------------------+-----------------+
    198 
    199 
    200 .. _ref-prep-remote:
    201 
    202 Prepare remote environment
    203 ^^^^^^^^^^^^^^^^^^^^^^^^^^
    204  
    205 The environment on the ECMWF server (such as *ecgate* or *cca*) is already
    206 prepared. ECMWF server provide all libraries via a module system which is
    207 going to be selected by ``flex_extract`` automatically.
    208 
    209 
    210 .. _ref-install-remote:
    211 
    212 Remote installation
    213 ^^^^^^^^^^^^^^^^^^^
    214 
    215 First, log in on one of the ECMWF Linux server, such as *ecgate* or *cca/ccb*.
    216 Substitute *<ecuid>* with your ECMWF user name:
    217 
    218 .. code-block:: bash
    219    
    220    ssh -X <ecuid>@ecaccess.ecmwf.int
    221 
    222 This will lead to the following output on the command line, asking for your
    223 password:
    224    
    225 .. code-block:: bash
    226 
    227    Authorized access only.
    228 
    229    ***************************************************************
    230       For further information, read the ECaccess documentation at:
    231 
    232       https://software.ecmwf.int/wiki/display/ECAC/ECaccess+Home
    233 
    234       You can also use ECaccess to load & download files from your
    235       EChome, ECscratch or ECfs directories using the ECaccess FTP
    236       server:
    237 
    238       ftp://uid@ecaccess.ecmwf.int/
    239 
    240       Please note you must use your UID and ActivID code to login!
    241    ***************************************************************
    242 
    243    <ecuid>@131.130.157.5's password: ***
    244    Select hostname (ecgate, cca, ccb) [ecgate]: ecgate
    245 
    246    [<ecuid>@ecgb11 ~]$
    247    
    248    
    249 Then, copy the ``flex_extract`` tar ball (from section :ref:`ref-download`)
    250 to the ``$HOME`` directory of the ECMWF Linux server via ``scp``.
    251 Substitute the *<localuser>* and *<server.edu>* placeholders with your credentials.
    252 Untar the file and change into the ``flex_extract`` root directory.
    253 
    254 .. code-block:: bash
    255 
    256    scp <localuser>@<server.edu>:/path/to/tarfile/flex_extract_vX.X.tar.gz  $HOME/
    257    cd $HOME
    258    tar xvf flex_extract_vX.X.tar.gz
    259    cd flex_extract_vX.X
    260    
    261 
    262 On these ECMWF servers, it is not necessary to prepare the environment or the
    263 ``Makefile`` for the Fortran program (``CONVERT2``) as described above.
    264 All third party libraries are available from a module system. The ``Makefile``
    265 is optimized for ECMWF servers and the compilation
    266 script ``compilejob.ksh``, which will be submitted by ``flex_extract`` to the
    267 batch job queue at ECMWF, does load all relevant modules from the ECMWF's module system.
    268 
    269 So there is just the need to execute the ``setup.sh`` script from the
    270 ``flex_extract`` root directory for installation.
    271 Before executing it, it is necessary to adapt some parameters from ``setup.sh``
    272 described in :doc:`Documentation/Input/setup`.
    273 
    274 Open ``setup.sh`` with your editor and adapt the values: 
    275 
    276 +----------------------------------------------+----------------------------------------------+   
    277 |   Take this for target = **ectrans**         |  Take this for target = **cca**              |
    278 +----------------------------------------------+----------------------------------------------+
    279 | .. code-block:: bash                         | .. code-block:: bash                         |
    280 |                                              |                                              |
    281 |   ...                                        |   ...                                        |   
    282 |   # -----------------------------------------|   # -----------------------------------------|
    283 |   # AVAILABLE COMMANDLINE ARGUMENTS TO SET   |   # AVAILABLE COMMANDLINE ARGUMENTS TO SET   |
    284 |   #                                          |   #                                          | 
    285 |   # THE USER HAS TO SPECIFY THESE PARAMETER  |   # THE USER HAS TO SPECIFY THESE PARAMETER  |
    286 |   #                                          |   #                                          |
    287 |   TARGET='ecgate'                            |   TARGET='cca'                               |
    288 |   MAKEFILE='Makefile.gfortran'               |   MAKEFILE='Makefile.CRAY'                   | 
    289 |   ECUID='uid'                                |   ECUID='uid'                                | 
    290 |   ECGID='gid'                                |   ECGID='gid'                                |
    291 |   GATEWAY=None                               |   GATEWAY=None                               |
    292 |   DESTINATION=None                           |   DESTINATION=None                           |
    293 |   INSTALLDIR=None                            |   INSTALLDIR=''                              |
    294 |   JOB_TEMPLATE='job.template'                |   JOB_TEMPLATE='job.template'                |
    295 |   CONTROLFILE='CONTROL_EA5'                  |   CONTROLFILE='CONTROL_EA5'                  |
    296 |   ...                                        |   ...                                        |   
    297 +----------------------------------------------+----------------------------------------------+
    298 
    299 :underline:`Please substitute the values of ECUID and ECGID
    300 with your own credentials and settings.`
    301 
    302 .. note::
    303 
    304    If a local gateway server is available the transfer of files could be done
    305    via the ``ECaccess`` commands. Therefore a valid *GATEWAY* and *DESTINATION*
    306    have to be present and should be set in the ``setup.sh`` file.
    307 
    308 
    309 Afterwards, type:
    310 
    311 .. code-block:: bash
    312 
    313    module load python3   
    314    ./setup.sh
    315    
    316 to start the installation. You should see the following output at the command line.
    317    
    318    
    319 .. code-block:: bash
    320 
    321    # Output of setup.sh
    322    Create tarball ...
    323    Job compilation script has been submitted to ecgate for installation in ${HOME}/flex_extract_vX.X
    324    You should get an email with subject "flexcompile" within the next few minutes!
    325 
    326    
    327 The email content should look like this with a "SUCCESS" statement in the last line:
    328 
    329 .. code-block:: bash
    330 
    331     gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp phgrreal.f
    332     gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp grphreal.f
    333     gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp ftrafo.f
    334     gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I. -I/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/include -g rwGRIB2.f90
    335     gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I. -I/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/include -g posnam.f
    336     gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I. -I/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/include -g preconvert.f90
    337     gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -g -O3 -o ./CONVERT2 ftrafo.o phgrreal.o grphreal.o rwGRIB2.o posnam.o preconvert.o -L/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/lib -leccodes_f90 -leccodes -ljasper -lpthread -L/usr/local/apps/jasper/1.900.1/LP64/lib -ljasper -lm -L/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -Wl,-rpath,/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -lemos.R64.D64.I32 -L/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -lfftw3   
    338     -rwxr-x---. 1 USER at 353134 May 23 12:27 CONVERT2
    339     SUCCESS!   
    340 
    341 
    342 
    343 
    344 
    345 
    346 
    347 
    348 
    349 
    350 
    351 
    352 
    353 
    354 
    355 
    356 
    357 
    358 
    359 
    360 
    361 
    362 
    363 .. _ref-gateway-mode:
    364 
    365 Gateway mode
    366 ------------
    367 
    368 
    369 .. _ref-req-gateway:
    370  
    371 Gateway environment requirements
    372 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    373 
    374 The environment on your local system has to provide these software packages
    375 and libraries, since ``flex_extract`` does only prepare the job script and send
    376 it to the ECMWF servers:
    377    
    378 * `Python3`_ or `Anaconda Python3`_
    379 * `numpy`_
    380 * `genshi`_
    381  
    382 
    383 .. _ref-prep-gateway:
    384 
    385 Prepare gateway environment
    386 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    387 
    388 The easiest way to install all required packages is to use the package management system of your Linux distribution. To do so, it is necessary to use a user with admin rights.
    389 The installation was tested on a *GNU/Linux Debian buster* and an *Ubuntu 18.04 Bionic Beaver* system.
    390 
    391 .. code-block:: sh
    392 
    393    # On a Linux Debian or Ubuntu system do
    394    # (if not already available):
    395    apt-get install python3
    396    apt-get install pip
    397    apt-get install genshi
    398    apt-get install numpy
    399 
    400 
    401 .. _ref-test-gateway:
    402 
    403 Test gateway environment
    404 ^^^^^^^^^^^^^^^^^^^^^^^^
    405  
    406 Check the availability of the python packages by typing ``python3`` in
    407 a terminal window and run the ``import`` commands in the python shell.
    408 If there are no error messages, you succeeded in setting up the environment.
    409 
    410 .. code-block:: python
    411    
    412    # check in python3 console
    413    import genshi
    414    import numpy
    415  
    416 
    417 
    418 .. _ref-install-gateway:
    419 
    420 Gateway installation
    421 ^^^^^^^^^^^^^^^^^^^^
    422 
    423 In this mode, access to the ECMWF computing and archiving facilities is enabled through an ECaccess gateway server on a local member state server. The ECaccess framework is necessary to interactively submit jobs to the ECMWF batch system and to transfer files between ECMWF and local gateway server.
    424 
    425 Please see `ECMWF's instructions on gateway server`_ to establish the gateway server if not already in place.
    426 Additionally, to be able to use the Ecaccess file transfer service **ectrans** please also create an association.
    427 The easiest way is to visit the ECaccess Member State Gateway website (e.g. msgatway.ecmwf.int) and follow the instructions in the short `ECaccess Presentation`_ (page 17 ff.). Additional documentation can be found on the `ECMWF ectrans site`_.
    428 
    429 After everything is set up you have to create an *ecaccess certificate* to be able to send and receive files from and to the ECMWF server. You can do this by using the ``ecaccess-certificate-create`` command on the gateway server. You will be prompted for your ECMWF member state user name and a password (which will be usually generated by a Token). This certificate has to be re-newed periodically (every 7 days).
    430 
    431 .. code-block:: bash
    432    
    433    $ ecaccess-certificate-create
    434    Please enter your user-id: example_username
    435    Your passcode: ***
    436    
    437 ``Flex_extract`` will be run on an ECMWF server which makes the setup the same as for the **remote mode**. In the ``setup.sh`` script `[ref] <Documentation/Input/setup.html>`_, select the ``Makefile.gfortran`` for the ``CONVERT2`` Fortran program and the ECMWF server (*target*) you would like to use.
    438 The job script, send to the job queue via the ECaccess software, selects again automatically the correct libraries from the module system. For enableing the file transfer you have to set the *ECUID*, *ECGID*, *GATEWAY* and *DESTINATION* parameter values.
    439  
    440 
    441 .. code-block:: bash
    442     :caption: 'Example settings for a gateway installation.'
    443     :name: setup.sh
    444    
    445     # -----------------------------------------------------------------
    446     # AVAILABLE COMMANDLINE ARGUMENTS TO SET
    447     #
    448     # THE USER HAS TO SPECIFY THESE PARAMETER
    449     #
    450     TARGET='ecgate'
    451     MAKEFILE='Makefile.gfortran'
    452     ECUID='uid'
    453     ECGID='gid'
    454     GATEWAY='server.example.edu'
    455     DESTINATION='example@genericSftp'
    456     INSTALLDIR=None
    457     JOB_TEMPLATE='job.template'
    458     CONTROLFILE='CONTROL_EA5'
    459 
    460 
    461 Afterwards, type:
    462 
    463 .. code-block:: bash
    464 
    465    $ ./setup.sh
    466    
    467 to start the installation. You should see the following output at the command line.
    468    
    469    
    470 .. code-block:: bash
    471 
    472    # Output of setup.sh
    473    Create tarball ...
    474    Job compilation script has been submitted to ecgate for installation in ${HOME}/flex_extract_vX.X
    475    You should get an email with subject "flexcompile" within the next few minutes!
    476 
    477    
    478 The email content should look like this with a "SUCCESS" statement in the last line:
    479 
    480 .. code-block:: bash
    481 
    482     gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp phgrreal.f
    483     gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp grphreal.f
    484     gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp ftrafo.f
    485     gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I. -I/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/include -g rwGRIB2.f90
    486     gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I. -I/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/include -g posnam.f
    487     gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I. -I/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/include -g preconvert.f90
    488     gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -g -O3 -o ./CONVERT2 ftrafo.o phgrreal.o grphreal.o rwGRIB2.o posnam.o preconvert.o -L/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/lib -leccodes_f90 -leccodes -ljasper -lpthread -L/usr/local/apps/jasper/1.900.1/LP64/lib -ljasper -lm -L/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -Wl,-rpath,/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -lemos.R64.D64.I32 -L/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -lfftw3   
    489     -rwxr-x---. 1 USER at 353134 May 23 12:27 CONVERT2
    490     SUCCESS!   
    491    
    492    
    493    
    494    
    495    
    496    
    497    
    498    
    499    
    500    
    501    
    502    
    503    
    504    
    505    
    506    
    507    
    508    
    509    
    510 .. _ref-local-mode:
    511 
    512 Local mode
    513 ----------
    514 
    515 
    516 
    517 .. _ref-req-local:
    518  
    519 Local environment requirements
    520 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    521 
    522 For the local access modes **member** and **public** there is no difference in
    523 the installation process.
    524 
    525 The environment on your local system has to provide these software packages
    526 and libraries, since all extraction and preparation is done at the local system:
    527 
    528 +------------------------------------------------+-----------------+
    529 |  Python part                                   | Fortran part    |
    530 +------------------------------------------------+-----------------+
    531 | * `Python3`_                                   | * `gfortran`_   |
    532 | * `numpy`_                                     | * `fftw3`_      |
    533 | * `genshi`_                                    | * `eccodes`_    |
    534 | * `eccodes for python`_                        | * `emoslib`_    |
    535 | * `ecmwf-api-client`_ (everything except ERA5) |                 |
    536 | * `cdsapi`_ (just for ERA5)                    |                 |
    537 +------------------------------------------------+-----------------+
    538 
    539 
    540 .. _ref-prep-local:
    541 
    542 Prepare local environment
    543 ^^^^^^^^^^^^^^^^^^^^^^^^^
    544 
    545 The easiest way to install all required packages is to use the package management system of your Linux distribution. To do so, it is necessary to use a user with admin rights.
    546 The installation was tested on a *Debian GNU/Linux buster/sid* and an *Ubuntu 18.04 Bionic Beaver* system.
    547 
    548 .. code-block:: sh
    549 
    550    # On a Linux Debian or Ubuntu system do
    551    # (if not already available):
    552    apt-get install python3 (usually available on normal Linux systems)
    553    apt-get install pip
    554    apt-get install gfortran
    555    apt-get install fftw3-dev
    556    apt-get install libeccodes-dev
    557    apt-get install libemos-dev
    558    apt-get install python3-eccodes
    559    apt-get install genshi
    560    apt-get install numpy
    561    pip install cdsapi
    562    pip install ecmwf-api-client
    563 
    564 .. note::
    565 
    566     In case you would like to use Anaconda Python we recommend you follow the installation instructions of
    567     `Anaconda Python Installation for Linux <https://docs.anaconda.com/anaconda/install/linux/>`_ and then install the
    568     ``eccodes`` package from ``conda`` with:
    569 
    570     .. code-block:: bash
    571 
    572        conda install conda-forge::python-eccodes
    573 
    574 
    575 The CDS API (cdsapi) and the ECMWF Web API (ecmwf-api-client) have both to be installed since ERA5 can only be retrieved with the ``CDS API`` and all other datasets with the ``ECMWF Web API``.     
    576        
    577 .. note::
    578 
    579     Since **public users** currently don't have access to the full *ERA5* dataset they can skip the installation of the ``CDS API``.
    580    
    581 Both user groups have to provide key's with their credentials for the Web API's in their home directory. Therefore, follow these instructions:
    582    
    583 ECMWF Web API:
    584    Go to `MARS access`_ website and log in with your credentials. Afterwards, on this site in section "Install ECMWF KEY" the key for the ECMWF Web API should be listed. Please follow the instructions in this section under point 1 (save the key in a file `.ecmwfapirc` in your home directory).
    585      
    586 CDS API:
    587    Go to 'CDS API registration'_ and register there too. Log in at the `cdsapi`_ website and follow the instructions at section "Install the CDS API key" to save your credentials in a `.cdsapirc` file.
    588      
    589 
    590 .. _ref-test-local:
    591    
    592 Test local environment
    593 ^^^^^^^^^^^^^^^^^^^^^^
    594 
    595 Check the availability of the system packages with ``dpkg -s <package-name> |  grep Status`` or ``rpm -q <package_name>``, depending on your system. For example:
    596 
    597 .. code-block:: sh
    598 
    599    $ dpkg -s libeccodes-dev |  grep Status
    600    # or
    601    $ rpm -q libeccodes-dev
    602  
    603 Afterwards, check the availability of the python packages by typing ``python3`` in
    604 a terminal window and run the ``import`` commands in the python shell. If there are no error messages, you succeeded in setting up the environment.
    605 
    606 .. code-block:: python
    607    
    608    # check in python3 console
    609    import eccodes
    610    import genshi
    611    import numpy
    612    import cdsapi
    613    import ecmwfapi
    614    
    615 
    616 
    617 Test the Web API's
    618 """"""""""""""""""
    619 
    620 You can start very simple test retrievals for both Web API's to be sure that everything works. This is recommended to minimize the range of possible errors using ``flex_extract`` later on.
    621 
    622 ECMWF Web API
    623 """""""""""""
    624 
    625 
    626 +----------------------------------------------------------+----------------------------------------------------------+
    627 |Please use this piece of python code for **Member user**: |Please use this piece of python code for **Public user**: |
    628 +----------------------------------------------------------+----------------------------------------------------------+
    629 |.. code-block:: python                                    |.. code-block:: python                                    |
    630 |                                                          |                                                          |
    631 |    from ecmwfapi import ECMWFService                     |    from ecmwfapi import ECMWFDataServer                  |
    632 |                                                          |                                                          |
    633 |    server = ECMWFService('mars')                         |    server = ECMWFDataServer()                            |
    634 |                                                          |                                                          |
    635 |    server.retrieve({                                     |    server.retrieve({                                     |
    636 |        'stream'    : "oper",                             |        'stream'    : "enda",                             |
    637 |        'levtype'   : "sfc",                              |        'levtype'   : "sfc",                              |
    638 |        'param'     : "165.128/166.128/167.128",          |        'param'     : "165.128/166.128/167.128",          |
    639 |        'dataset'   : "interim",                          |        'dataset'   : "cera20c",                          |
    640 |        'step'      : "0",                                |        'step'      : "0",                                |
    641 |        'grid'      : "0.75/0.75",                        |        'grid'      : "1./1.",                            |
    642 |        'time'      : "00/06/12/18",                      |        'time'      : "00/06/12/18",                      |
    643 |        'date'      : "2014-07-01/to/2014-07-31",         |        'date'      : "2000-07-01/to/2000-07-31",         |
    644 |        'type'      : "an",                               |        'type'      : "an",                               |
    645 |        'class'     : "ei",                               |        'class'     : "ep",                               |
    646 |        'target'    : "download_erainterim_ecmwfapi.grib" |        'target'    : "download_cera20c_ecmwfapi.grib"    |
    647 |    })                                                    |    })                                                    |
    648 +----------------------------------------------------------+----------------------------------------------------------+
    649 
    650            
    651    
    652 CDS API
    653 """""""
    654 
    655 Since ERA5 extraction with CDS API might take some time due to the very high number of requests, you can start by retrieving some online stored pressure levels (not from MARS). This is usually much faster and gives a quick result to find out if the web API works:
    656 
    657 Please use this piece of python code to retrieve a small sample of *ERA5* pressure levels:
    658 
    659 .. code-block:: python
    660 
    661     import cdsapi
    662    
    663     c = cdsapi.Client()
    664    
    665     c.retrieve("reanalysis-era5-pressure-levels",
    666     {
    667     "variable": "temperature",
    668     "pressure_level": "1000",
    669     "product_type": "reanalysis",
    670     "year": "2008",
    671     "month": "01",
    672     "day": "01",
    673     "time": "12:00",
    674     "format": "grib"
    675     },
    676     "download_cdsapi.grib")
    677 
    678 
    679 Afterwards, when you know that the CDS API generally works, you can try to extract some
    680 data from the MARS archive. From the latest experience we know that this can take a while.   
    681 
    682 .. **Member user**
    683 
    684 Please use this piece of python code to retrieve a small *ERA5* data sample as a **member user**! The **public user** doesn't have access to the full *ERA5* dataset!
    685 
    686 .. code-block:: python
    687 
    688    import cdsapi
    689    
    690    c = cdsapi.Client()
    691    
    692    c.retrieve('reanalysis-era5-complete',
    693    {
    694        'class'   : 'ea',
    695        'expver'  : '1',
    696        'stream'  : 'oper',
    697        'type'    : 'fc',
    698        'step'    : '3/to/12/by/3',
    699        'param'   : '130.128',
    700        'levtype' : 'ml',
    701        'levelist': '135/to/137',
    702        'date'    : '2013-01-01',
    703        'time'    : '06/18',
    704        'area'    : '50/-5/40/5',
    705        'grid'    : '1.0/1.0',
    706        'format'  : 'grib',
    707    }, 'download_era5_cdsapi.grib')
    708 
    709 
    710 ..  ********************** COMMENTED OUT FOR FUTURE
    711     ********************** PUBLIC RETRIEVAL IS CURRENTLY NOT ACCESSIBLE
    712    
    713     **Public user**
    714     Please use this piece of python code:
    715 
    716     .. code-block:: python
    717 
    718        import cdsapi
    719        
    720        c = cdsapi.Client()
    721        
    722        c.retrieve('reanalysis-era5-complete',
    723        {
    724            'class'   : 'ea',
    725            'dataset' : 'era5',
    726            'expver'  : '1',
    727            'stream'  : 'oper',
    728            'type'    : 'fc',
    729            'step'    : '3/to/12/by/3',
    730            'param'   : '130.128',
    731            'levtype' : 'ml',
    732            'levelist': '135/to/137',
    733            'date'    : '2013-01-01',
    734            'time'    : '06/18',
    735            'area'    : '50/-5/40/5',
    736            'grid'    : '1.0/1.0',
    737            'format'  : 'grib',
    738        }, 'download_era5_cdsapi.grib')
    739 
    740 
    741 
    742 
    743 
    744 
    745 .. _ref-install-local:
    746 
    747 Local installation
    748 ^^^^^^^^^^^^^^^^^^
    749 
    750 First prepare the Fortran ``Makefile`` for your environment and set it
    751 in the ``setup.sh`` script. (See section :ref:`ref-convert` for information on
    752 the Fortran program.)
    753 ``flex_extract`` has already two ``Makefiles`` prepared for te ``gfortran`` and
    754 the ``ifort`` compiler:
    755 
    756  * Makefile.local.gfortran
    757  * Makefile.local.ifort
    758 
    759 They can be found in the path ``flex_extract_vX.X/source/fortran``, where
    760 ``vX.X`` should be substituted with the current version number.
    761 
    762 .. caution::   
    763    It is necessary to adapt **ECCODES_INCLUDE_DIR** and **ECCODES_LIB** in these
    764    ``Makefiles``.
    765 
    766 
    767 So starting from the root directory of ``flex_extract``,
    768 go to the ``Fortran`` source directory and open the ``Makefile`` of your
    769 choice to modify with an editor of your choice. We use the ``nedit`` in this case.
    770 
    771 .. code-block:: bash
    772 
    773    cd flex_extract_vX.X/source/fortran
    774    nedit Makefile.local.gfortran
    775 
    776 Edit the pathes to the ``eccodes`` library on your local machine.
    777 
    778 .. caution::
    779    This can vary from system to system.
    780    It is suggested to use a command like
    781 
    782    .. code-block:: bash
    783 
    784       # for the ECCODES_INCLUDE_DIR path do:
    785       $ dpkg -L libeccodes-dev | grep eccodes.mod
    786       # for the ECCODES_LIB path do:
    787       $ dpkg -L libeccodes-dev | grep libeccodes.so
    788      
    789    to find out the path to the ``eccodes`` library.
    790    
    791 Substitute these paths in the ``Makefile`` for parameters **ECCODES_INCLUDE_DIR**
    792 and **ECCODES_LIB** and save it.
    793 
    794 .. code-block:: bash
    795 
    796    # these are the paths on a current Debian 10 Testing system (May 2019)
    797    ECCODES_INCLUDE_DIR=/usr/lib/x86_64-linux-gnu/fortran/gfortran-mod-15/
    798    ECCODES_LIB= -L/usr/lib -leccodes_f90 -leccodes -lm 
    799    
    800    
    801 The Fortran program called ``CONVERT2`` will be compiled during the
    802 installation process to get an executable. Therefore the ``Makefile``
    803 has to be set in the ``setup.sh`` script.
    804 
    805 In the root directory of ``flex_extract``, open the ``setup.sh`` script
    806 and adapt the installation parameters in the section labelled with
    807 "AVAILABLE COMMANDLINE ARGUMENTS TO SET" like shown below.
    808 
    809 
    810 .. code-block:: bash
    811    :caption: 'Example settings for a local installation.'
    812    :name: setup.sh
    813    
    814    ...
    815    # -----------------------------------------------------------------
    816    # AVAILABLE COMMANDLINE ARGUMENTS TO SET
    817    #
    818    # THE USER HAS TO SPECIFY THESE PARAMETER
    819    #
    820    TARGET='local'
    821    MAKEFILE='Makefile.local.gfortran'
    822    ECUID=None
    823    ECGID=None
    824    GATEWAY=None
    825    DESTINATION=None
    826    INSTALLDIR=None
    827    JOB_TEMPLATE='job.template'
    828    CONTROLFILE='CONTROL_EA5'
    829    ...
    830 
    831 
    832 Afterwards, type:
    833 
    834 .. code-block:: bash
    835 
    836    $ ./setup.sh
    837    
    838 to start the installation. You should see the following output at the command line.
    839    
    840    
    841 .. code-block:: bash
    842 
    843    # Output of setup.sh   
    844    WARNING: installdir has not been specified
    845    flex_extract will be installed in here by compiling the Fortran source in /raid60/nas/tmc/Anne/Interpolation/flexextract/flex_extract_v7.1/source/fortran
    846    Install flex_extract_v7.1 software at local in directory /raid60/nas/tmc/Anne/Interpolation/flexextract/flex_extract_v7.1
    847 
    848    Using makefile: Makefile.local.gfortran
    849    gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp phgrreal.f
    850    gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp grphreal.f
    851    gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp ftrafo.f
    852    gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -fopenmp -I. -I/usr/local/gcc-4.9.3/grib_api-1.14.3/include -O3 rwGRIB2.f90
    853    gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -fopenmp -I. -I/usr/local/gcc-4.9.3/grib_api-1.14.3/include -O3 posnam.f
    854    gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -fopenmp -I. -I/usr/local/gcc-4.9.3/grib_api-1.14.3/include -O3 preconvert.f90
    855    gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -O3 -O3 -fopenmp -o ./CONVERT2 ftrafo.o phgrreal.o grphreal.o rwGRIB2.o posnam.o preconvert.o -L/usr/local/gcc-4.9.3/grib_api-1.14.3/lib -Bstatic  -lgrib_api_f77 -lgrib_api_f90 -lgrib_api -Bdynamic  -lm  -ljasper -lemosR64
    856 
    857    -rwxrwxr-x. 1 philipa8 tmc 282992 May 23 22:27 ./CONVERT2
    858 
    859 
    860 
    861 
    862 
     168Select one of the following modes to install:
     169
     170    :doc:`Installation/remote`
     171   
     172    :doc:`Installation/gateway`
     173   
     174    :doc:`Installation/local`
     175
     176
     177.. toctree::
     178    :hidden:
     179    :maxdepth: 2
     180   
     181    Installation/remote
     182    Installation/gateway
     183    Installation/local
     184
     185
     186
     187
     188
     189
     190
     191
     192
     193
     194
     195
     196
     197
     198
     199
     200   
     201   
     202   
    863203
    864204``Flex_extract`` in combination with ``FLEXPART``
    865205=================================================
    866206
    867 Some users might like to incorporate ``flex_extract`` directly into the ``FLEXPART``
    868 distribution. Then the installation path has to be changed by setting the parameter
    869 `installdir` in the ``setup.sh`` file to the ``script`` directory in the ``FLEXPART`` root directoy.
     207Some users might wish to incorporate ``flex_extract`` directly into the ``FLEXPART`` distribution. Then the installation path has to be changed by setting the parameter `installdir` in the ``setup.sh`` file to the ``script`` directory in the ``FLEXPART`` root directoy.
    870208
    871209
    … …  
    885223--------------------
    886224
    887 To check if the compilation of the Fortran program ``CONVERT2`` was successful
    888 a quick program call on a minimal prepared dataset can be done.
    889 
    890 For this, go from the ``flex_extract`` root directory to the test
    891 directory and call the executable of the Fortran program.
     225To check whether the Fortran program ``CONVERT2`` has been compiled and runs properly, it can be applied to a prepared minimal dataset.
     226
     227For this, go from the ``flex_extract`` root directory to the ``Testing/Installation/Convert/`` directory and execute the Fortran program.
    892228
    893229.. note::
    … …  
    896232.. code-block:: bash
    897233   
    898    cd test/Installation/Convert
    899    # call the Fortran progam without arguments
    900    ../../../source/fortran/CONVERT2
    901 
    902 The installation was successfull if it showed the following output:
     234   cd Testing/Installation/Convert
     235   # execute the Fortran progam without arguments
     236   ../../../Source/Fortran/CONVERT2
     237
     238The installation was successfull if you obtain on standard output:
    903239
    904240.. code-block:: bash
    … …  
    912248   SUCCESSFULLY FINISHED CONVERT_PRE: CONGRATULATIONS
    913249
    914 Now go back to the root directoy:
     250Now go back to the root directory:
    915251
    916252.. code-block:: bash

  • For_developers/Sphinx/source/quick_start.rst

    rba99230 r30f7911  
    337337.. todo::
    338338
    339     @LEO: please check the complete describtion and functionality of the CONTROL FILEs
     339    @LEO: please check the complete description and functionality of the CONTROL FILEs
    340340   
    341341Operational data
Note: See TracChangeset for help on using the changeset viewer.
hosted by ZAMG