source: flex_extract.git/Documentation/html/_sources/Installation/gateway.rst.txt @ d720895

*************************
Gateway mode installation
*************************

.. role:: underline
    :class: underline
    
.. toctree::
    :hidden:
    :maxdepth: 2
    
.. _Python 3: https://docs.python.org/3/
.. _Python3: https://www.python.org/downloads/
.. _Anaconda Python3: https://www.anaconda.com/distribution/#download-section

.. _numpy: http://www.numpy.org/
.. _ecmwf-api-client: https://confluence.ecmwf.int/display/WEBAPI/ECMWF+Web+API+Home
.. _cdsapi: https://cds.climate.copernicus.eu/api-how-to
.. _genshi: https://genshi.edgewall.org/
.. _eccodes for python: https://packages.debian.org/sid/python3-eccodes 
.. _eccodes for conda: https://anaconda.org/conda-forge/eccodes
.. _gfortran: https://gcc.gnu.org/wiki/GFortran
.. _fftw3: http://www.fftw.org
.. _eccodes: https://software.ecmwf.int/wiki/display/ECC
.. _emoslib: https://software.ecmwf.int/wiki/display/EMOS/Emoslib
.. _member state: https://www.ecmwf.int/en/about/who-we-are/member-states 
.. _registration form: https://apps.ecmwf.int/registration/
.. _CDS API registration: https://cds.climate.copernicus.eu/user/register
.. _ECMWF ectrans site: https://confluence.ecmwf.int/display/ECAC/Unattended+file+transfer+-+ectrans
.. _ECaccess Presentation: https://confluence.ecmwf.int/download/attachments/45759146/ECaccess.pdf
.. _ECMWF's instructions on gateway server: https://confluence.ecmwf.int/display/ECAC/ECaccess+Home
.. _Computing Representative: https://www.ecmwf.int/en/about/contact-us/computing-representatives
.. _MARS access: https://confluence.ecmwf.int//display/WEBAPI/Access+MARS

.. _download section: https://www.flexpart.eu/downloads



.. _ref-gateway-mode:



.. _ref-req-gateway: 
 
Gateway mode - dependencies
===========================

* `Python3`_ 
* `numpy`_
* `genshi`_
 

.. _ref-prep-gateway:

Prepare gateway environment
===========================

In this mode, access to the ECMWF computing and archiving facilities is enabled through an ECaccess gateway server on a local member state server. The ECaccess framework is necessary to interactively submit jobs to the ECMWF batch system and to transfer files between ECMWF and local gateway server. 

Please see `ECMWF's instructions on gateway server`_ to establish the gateway server if not already in place. Additionally, to be able to use the ECaccess file transfer service **ectrans** it is necessary to create a so-called association. The easiest way is to visit the ECaccess Member State Gateway website (e.g. https://msgatway.ecmwf.int) and follow the instructions in the short `ECaccess Presentation`_ (page 17 ff.). Additional documentation can be found on the `ECMWF ectrans site`_.

The next step is to create an *ecaccess certificate* to be able to send and receive files from and to the ECMWF server. You can do this by using the ``ecaccess-certificate-create`` command on the gateway server. You will be prompted for your ECMWF member state user name and a password (generated by a token). This certificate has to be renewed periodically (every 7 days).


.. code-block:: bash
   
   $ ecaccess-certificate-create
   Please enter your user-id: example_username
   Your passcode: ***
   
   
   
The easiest way to install all required packages is to use the package management system of your Linux distribution which required admin rights.
The installation was tested under *GNU/Linux Debian buster* and *Ubuntu 18.04 Bionic Beaver*.

.. code-block:: sh


   # On a Debian or Debian-derived sytem (e. g. Ubuntu) system you may use the following commands (or equivalent commands of your preferred package manager):
   # (if not already available):
   apt-get install python3
   apt-get install python3-genshi
   apt-get install python3-numpy
   

.. _ref-test-gateway:

Test gateway environment
========================

Check the availability of the python packages by typing ``python3`` in
a terminal window and run the ``import`` commands in the python shell. 
If there are no error messages, you succeeded in setting up the environment.
 
.. code-block:: python
    
   # check in python3 console
   import genshi
   import numpy
 


.. _ref-install-gateway:

Gateway installation
====================

``Flex_extract`` will be run on an ECMWF server which makes the setup the same as for the **remote mode**. In the ``setup.sh`` script `[ref] <Documentation/Input/setup.html>`_, select the ``makefile.gfortran`` for the ``CONVERT2`` Fortran program and the ECMWF server (*target*) you would like to use. 
The job script, send to the job queue via the ECaccess software, loads the required modules from the module system. For enabling the file transfer via the gateway, you have to set the *ECUID*, *ECGID*, *GATEWAY* and *DESTINATION* parameter values.
   

.. code-block:: bash
    :caption: 'Example settings for a gateway installation.'
    :name: setup.sh
    
    # -----------------------------------------------------------------
    # AVAILABLE COMMANDLINE ARGUMENTS TO SET
    #
    # THE USER HAS TO SPECIFY THESE PARAMETER
    #
    TARGET='ecgate'
    MAKEFILE='makefile.gfortran'
    ECUID='uid'
    ECGID='gid'
    GATEWAY='localserver.tld'
    DESTINATION='association_name@genericSftp'
    INSTALLDIR=None
    JOB_TEMPLATE='job.template'
    CONTROLFILE='CONTROL_EA5'


Afterwards, type:

.. code-block:: bash

   $ ./setup.sh
   
to start the installation. You should see the following on standard output. 
    
    
.. code-block:: bash

   # Output of setup.sh
   Create tarball ...
   Job compilation script has been submitted to ecgate for installation in ${HOME}/flex_extract_vX.X
   You should get an email with subject "flexcompile" within the next few minutes!

``Flex_extract`` uses the email address connectd to the user account on ECMWF servers. The email content should look like this with a "SUCCESS" statement in the last line:

.. code-block:: bash

    gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp phgrreal.f
    gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp grphreal.f
    gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp ftrafo.f
    gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I. -I/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/include -g rwGRIB2.f90
    gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I. -I/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/include -g posnam.f
    gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I. -I/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/include -g preconvert.f90
    gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -g -O3 -o ./CONVERT2 ftrafo.o phgrreal.o grphreal.o rwGRIB2.o posnam.o preconvert.o -L/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/lib -leccodes_f90 -leccodes -ljasper -lpthread -L/usr/local/apps/jasper/1.900.1/LP64/lib -ljasper -lm -L/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -Wl,-rpath,/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -lemos.R64.D64.I32 -L/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -lfftw3   
    -rwxr-x---. 1 USER at 353134 May 23 12:27 CONVERT2
    SUCCESS!