source: flex_extract.git/Documentation/html/_sources/Installation/remote.rst.txt @ 30f7911

************************
Remote mode installation
************************

.. role:: underline
    :class: underline
    
.. toctree::
    :hidden:
    :maxdepth: 2    
    
     
.. _Python 3: https://docs.python.org/3/
.. _Python3: https://www.python.org/downloads/
.. _Anaconda Python3: https://www.anaconda.com/distribution/#download-section

.. _numpy: http://www.numpy.org/
.. _ecmwf-api-client: https://confluence.ecmwf.int/display/WEBAPI/ECMWF+Web+API+Home
.. _cdsapi: https://cds.climate.copernicus.eu/api-how-to
.. _genshi: https://genshi.edgewall.org/
.. _eccodes for python: https://packages.debian.org/sid/python3-eccodes 
.. _eccodes for conda: https://anaconda.org/conda-forge/eccodes
.. _gfortran: https://gcc.gnu.org/wiki/GFortran
.. _fftw3: http://www.fftw.org
.. _eccodes: https://software.ecmwf.int/wiki/display/ECC
.. _emoslib: https://software.ecmwf.int/wiki/display/EMOS/Emoslib
.. _member state: https://www.ecmwf.int/en/about/who-we-are/member-states 
.. _registration form: https://apps.ecmwf.int/registration/
.. _CDS API registration: https://cds.climate.copernicus.eu/user/register
.. _ECMWF ectrans site: https://confluence.ecmwf.int/display/ECAC/Unattended+file+transfer+-+ectrans
.. _ECaccess Presentation: https://confluence.ecmwf.int/download/attachments/45759146/ECaccess.pdf
.. _ECMWF's instructions on gateway server: https://confluence.ecmwf.int/display/ECAC/ECaccess+Home
.. _Computing Representative: https://www.ecmwf.int/en/about/contact-us/computing-representatives
.. _MARS access: https://confluence.ecmwf.int//display/WEBAPI/Access+MARS

.. _download section: https://www.flexpart.eu/downloads


.. _ref-remote-mode: 


.. _ref-req-remote: 
 
Remote environment requirements
===============================

The environment for ``flex_extract`` on ECMWF servers has to provide these 
software packages and libraries:
    
+---------------------------+-----------------+
|  Python part              | Fortran part    |
+---------------------------+-----------------+
| * `Python3`_              | * `gfortran`_   |
| * `numpy`_                | * `fftw3`_      |
| * `genshi`_               | * `eccodes`_    |
| * `eccodes for python`_   | * `emoslib`_    |
+---------------------------+-----------------+


.. _ref-prep-remote:

Prepare remote environment
==========================
 
The environment on the ECMWF server (such as *ecgate* or *cca*) is already 
prepared. ECMWF server provide all libraries via a module system which is 
going to be selected by ``flex_extract`` automatically.


.. _ref-install-remote:

Remote installation
===================

First, log in on one of the ECMWF Linux server, such as *ecgate* or *cca/ccb*. 
Substitute *<ecuid>* with your ECMWF user name:

.. code-block:: bash
   
   ssh -X <ecuid>@ecaccess.ecmwf.int

This will lead to the following output on the command line, asking for your 
password:
   
.. code-block:: bash

   Authorized access only.

   ***************************************************************
      For further information, read the ECaccess documentation at:

      https://software.ecmwf.int/wiki/display/ECAC/ECaccess+Home

      You can also use ECaccess to load & download files from your
      EChome, ECscratch or ECfs directories using the ECaccess FTP
      server:

      ftp://uid@ecaccess.ecmwf.int/

      Please note you must use your UID and ActivID code to login!
   ***************************************************************

   <ecuid>@131.130.157.5's password: ***
   Select hostname (ecgate, cca, ccb) [ecgate]: ecgate

   [<ecuid>@ecgb11 ~]$ 
   
   
Then, copy the ``flex_extract`` tar ball (from section :ref:`ref-download`) 
to the ``$HOME`` directory of the ECMWF Linux server via ``scp``.
Substitute the *<localuser>* and *<server.edu>* placeholders with your credentials. 
Untar the file and change into the ``flex_extract`` root directory. 

.. code-block:: bash

   scp <localuser>@<server.edu>:/path/to/tarfile/flex_extract_vX.X.tar.gz  $HOME/
   cd $HOME
   tar xvf flex_extract_vX.X.tar.gz
   cd flex_extract_vX.X
   

On these ECMWF servers, it is not necessary to prepare the environment or the 
``Makefile`` for the Fortran program (``CONVERT2``) as described above. 
All third party libraries are available from a module system. The ``Makefile``
is optimized for ECMWF servers and the compilation 
script ``compilejob.ksh``, which will be submitted by ``flex_extract`` to the 
batch job queue at ECMWF, does load all relevant modules from the ECMWF's module system. 

So there is just the need to execute the ``setup.sh`` script from the 
``flex_extract`` root directory for installation. 
Before executing it, it is necessary to adapt some parameters from ``setup.sh``
described in :doc:`Documentation/Input/setup`. 

Open ``setup.sh`` with your editor and adapt the values:  

+----------------------------------------------+----------------------------------------------+   
|   Take this for target = **ectrans**         |  Take this for target = **cca**              | 
+----------------------------------------------+----------------------------------------------+
| .. code-block:: bash                         | .. code-block:: bash                         | 
|                                              |                                              | 
|   ...                                        |   ...                                        |   
|   # -----------------------------------------|   # -----------------------------------------|
|   # AVAILABLE COMMANDLINE ARGUMENTS TO SET   |   # AVAILABLE COMMANDLINE ARGUMENTS TO SET   |
|   #                                          |   #                                          |  
|   # THE USER HAS TO SPECIFY THESE PARAMETER  |   # THE USER HAS TO SPECIFY THESE PARAMETER  | 
|   #                                          |   #                                          |
|   TARGET='ecgate'                            |   TARGET='cca'                               |
|   MAKEFILE='Makefile.gfortran'               |   MAKEFILE='Makefile.CRAY'                   |  
|   ECUID='uid'                                |   ECUID='uid'                                |  
|   ECGID='gid'                                |   ECGID='gid'                                |
|   GATEWAY=None                               |   GATEWAY=None                               |
|   DESTINATION=None                           |   DESTINATION=None                           | 
|   INSTALLDIR=None                            |   INSTALLDIR=''                              | 
|   JOB_TEMPLATE='job.template'                |   JOB_TEMPLATE='job.template'                |
|   CONTROLFILE='CONTROL_EA5'                  |   CONTROLFILE='CONTROL_EA5'                  | 
|   ...                                        |   ...                                        |   
+----------------------------------------------+----------------------------------------------+

:underline:`Please substitute the values of ECUID and ECGID
with your own credentials and settings.`

.. note::

   If a local gateway server is available the transfer of files could be done
   via the ``ECaccess`` commands. Therefore a valid *GATEWAY* and *DESTINATION*
   have to be present and should be set in the ``setup.sh`` file. 


Afterwards, type:

.. code-block:: bash

   module load python3   
   ./setup.sh
   
to start the installation. You should see the following output at the command line. 
    
    
.. code-block:: bash

   # Output of setup.sh
   Create tarball ...
   Job compilation script has been submitted to ecgate for installation in ${HOME}/flex_extract_vX.X
   You should get an email with subject "flexcompile" within the next few minutes!

    
The email content should look like this with a "SUCCESS" statement in the last line:

.. code-block:: bash

    gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp phgrreal.f
    gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp grphreal.f
    gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp ftrafo.f
    gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I. -I/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/include -g rwGRIB2.f90
    gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I. -I/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/include -g posnam.f
    gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I. -I/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/include -g preconvert.f90
    gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -g -O3 -o ./CONVERT2 ftrafo.o phgrreal.o grphreal.o rwGRIB2.o posnam.o preconvert.o -L/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/lib -leccodes_f90 -leccodes -ljasper -lpthread -L/usr/local/apps/jasper/1.900.1/LP64/lib -ljasper -lm -L/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -Wl,-rpath,/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -lemos.R64.D64.I32 -L/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -lfftw3   
    -rwxr-x---. 1 USER at 353134 May 23 12:27 CONVERT2
    SUCCESS!