Context Navigation

installation.rst @ 41c9dbc

ctbtodev

Last change on this file since 41c9dbc was 41c9dbc, checked in by Anne Philipp <anne.philipp@…>, 5 years ago
Large update of Sphinx doc; moved Sphinx source to dev dir; Cleaned control files
Property mode set to `100644`
File size: 35.0 KB

Installation

.. toctree::
    :hidden:
    :maxdepth: 2

flex_extract is a software package which contains a mix of Python and Shell scripts as well as a Fortran program. These components rely on a couple of third party libraries which need to be prepared first before starting the installation process. For now, the software is only tested for a Linux/Unix environment. Feel free to try it out on other platforms.

Start with the decision of which user group you belong to and follow the instructions at :ref:`ref-registration` to get an account at ECMWF. Considering your user group and the method of applying flex_extract there appear to be 4 application modes:

Remote (for member state users only) :ref:`[installation]<ref-remote-mode>`
?
Gateway (for member state users only) :ref:`[installation]<ref-gateway-mode>`
?
Local member :ref:`[installation]<ref-local-mode>`
?
Local public :ref:`[installation]<ref-local-mode>`
?

More information can be found in :doc:`Documentation/Overview/app_modes`.

Note

If you encounter any problems in the installation process, you can ask for :doc:`support`.

Registration at ECMWF

Decide which user group you belong to and follow the instructions for registration:

Member state user:: To get a member-state user account, users have to be a resident of a member state. In that case, you can contact your Computing Representative for granting access. If you would like to use the local application mode to retrieve ERA5 data you'd have to register at the Copernicus Climate Data Store also.
Public user:: To be able to download all intended public datasets with flex_extract such as ERA-Interim, CERA-20C and ERA5, the public user has to create 2 accounts. At first, registration at the ECMWF website by filling out this registration form is needed to be able to retrieve ERA-Interim and CERA-20C data. For the retrievement of ERA5 a registration at the Copernicus Climate Data Store is needed in addition.

Agree on Licences for Public Datasets

Each public dataset which is intended to be downloaded by flex_extract has its own licence which has to be accepted, regardless of the user group.

For the ERA-Interim and CERA-20C datasets this can be done at the ECMWF website Available ECMWF Public Datasets. Log in and follow the licence links on the right side for each dataset and accept it.

For the ERA5 dataset this has to be done at the Climate Data Store (CDS) website. Log in with your credentials and then select on the left panel the product type "Reanalysis" for finding ERA5 datasets. Then follow the link of a title with ERA5 (anyone) to the full dataset record, click on tab "Download data" and scroll down. There is a section "Terms of use" where you have to click the Accept terms button.

Download `flex_extract`

There are 2 options to download flex_extract:

tar ball

You can download the latest prepared release tar ball from the download section of our FLEXPART community website and then untar the file. Substitute the <ID> in the wget command with the ID-number of the flex_extract release tar ball in the list of downloads at the community website.

wget https://www.flexpart.eu/downloads/<ID>
tar -xvf <flex_extract_vX.X.tar>

git repo

Or you can clone the current release version from our git repository master branch.

$ git clone https://www.flexpart.eu/gitmob/flexpart

Environment Requirements

This is a list of the general environment requirements for flex_extract. What is required exactly for each application mode will be described in the specifc installation section.

To run the python part of flex_extract a Python 3 environment is needed. We tested flex_extract with a normal Linux Python package distribution and Anaconda Python. Except for Python3 all are python packages which can be installed via pip.

Python3 or Anaconda Python3
numpy
ecmwf-api-client (Web Interface to ECMWF servers for datasets except ERA5)
cdsapi (Web Interface to C3S servers for ERA5)
genshi
eccodes for python

For the Fortran part of flex_extract we need the following distribution packages:

Installation of `flex_extract`

The actual installation of flex_extract will be done by executing a Shell script called setup.sh. It defines some parameters and calls a Python script by giving the parameters as command line arguments. More information on the script and its parameters can be found at :doc:`Documentation/Input/setup`.

For each application mode installation section we describe the requirements for the explicit environment and how it is installed, test if it works and how the actual flex_extract installation has to be done. At the users local side not all software has to be present for flex_extract.

Remote Mode

Remote Environment Requirements

The environment for flex_extract on ECMWF servers has to provide these software packages and libraries:

Python part	Fortran part
Python3 numpy genshi eccodes for python	gfortran fftw3 eccodes emoslib

Prepare Remote Environment

The environment on the ECMWF server (such as ecgate or cca) is already prepared. ECMWF server provide all libraries via a module system which is going to be selected by flex_extract automatically.

Remote Installation

First, log in on one of the ECMWF Linux server, such as ecgate or cca/ccb. Substitute <ecuid> with your ECMWF user name:

ssh -X <ecuid>@ecaccess.ecmwf.int

This will lead to the following output on the command line, asking for your password:

Authorized access only.
***************************************************************
   For further information, read the ECaccess documentation at:
   https://software.ecmwf.int/wiki/display/ECAC/ECaccess+Home
   You can also use ECaccess to load & download files from your
   EChome, ECscratch or ECfs directories using the ECaccess FTP
   server:
   ftp://uid@ecaccess.ecmwf.int/
   Please note you must use your UID and ActivID code to login!
***************************************************************
<ecuid>@131.130.157.5's password: ***
Select hostname (ecgate, cca, ccb) [ecgate]: ecgate
[<ecuid>@ecgb11 ~]$

Then, copy the flex_extract tar ball (from section :ref:`ref-download`) to the $HOME directory of the ECMWF Linux server via scp. Substitute the <localuser> and <server.edu> placeholders with your credentials. Untar the file and change into the flex_extract root directory.

scp <localuser>@<server.edu>:/path/to/tarfile/flex_extract_vX.X.tar.gz  $HOME/
cd $HOME
tar xvf flex_extract_vX.X.tar.gz
cd flex_extract_vX.X

On these ECMWF servers, it is not necessary to prepare the environment or the Makefile for the Fortran program (CONVERT2) as described above. All third party libraries are available from a module system. The Makefile is optimized for ECMWF servers and the compilation script compilejob.ksh, which will be submitted by flex_extract to the batch job queue at ECMWF, does load all relevant modules from the ECMWF's module system.

So there is just the need to execute the setup.sh script from the flex_extract root directory for installation. Before executing it, it is necessary to adapt some parameters from setup.sh described in :ref:`ref-setup`.

Open setup.sh with your editor and adapt the values:

Take this for target = ectrans

Take this for target = cca

...
# -----------------------------------------
# AVAILABLE COMMANDLINE ARGUMENTS TO SET
#
# THE USER HAS TO SPECIFY THESE PARAMETER
#
TARGET='ecgate'
MAKEFILE='Makefile.gfortran'
ECUID='uid'
ECGID='gid'
GATEWAY=None
DESTINATION=None
INSTALLDIR=None
JOB_TEMPLATE='job.template'
CONTROLFILE='CONTROL_EA5'
...

...
# -----------------------------------------
# AVAILABLE COMMANDLINE ARGUMENTS TO SET
#
# THE USER HAS TO SPECIFY THESE PARAMETER
#
TARGET='cca'
MAKEFILE='Makefile.CRAY'
ECUID='uid'
ECGID='gid'
GATEWAY=None
DESTINATION=None
INSTALLDIR=''
JOB_TEMPLATE='job.template'
CONTROLFILE='CONTROL_EA5'
...

Please substitute the values of ECUID and ECGID with your own credentials and settings.

Note

If a local gateway server is available the transfer of files could be done via the ECaccess commands. Therefore a valid GATEWAY and DESTINATION have to be present and should be set in the setup.sh file.

Afterwards, type:

module load python3
./setup.sh

to start the installation. You should see the following output at the command line.

# Output of setup.sh
Create tarball ...
Job compilation script has been submitted to ecgate for installation in ${HOME}/flex_extract_vX.X
You should get an email with subject "flexcompile" within the next few minutes!

The email content should look like this with a "SUCCESS" statement in the last line:

gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp phgrreal.f
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp grphreal.f
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp ftrafo.f
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I. -I/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/include -g rwGRIB2.f90
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I. -I/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/include -g posnam.f
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I. -I/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/include -g preconvert.f90
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -g -O3 -o ./CONVERT2 ftrafo.o phgrreal.o grphreal.o rwGRIB2.o posnam.o preconvert.o -L/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/lib -leccodes_f90 -leccodes -ljasper -lpthread -L/usr/local/apps/jasper/1.900.1/LP64/lib -ljasper -lm -L/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -Wl,-rpath,/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -lemos.R64.D64.I32 -L/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -lfftw3
-rwxr-x---. 1 USER at 353134 May 23 12:27 CONVERT2
SUCCESS!

Gateway Mode

Gateway Environment Requirements

The environment on your local system has to provide these software packages and libraries, since flex_extract does only prepare the job script and send it to the ECMWF servers:

Prepare Gateway Environment

The easiest way to install all required packages is to use the package management system of your Linux distribution. To do so, it is necessary to use a user with admin rights. The installation was tested on a Debian GNU/Linux buster/sid and an Ubuntu 18.04 Bionic Beaver system.

# On a Linux Debian or Ubuntu system do
# (if not already available):
sudo apt -PV install python3
sudo apt -PV install pip
pip install genshi
pip install numpy

Test Gateway Environment

Check the availability of the python packages by typing python3 in a terminal window and run the import commands in the python shell. If there are no error messages, you succeeded in setting up the environment.

# check in python3 console
import genshi
import numpy

Gateway Installation

In this mode, access to the ECMWF computing and archiving facilities is enabled through an ECaccess gateway server on a local member state server. The ECaccess framework is necessary to interactively submit jobs to the ECMWF batch system and to transfer files between ECMWF and local gateway server.

Please see ECMWF's instructions on gateway server to establish the gateway server if not already in place. Additionally, to be able to use the Ecaccess file transfer service ectrans please also create an association. The easiest way is to visit the ECaccess Member State Gateway website (e.g. msgatway.ecmwf.int) and follow the instructions in the short ECaccess Presentation (page 17 ff.). Additional documentation can be found on the ECMWF ectrans site.

After everything is set up you have to create an ecaccess certificate to be able to send and receive files from and to the ECMWF server. You can do this by using the ecaccess-certificate-create command on the gateway server. You will be prompted for your ECMWF member state user name and a password (which will be usually generated by a Token). This certificate has to be re-newed periodically (every 7 days).

$ ecaccess-certificate-create
Please enter your user-id: example_username
Your passcode: ***

Flex_extract will be run on an ECMWF server which makes the setup the same as for the remote mode. In the setup.sh script :ref:`[ref] <ref-setup>`, select the Makefile.gfortran for the CONVERT2 Fortran program and the ECMWF server (target) you would like to use. The job script, send to the job queue via the ECaccess software, selects again automatically the correct libraries from the module system. For enableing the file transfer you have to set the ECUID, ECGID, GATEWAY and DESTINATION parameter values.

.. code-block:: bash
    :caption: 'Example settings for a gateway installation.'
    :name: setup.sh
    # -----------------------------------------------------------------
    # AVAILABLE COMMANDLINE ARGUMENTS TO SET
    #
    # THE USER HAS TO SPECIFY THESE PARAMETER
    #
    TARGET='ecgate'
    MAKEFILE='Makefile.gfortran'
    ECUID='uid'
    ECGID='gid'
    GATEWAY='server.example.edu'
    DESTINATION='example@genericSftp'
    INSTALLDIR=None
    JOB_TEMPLATE='job.template'
    CONTROLFILE='CONTROL_EA5'

Afterwards, type:

$ ./setup.sh

to start the installation. You should see the following output at the command line.

# Output of setup.sh
Create tarball ...
Job compilation script has been submitted to ecgate for installation in ${HOME}/flex_extract_vX.X
You should get an email with subject "flexcompile" within the next few minutes!

The email content should look like this with a "SUCCESS" statement in the last line:

gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp phgrreal.f
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp grphreal.f
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp ftrafo.f
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I. -I/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/include -g rwGRIB2.f90
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I. -I/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/include -g posnam.f
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I. -I/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/include -g preconvert.f90
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -g -O3 -o ./CONVERT2 ftrafo.o phgrreal.o grphreal.o rwGRIB2.o posnam.o preconvert.o -L/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/lib -leccodes_f90 -leccodes -ljasper -lpthread -L/usr/local/apps/jasper/1.900.1/LP64/lib -ljasper -lm -L/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -Wl,-rpath,/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -lemos.R64.D64.I32 -L/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -lfftw3
-rwxr-x---. 1 USER at 353134 May 23 12:27 CONVERT2
SUCCESS!

Local Mode

Local Environment Requirements

For the local access modes member and public there is no difference in the installation process.

The environment on your local system has to provide these software packages and libraries, since all extraction and preparation is done at the local system:

Python part	Fortran part
Python3 numpy genshi eccodes for python ecmwf-api-client (everything except ERA5) cdsapi (just for ERA5)	gfortran fftw3 eccodes emoslib

Prepare Local Environment

# On a Linux Debian or Ubuntu system do
# (if not already available):
sudo apt -PV install python3
sudo apt -PV install pip
sudo apt -PV install gfortran
sudo apt -PV install fftw3-dev
sudo apt -PV install libeccodes-dev
sudo apt -PV install libemos-dev
sudo apt -PV install python3-eccodes
pip install genshi
pip install numpy
pip install cdsapi
pip install ecmwf-api-client

Note

In case you would like to use Anaconda Python we recommend you follow the installation instructions of Anaconda Python Installation for Linux and then install the eccodes package from conda with:

conda install conda-forge::python-eccodes

The CDS API (cdsapi) and the ECMWF Web API (ecmwf-api-client) have both to be installed since ERA5 can only be retrieved with the CDS API and all other datasets with the ECMWF Web API.

Both user groups have to provide key's with their credentials for the Web API's in their home directory. Therefore, follow these instructions:

ECMWF Web API:: Go to MARS access website and log in with your credentials. Afterwards, on this site in section "Install ECMWF KEY" the key for the ECMWF Web API should be listed. Please follow the instructions in this section under point 1 (save the key in a file .ecmwfapirc in your home directory).
CDS API:: Go to 'CDS API registration'_ and register there too. Log in at the cdsapi website and follow the instructions at section "Install the CDS API key" to save your credentials in a .cdsapirc file.

Test Local Environment

Check the availability of the system packages with dpkg -s <package-name> | grep Status or rpm -q <package_name>, depending on your system. For example:

$ dpkg -s libeccodes-dev |  grep Status
# or
$ rpm -q libeccodes-dev

Afterwards, check the availability of the python packages by typing python3 in a terminal window and run the import commands in the python shell. If there are no error messages, you succeeded in setting up the environment.

# check in python3 console
import eccodes
import genshi
import numpy
import cdsapi
import ecmwfapi

Test the Web API's

You can start very simple test retrievals for both Web API's to be sure that everything works. This is recommended to minimize the range of possible errors using flex_extract later on.