Local mode - dependencies

The installation is the same for the access modes member and public.

The environment on your local system has to provide these software packages and libraries, since the preparation of the extraction and the post-processing is done on the local machine:

Prepare local environment

The easiest way to install all required packages is to use the package management system of your Linux distribution which requires admin rights. The installation was tested on a Debian GNU/Linux buster and an Ubuntu 18.04 Bionic Beaver system.

# On a Debian or Debian-derived sytem (e. g. Ubuntu) system you may use the following commands (or equivalent commands of your preferred package manager):
# (if not already available):
 apt-get install python3 (usually already available on GNU/Linux systems)
 apt-get install python3-eccodes
 apt-get install python3-genshi
 apt-get install python3-numpy
 apt-get install gfortran
 apt-get install fftw3-dev
 apt-get install libeccodes-dev
 apt-get install libemos-dev
# Some of these packages will pull in further packages as dependencies. This is fine, and some are even needed by ``flex_extract''.

# As currently the CDS and ECMWF API packages are not available as Debian packages, they need to be installed outside of the Debian (Ubuntu etc.) package management system. The recommended way is:
 apt-get install pip
 pip install cdsapi
 pip install ecmwf-api-client

conda install conda-forge::python-eccodes

The CDS API (cdsapi) is required for ERA5 data and the ECMWF Web API (ecmwf-api-client) for all other public datasets.

Both user groups have to provide keys with their credentials for the Web API's in their home directory. Therefore, follow these instructions:

ECMWF Web API:

Go to MARS access website and log in with your credentials. Afterwards, on this site in section "Install ECMWF KEY" the key for the ECMWF Web API should be listed. Please follow the instructions in this section under 1 (save the key in a file .ecmwfapirc in your home directory).

CDS API:

Go to CDS API registration and register there too. Log in at the cdsapi website and follow the instructions at section "Install the CDS API key" to save your credentials in a .cdsapirc file.

Test local environment

Check the availability of the python packages by typing python3 in a terminal window and run the import commands in the python shell. If there are no error messages, you succeeded in setting up the environment.

# check in python3 console
import eccodes
import genshi
import numpy
import cdsapi
import ecmwfapi

from ecmwfapi import ECMWFService
server = ECMWFService('mars')
server.retrieve({
    'stream'    : "oper",
    'levtype'   : "sfc",
    'param'     : "165.128/166.128/167.128",
    'dataset'   : "interim",
    'step'      : "0",
    'grid'      : "0.75/0.75",
    'time'      : "00/06/12/18",
    'date'      : "2014-07-01/to/2014-07-31",
    'type'      : "an",
    'class'     : "ei",
    'target'    : "download_erainterim_ecmwfapi.grib"
})

from ecmwfapi import ECMWFDataServer
server = ECMWFDataServer()
server.retrieve({
    'stream'    : "enda",
    'levtype'   : "sfc",
    'param'     : "165.128/166.128/167.128",
    'dataset'   : "cera20c",
    'step'      : "0",
    'grid'      : "1./1.",
    'time'      : "00/06/12/18",
    'date'      : "2000-07-01/to/2000-07-31",
    'type'      : "an",
    'class'     : "ep",
    'target'    : "download_cera20c_ecmwfapi.grib"
})

import cdsapi
c = cdsapi.Client()
c.retrieve("reanalysis-era5-pressure-levels",
{
"variable": "temperature",
"pressure_level": "1000",
"product_type": "reanalysis",
"year": "2008",
"month": "01",
"day": "01",
"time": "12:00",
"format": "grib"
},
"download_cdsapi.grib")

import cdsapi
c = cdsapi.Client()
c.retrieve('reanalysis-era5-complete',
{
    'class'   : 'ea',
    'expver'  : '1',
    'stream'  : 'oper',
    'type'    : 'fc',
    'step'    : '3/to/12/by/3',
    'param'   : '130.128',
    'levtype' : 'ml',
    'levelist': '135/to/137',
    'date'    : '2013-01-01',
    'time'    : '06/18',
    'area'    : '50/-5/40/5',
    'grid'    : '1.0/1.0',
    'format'  : 'grib',
}, 'download_era5_cdsapi.grib')

Local installation

First prepare the Fortran makefile for your environment and set it in the setup.sh script. (See section :ref:`Fortran Makefile <ref-convert>` for more information.) flex_extract comes with two makefiles prepared for the gfortran compiler. One for the normal use makefile_fast and one for debugging makefile_debug which is usually only resonable for developers.

They assume that eccodes and emoslib are installed as distribution packages and can be found at flex_extract_vX.X/Source/Fortran, where vX.X should be substituted with the current version number.

So starting from the root directory of flex_extract, go to the Fortran source directory and open the makefile of your choice to modify with an editor of your choice. We use the nedit in this case.

cd flex_extract_vX.X/Source/Fortran
nedit makefile_fast

Edit the paths to the eccodes library on your local machine.

# for the ECCODES_INCLUDE_DIR path do:
$ dpkg -L libeccodes-dev | grep eccodes.mod
# for the ECCODES_LIB path do:
$ dpkg -L libeccodes-dev | grep libeccodes.so

Substitute these paths in the makefile for parameters ECCODES_INCLUDE_DIR and ECCODES_LIB and save it.

# these are the paths on a current Debian 10 Testing system (May 2019)
ECCODES_INCLUDE_DIR=/usr/lib/x86_64-linux-gnu/fortran/gfortran-mod-15/
ECCODES_LIB= -L/usr/lib -leccodes_f90 -leccodes -lm

The Fortran program called calc_etadot will be compiled during the installation process.Therefore the name of the makefile to be used needs to be given in setup.sh.

In the root directory of flex_extract, open the setup.sh script and adapt the installation parameters in the section labelled with "AVAILABLE COMMANDLINE ARGUMENTS TO SET" like shown below.

.. code-block:: bash
   :caption: 'Example settings for a local installation.'
   :name: setup.sh
   ...
   # -----------------------------------------------------------------
   # AVAILABLE COMMANDLINE ARGUMENTS TO SET
   #
   # THE USER HAS TO SPECIFY THESE PARAMETER
   #
   TARGET='local'
   MAKEFILE='makefile_fast'
   ECUID=None
   ECGID=None
   GATEWAY=None
   DESTINATION=None
   INSTALLDIR=None
   JOB_TEMPLATE='job.template'
   CONTROLFILE='CONTROL_EA5'
   ...

Afterwards, type:

$ ./setup.sh

to start the installation. You should see the following standard output.

# Output of setup.sh
    WARNING: installdir has not been specified
    flex_extract will be installed in here by compiling the Fortran source in <path-to-flex_extract>/flex_extract_v7.1/Source/Fortran
    Install flex_extract_v7.1 software at local in directory <path-to-flex_extract>/flex_extract_v7.1
    Using makefile: makefile_fast
    gfortran   -O3 -march=native -Bstatic -leccodes_f90 -leccodes -Bdynamic -lm -ljasper -lemosR64 -I. -I/usr/local/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c     ./rwgrib2.f90
    gfortran   -O3 -march=native -Bstatic -leccodes_f90 -leccodes -Bdynamic -lm -ljasper -lemosR64 -I. -I/usr/local/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c     ./phgrreal.f90
    gfortran   -O3 -march=native -Bstatic -leccodes_f90 -leccodes -Bdynamic -lm -ljasper -lemosR64 -I. -I/usr/local/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c     ./grphreal.f90
    gfortran   -O3 -march=native -Bstatic -leccodes_f90 -leccodes -Bdynamic -lm -ljasper -lemosR64 -I. -I/usr/local/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c     ./ftrafo.f90
    gfortran   -O3 -march=native -Bstatic -leccodes_f90 -leccodes -Bdynamic -lm -ljasper -lemosR64 -I. -I/usr/local/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c     ./calc_etadot.f90
    gfortran   -O3 -march=native -Bstatic -leccodes_f90 -leccodes -Bdynamic -lm -ljasper -lemosR64 -I. -I/usr/local/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c     ./posnam.f90
    gfortran  rwgrib2.o calc_etadot.o ftrafo.o grphreal.o posnam.o phgrreal.o -o calc_etadot_fast.out  -O3 -march=native -Bstatic -leccodes_f90 -leccodes -Bdynamic -lm -ljasper -lemosR64 -fopenmp
    ln -sf calc_etadot_fast.out calc_etadot
    lrwxrwxrwx. 1 <username> tmc 20 15. Mär 13:31 ./calc_etadot -> calc_etadot_fast.out