Local mode installation¶

Local environment requirements¶

For the local access modes member and public there is no difference in the installation process.

The environment on your local system has to provide these software packages and libraries, since all extraction and preparation is done at the local system:

Python part	Fortran part
Python3 numpy genshi eccodes for python ecmwf-api-client (everything except ERA5) cdsapi (just for ERA5)	gfortran fftw3 eccodes emoslib

Prepare local environment¶

The easiest way to install all required packages is to use the package management system of your Linux distribution. To do so, it is necessary to use a user with admin rights. The installation was tested on a Debian GNU/Linux buster/sid and an Ubuntu 18.04 Bionic Beaver system.

# On a Linux Debian or Ubuntu system do
# (if not already available):
apt-get install python3 (usually available on normal Linux systems)
apt-get install pip
apt-get install gfortran
apt-get install fftw3-dev
apt-get install libeccodes-dev
apt-get install libemos-dev
apt-get install python3-eccodes
apt-get install genshi
apt-get install numpy
pip install cdsapi
pip install ecmwf-api-client

Note

In case you would like to use Anaconda Python we recommend you follow the installation instructions of Anaconda Python Installation for Linux and then install the eccodes package from conda with:

conda install conda-forge::python-eccodes

The CDS API (cdsapi) and the ECMWF Web API (ecmwf-api-client) have both to be installed since ERA5 can only be retrieved with the CDS API and all other datasets with the ECMWF Web API.

Note

Since public users currently don’t have access to the full ERA5 dataset they can skip the installation of the CDS API.

Both user groups have to provide key’s with their credentials for the Web API’s in their home directory. Therefore, follow these instructions:

ECMWF Web API:: Go to MARS access website and log in with your credentials. Afterwards, on this site in section “Install ECMWF KEY” the key for the ECMWF Web API should be listed. Please follow the instructions in this section under point 1 (save the key in a file .ecmwfapirc in your home directory).
CDS API:: Go to ‘CDS API registration’_ and register there too. Log in at the cdsapi website and follow the instructions at section “Install the CDS API key” to save your credentials in a .cdsapirc file.

Test local environment¶

Check the availability of the system packages with dpkg -s <package-name> | grep Status or rpm -q <package_name>, depending on your system. For example:

$ dpkg -s libeccodes-dev |  grep Status
# or
$ rpm -q libeccodes-dev

Afterwards, check the availability of the python packages by typing python3 in a terminal window and run the import commands in the python shell. If there are no error messages, you succeeded in setting up the environment.

# check in python3 console
import eccodes
import genshi
import numpy
import cdsapi
import ecmwfapi

Test the Web API’s¶

You can start very simple test retrievals for both Web API’s to be sure that everything works. This is recommended to minimize the range of possible errors using flex_extract later on.

ECMWF Web API¶

Please use this piece of python code for Member user:

Please use this piece of python code for Public user:

from ecmwfapi import ECMWFService

server = ECMWFService('mars')

server.retrieve({
    'stream'    : "oper",
    'levtype'   : "sfc",
    'param'     : "165.128/166.128/167.128",
    'dataset'   : "interim",
    'step'      : "0",
    'grid'      : "0.75/0.75",
    'time'      : "00/06/12/18",
    'date'      : "2014-07-01/to/2014-07-31",
    'type'      : "an",
    'class'     : "ei",
    'target'    : "download_erainterim_ecmwfapi.grib"
})

from ecmwfapi import ECMWFDataServer

server = ECMWFDataServer()

server.retrieve({
    'stream'    : "enda",
    'levtype'   : "sfc",
    'param'     : "165.128/166.128/167.128",
    'dataset'   : "cera20c",
    'step'      : "0",
    'grid'      : "1./1.",
    'time'      : "00/06/12/18",
    'date'      : "2000-07-01/to/2000-07-31",
    'type'      : "an",
    'class'     : "ep",
    'target'    : "download_cera20c_ecmwfapi.grib"
})

CDS API¶

Since ERA5 extraction with CDS API might take some time due to the very high number of requests, you can start by retrieving some online stored pressure levels (not from MARS). This is usually much faster and gives a quick result to find out if the web API works:

Please use this piece of python code to retrieve a small sample of ERA5 pressure levels:

import cdsapi

c = cdsapi.Client()

c.retrieve("reanalysis-era5-pressure-levels",
{
"variable": "temperature",
"pressure_level": "1000",
"product_type": "reanalysis",
"year": "2008",
"month": "01",
"day": "01",
"time": "12:00",
"format": "grib"
},
"download_cdsapi.grib")

Afterwards, when you know that the CDS API generally works, you can try to extract some data from the MARS archive. From the latest experience we know that this can take a while.

Please use this piece of python code to retrieve a small ERA5 data sample as a member user! The public user doesn’t have access to the full ERA5 dataset!

import cdsapi

c = cdsapi.Client()

c.retrieve('reanalysis-era5-complete',
{
    'class'   : 'ea',
    'expver'  : '1',
    'stream'  : 'oper',
    'type'    : 'fc',
    'step'    : '3/to/12/by/3',
    'param'   : '130.128',
    'levtype' : 'ml',
    'levelist': '135/to/137',
    'date'    : '2013-01-01',
    'time'    : '06/18',
    'area'    : '50/-5/40/5',
    'grid'    : '1.0/1.0',
    'format'  : 'grib',
}, 'download_era5_cdsapi.grib')

Local installation¶

First prepare the Fortran Makefile for your environment and set it in the setup.sh script. (See section ref-convert for information on the Fortran program.) flex_extract has already two Makefiles prepared for te gfortran and the ifort compiler:

Makefile.local.gfortran

Makefile.local.ifort

They can be found in the path flex_extract_vX.X/source/fortran, where vX.X should be substituted with the current version number.

Caution

It is necessary to adapt ECCODES_INCLUDE_DIR and ECCODES_LIB in these Makefiles.

So starting from the root directory of flex_extract, go to the Fortran source directory and open the Makefile of your choice to modify with an editor of your choice. We use the nedit in this case.

cd flex_extract_vX.X/source/fortran
nedit Makefile.local.gfortran

Edit the pathes to the eccodes library on your local machine.

Caution

This can vary from system to system. It is suggested to use a command like

# for the ECCODES_INCLUDE_DIR path do:
$ dpkg -L libeccodes-dev | grep eccodes.mod
# for the ECCODES_LIB path do:
$ dpkg -L libeccodes-dev | grep libeccodes.so

to find out the path to the eccodes library.

Substitute these paths in the Makefile for parameters ECCODES_INCLUDE_DIR and ECCODES_LIB and save it.

# these are the paths on a current Debian 10 Testing system (May 2019)
ECCODES_INCLUDE_DIR=/usr/lib/x86_64-linux-gnu/fortran/gfortran-mod-15/
ECCODES_LIB= -L/usr/lib -leccodes_f90 -leccodes -lm

The Fortran program called CONVERT2 will be compiled during the installation process to get an executable. Therefore the Makefile has to be set in the setup.sh script.

In the root directory of flex_extract, open the setup.sh script and adapt the installation parameters in the section labelled with “AVAILABLE COMMANDLINE ARGUMENTS TO SET” like shown below.

‘Example settings for a local installation.’¶

...
# -----------------------------------------------------------------
# AVAILABLE COMMANDLINE ARGUMENTS TO SET
#
# THE USER HAS TO SPECIFY THESE PARAMETER
#
TARGET='local'
MAKEFILE='Makefile.local.gfortran'
ECUID=None
ECGID=None
GATEWAY=None
DESTINATION=None
INSTALLDIR=None
JOB_TEMPLATE='job.template'
CONTROLFILE='CONTROL_EA5'
...

Afterwards, type:

$ ./setup.sh

to start the installation. You should see the following output at the command line.

# Output of setup.sh
WARNING: installdir has not been specified
flex_extract will be installed in here by compiling the Fortran source in /raid60/nas/tmc/Anne/Interpolation/flexextract/flex_extract_v7.1/source/fortran
Install flex_extract_v7.1 software at local in directory /raid60/nas/tmc/Anne/Interpolation/flexextract/flex_extract_v7.1

Using makefile: Makefile.local.gfortran
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp phgrreal.f
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp grphreal.f
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp ftrafo.f
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -fopenmp -I. -I/usr/local/gcc-4.9.3/grib_api-1.14.3/include -O3 rwGRIB2.f90
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -fopenmp -I. -I/usr/local/gcc-4.9.3/grib_api-1.14.3/include -O3 posnam.f
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -fopenmp -I. -I/usr/local/gcc-4.9.3/grib_api-1.14.3/include -O3 preconvert.f90
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -O3 -O3 -fopenmp -o ./CONVERT2 ftrafo.o phgrreal.o grphreal.o rwGRIB2.o posnam.o preconvert.o -L/usr/local/gcc-4.9.3/grib_api-1.14.3/lib -Bstatic  -lgrib_api_f77 -lgrib_api_f90 -lgrib_api -Bdynamic  -lm  -ljasper -lemosR64

-rwxrwxr-x. 1 philipa8 tmc 282992 May 23 22:27 ./CONVERT2