*********************** Local mode installation *********************** .. role:: underline :class: underline .. toctree:: :hidden: :maxdepth: 2 .. _Python 3: https://docs.python.org/3/ .. _Python3: https://www.python.org/downloads/ .. _Anaconda Python3: https://www.anaconda.com/distribution/#download-section .. _numpy: http://www.numpy.org/ .. _ecmwf-api-client: https://confluence.ecmwf.int/display/WEBAPI/ECMWF+Web+API+Home .. _cdsapi: https://cds.climate.copernicus.eu/api-how-to .. _genshi: https://genshi.edgewall.org/ .. _eccodes for python: https://packages.debian.org/sid/python3-eccodes .. _eccodes for conda: https://anaconda.org/conda-forge/eccodes .. _gfortran: https://gcc.gnu.org/wiki/GFortran .. _fftw3: http://www.fftw.org .. _eccodes: https://software.ecmwf.int/wiki/display/ECC .. _emoslib: https://software.ecmwf.int/wiki/display/EMOS/Emoslib .. _member state: https://www.ecmwf.int/en/about/who-we-are/member-states .. _registration form: https://apps.ecmwf.int/registration/ .. _CDS API registration: https://cds.climate.copernicus.eu/user/register .. _ECMWF ectrans site: https://confluence.ecmwf.int/display/ECAC/Unattended+file+transfer+-+ectrans .. _ECaccess Presentation: https://confluence.ecmwf.int/download/attachments/45759146/ECaccess.pdf .. _ECMWF's instructions on gateway server: https://confluence.ecmwf.int/display/ECAC/ECaccess+Home .. _Computing Representative: https://www.ecmwf.int/en/about/contact-us/computing-representatives .. _MARS access: https://confluence.ecmwf.int//display/WEBAPI/Access+MARS .. _download section: https://www.flexpart.eu/downloads .. _ref-local-mode: .. _ref-req-local: Local environment requirements ============================== For the local access modes **member** and **public** there is no difference in the installation process. The environment on your local system has to provide these software packages and libraries, since all extraction and preparation is done at the local system: +------------------------------------------------+-----------------+ | Python part | Fortran part | +------------------------------------------------+-----------------+ | * `Python3`_ | * `gfortran`_ | | * `numpy`_ | * `fftw3`_ | | * `genshi`_ | * `eccodes`_ | | * `eccodes for python`_ | * `emoslib`_ | | * `ecmwf-api-client`_ (everything except ERA5) | | | * `cdsapi`_ (just for ERA5) | | +------------------------------------------------+-----------------+ .. _ref-prep-local: Prepare local environment ========================= The easiest way to install all required packages is to use the package management system of your Linux distribution. To do so, it is necessary to use a user with admin rights. The installation was tested on a *Debian GNU/Linux buster/sid* and an *Ubuntu 18.04 Bionic Beaver* system. .. code-block:: sh # On a Linux Debian or Ubuntu system do # (if not already available): apt-get install python3 (usually available on normal Linux systems) apt-get install pip apt-get install gfortran apt-get install fftw3-dev apt-get install libeccodes-dev apt-get install libemos-dev apt-get install python3-eccodes apt-get install genshi apt-get install numpy pip install cdsapi pip install ecmwf-api-client .. note:: In case you would like to use Anaconda Python we recommend you follow the installation instructions of `Anaconda Python Installation for Linux `_ and then install the ``eccodes`` package from ``conda`` with: .. code-block:: bash conda install conda-forge::python-eccodes The CDS API (cdsapi) and the ECMWF Web API (ecmwf-api-client) have both to be installed since ERA5 can only be retrieved with the ``CDS API`` and all other datasets with the ``ECMWF Web API``. .. note:: Since **public users** currently don't have access to the full *ERA5* dataset they can skip the installation of the ``CDS API``. Both user groups have to provide key's with their credentials for the Web API's in their home directory. Therefore, follow these instructions: ECMWF Web API: Go to `MARS access`_ website and log in with your credentials. Afterwards, on this site in section "Install ECMWF KEY" the key for the ECMWF Web API should be listed. Please follow the instructions in this section under point 1 (save the key in a file `.ecmwfapirc` in your home directory). CDS API: Go to 'CDS API registration'_ and register there too. Log in at the `cdsapi`_ website and follow the instructions at section "Install the CDS API key" to save your credentials in a `.cdsapirc` file. .. _ref-test-local: Test local environment ====================== Check the availability of the system packages with ``dpkg -s | grep Status`` or ``rpm -q ``, depending on your system. For example: .. code-block:: sh $ dpkg -s libeccodes-dev | grep Status # or $ rpm -q libeccodes-dev Afterwards, check the availability of the python packages by typing ``python3`` in a terminal window and run the ``import`` commands in the python shell. If there are no error messages, you succeeded in setting up the environment. .. code-block:: python # check in python3 console import eccodes import genshi import numpy import cdsapi import ecmwfapi Test the Web API's ------------------ You can start very simple test retrievals for both Web API's to be sure that everything works. This is recommended to minimize the range of possible errors using ``flex_extract`` later on. ECMWF Web API ^^^^^^^^^^^^^ +----------------------------------------------------------+----------------------------------------------------------+ |Please use this piece of python code for **Member user**: |Please use this piece of python code for **Public user**: | +----------------------------------------------------------+----------------------------------------------------------+ |.. code-block:: python |.. code-block:: python | | | | | from ecmwfapi import ECMWFService | from ecmwfapi import ECMWFDataServer | | | | | server = ECMWFService('mars') | server = ECMWFDataServer() | | | | | server.retrieve({ | server.retrieve({ | | 'stream' : "oper", | 'stream' : "enda", | | 'levtype' : "sfc", | 'levtype' : "sfc", | | 'param' : "165.128/166.128/167.128", | 'param' : "165.128/166.128/167.128", | | 'dataset' : "interim", | 'dataset' : "cera20c", | | 'step' : "0", | 'step' : "0", | | 'grid' : "0.75/0.75", | 'grid' : "1./1.", | | 'time' : "00/06/12/18", | 'time' : "00/06/12/18", | | 'date' : "2014-07-01/to/2014-07-31", | 'date' : "2000-07-01/to/2000-07-31", | | 'type' : "an", | 'type' : "an", | | 'class' : "ei", | 'class' : "ep", | | 'target' : "download_erainterim_ecmwfapi.grib" | 'target' : "download_cera20c_ecmwfapi.grib" | | }) | }) | +----------------------------------------------------------+----------------------------------------------------------+ CDS API ^^^^^^^ Since ERA5 extraction with CDS API might take some time due to the very high number of requests, you can start by retrieving some online stored pressure levels (not from MARS). This is usually much faster and gives a quick result to find out if the web API works: Please use this piece of python code to retrieve a small sample of *ERA5* pressure levels: .. code-block:: python import cdsapi c = cdsapi.Client() c.retrieve("reanalysis-era5-pressure-levels", { "variable": "temperature", "pressure_level": "1000", "product_type": "reanalysis", "year": "2008", "month": "01", "day": "01", "time": "12:00", "format": "grib" }, "download_cdsapi.grib") Afterwards, when you know that the CDS API generally works, you can try to extract some data from the MARS archive. From the latest experience we know that this can take a while. .. **Member user** Please use this piece of python code to retrieve a small *ERA5* data sample as a **member user**! The **public user** doesn't have access to the full *ERA5* dataset! .. code-block:: python import cdsapi c = cdsapi.Client() c.retrieve('reanalysis-era5-complete', { 'class' : 'ea', 'expver' : '1', 'stream' : 'oper', 'type' : 'fc', 'step' : '3/to/12/by/3', 'param' : '130.128', 'levtype' : 'ml', 'levelist': '135/to/137', 'date' : '2013-01-01', 'time' : '06/18', 'area' : '50/-5/40/5', 'grid' : '1.0/1.0', 'format' : 'grib', }, 'download_era5_cdsapi.grib') .. ********************** COMMENTED OUT FOR FUTURE ********************** PUBLIC RETRIEVAL IS CURRENTLY NOT ACCESSIBLE **Public user** Please use this piece of python code: .. code-block:: python import cdsapi c = cdsapi.Client() c.retrieve('reanalysis-era5-complete', { 'class' : 'ea', 'dataset' : 'era5', 'expver' : '1', 'stream' : 'oper', 'type' : 'fc', 'step' : '3/to/12/by/3', 'param' : '130.128', 'levtype' : 'ml', 'levelist': '135/to/137', 'date' : '2013-01-01', 'time' : '06/18', 'area' : '50/-5/40/5', 'grid' : '1.0/1.0', 'format' : 'grib', }, 'download_era5_cdsapi.grib') .. _ref-install-local: Local installation ================== First prepare the Fortran ``Makefile`` for your environment and set it in the ``setup.sh`` script. (See section :ref:`ref-convert` for information on the Fortran program.) ``flex_extract`` has already two ``Makefiles`` prepared for te ``gfortran`` and the ``ifort`` compiler: * Makefile.local.gfortran * Makefile.local.ifort They can be found in the path ``flex_extract_vX.X/source/fortran``, where ``vX.X`` should be substituted with the current version number. .. caution:: It is necessary to adapt **ECCODES_INCLUDE_DIR** and **ECCODES_LIB** in these ``Makefiles``. So starting from the root directory of ``flex_extract``, go to the ``Fortran`` source directory and open the ``Makefile`` of your choice to modify with an editor of your choice. We use the ``nedit`` in this case. .. code-block:: bash cd flex_extract_vX.X/source/fortran nedit Makefile.local.gfortran Edit the pathes to the ``eccodes`` library on your local machine. .. caution:: This can vary from system to system. It is suggested to use a command like .. code-block:: bash # for the ECCODES_INCLUDE_DIR path do: $ dpkg -L libeccodes-dev | grep eccodes.mod # for the ECCODES_LIB path do: $ dpkg -L libeccodes-dev | grep libeccodes.so to find out the path to the ``eccodes`` library. Substitute these paths in the ``Makefile`` for parameters **ECCODES_INCLUDE_DIR** and **ECCODES_LIB** and save it. .. code-block:: bash # these are the paths on a current Debian 10 Testing system (May 2019) ECCODES_INCLUDE_DIR=/usr/lib/x86_64-linux-gnu/fortran/gfortran-mod-15/ ECCODES_LIB= -L/usr/lib -leccodes_f90 -leccodes -lm The Fortran program called ``CONVERT2`` will be compiled during the installation process to get an executable. Therefore the ``Makefile`` has to be set in the ``setup.sh`` script. In the root directory of ``flex_extract``, open the ``setup.sh`` script and adapt the installation parameters in the section labelled with "AVAILABLE COMMANDLINE ARGUMENTS TO SET" like shown below. .. code-block:: bash :caption: 'Example settings for a local installation.' :name: setup.sh ... # ----------------------------------------------------------------- # AVAILABLE COMMANDLINE ARGUMENTS TO SET # # THE USER HAS TO SPECIFY THESE PARAMETER # TARGET='local' MAKEFILE='Makefile.local.gfortran' ECUID=None ECGID=None GATEWAY=None DESTINATION=None INSTALLDIR=None JOB_TEMPLATE='job.template' CONTROLFILE='CONTROL_EA5' ... Afterwards, type: .. code-block:: bash $ ./setup.sh to start the installation. You should see the following output at the command line. .. code-block:: bash # Output of setup.sh WARNING: installdir has not been specified flex_extract will be installed in here by compiling the Fortran source in /raid60/nas/tmc/Anne/Interpolation/flexextract/flex_extract_v7.1/source/fortran Install flex_extract_v7.1 software at local in directory /raid60/nas/tmc/Anne/Interpolation/flexextract/flex_extract_v7.1 Using makefile: Makefile.local.gfortran gfortran -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore -ffixed-line-length-132 -fopenmp -fconvert=big-endian -c -g -O3 -fopenmp phgrreal.f gfortran -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore -ffixed-line-length-132 -fopenmp -fconvert=big-endian -c -g -O3 -fopenmp grphreal.f gfortran -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore -ffixed-line-length-132 -fopenmp -fconvert=big-endian -c -g -O3 -fopenmp ftrafo.f gfortran -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore -ffixed-line-length-132 -fopenmp -fconvert=big-endian -c -O3 -fopenmp -I. -I/usr/local/gcc-4.9.3/grib_api-1.14.3/include -O3 rwGRIB2.f90 gfortran -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore -ffixed-line-length-132 -fopenmp -fconvert=big-endian -c -O3 -fopenmp -I. -I/usr/local/gcc-4.9.3/grib_api-1.14.3/include -O3 posnam.f gfortran -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore -ffixed-line-length-132 -fopenmp -fconvert=big-endian -c -O3 -fopenmp -I. -I/usr/local/gcc-4.9.3/grib_api-1.14.3/include -O3 preconvert.f90 gfortran -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore -ffixed-line-length-132 -fopenmp -fconvert=big-endian -O3 -O3 -fopenmp -o ./CONVERT2 ftrafo.o phgrreal.o grphreal.o rwGRIB2.o posnam.o preconvert.o -L/usr/local/gcc-4.9.3/grib_api-1.14.3/lib -Bstatic -lgrib_api_f77 -lgrib_api_f90 -lgrib_api -Bdynamic -lm -ljasper -lemosR64 -rwxrwxr-x. 1 philipa8 tmc 282992 May 23 22:27 ./CONVERT2