source:
flex_extract.git/For_developers/Sphinx/Source/Installation/local.rst
@
47be2684
Last change on this file since 47be2684 was 47be2684, checked in by Leopold Haimberger <leopold.haimberger@…>, 4 years ago | |
---|---|
|
|
File size: 17.7 KB |
Local mode installation
.. toctree:: :hidden: :maxdepth: 2
Local mode - dependencies
The installation is the same for the access modes member and public.
The environment on your local system has to provide the following software packages and libraries, since the preparation of the extraction and the post-processing is done on the local machine:
Python code | Fortran code |
|
Preparing the local environment
The easiest way to install all required packages is to use the package management system of your Linux distribution which requires admin rights. The installation was tested on a Debian GNU/Linux buster and an Ubuntu 18.04 Bionic Beaver system.
# On a Debian or Debian-derived (e. g. Ubuntu) system, # you may use the following commands (or equivalent commands of your preferred package manager): # (if respective packages are not already available): apt-get install python3 (usually already available on GNU/Linux systems) apt-get install python3-eccodes apt-get install python3-genshi apt-get install python3-numpy apt-get install gfortran apt-get install fftw3-dev apt-get install libeccodes-dev apt-get install libemos-dev # Some of these packages will pull in further packages as dependencies. # This is fine, and some are even needed by ``flex_extract''. # As currently the CDS and ECMWF API packages are not available as Debian packages, # they need to be installed outside of the Debian (Ubuntu etc.) package management system. # The recommended way is: apt-get install pip pip install cdsapi pip install ecmwf-api-client
Note
If you are using Anaconda Python, we recommend to follow the installation instructions of Anaconda Python Installation for Linux and then install the eccodes package from conda with:
conda install conda-forge::python-eccodes
The CDS API (cdsapi) is required for ERA5 data and the ECMWF Web API (ecmwf-api-client) for all other public datasets.
Note
Since public users currently don't have access to the full ERA5 dataset, they can skip the installation of the CDS API.
Both user groups have to provide keys with their credentials for the Web APIs in their home directory, following these instructions:
- ECMWF Web API:
- Go to the MARS access website and log in with your credentials. Afterwards, go to the section "Install ECMWF KEY", where the key for the ECMWF Web API should be listed. Please follow the instructions in this section under 1 (save the key in a file .ecmwfapirc in your home directory).
- CDS API:
- Go to CDS API registration and register there, too. Log in on the cdsapi website and follow the instructions in the section "Install the CDS API key" to save your credentials in file .cdsapirc.
Testing the local environment
Check the availability of the python packages by typing python3 in a terminal window and run the import commands in the python shell:
# check in python3 console import eccodes import genshi import numpy import cdsapi import ecmwfapi
If there are no error messages, you succeeded in setting up the environment.
Testing the Web APIs
You can start very simple test retrievals for both Web APIs to be sure that everything works. This is recommended to minimise the range of possible errors using flex_extract later on.
ECMWF Web API
Please use this Python code snippet as a Member user: | Please use this Python code snippet as a Public user: |
from ecmwfapi import ECMWFService server = ECMWFService('mars') server.retrieve({ 'stream' : "oper", 'levtype' : "sfc", 'param' : "165.128/166.128/167.128", 'dataset' : "interim", 'step' : "0", 'grid' : "0.75/0.75", 'time' : "00/06/12/18", 'date' : "2014-07-01/to/2014-07-31", 'type' : "an", 'class' : "ei", 'target' : "download_erainterim_ecmwfapi.grib" }) |
from ecmwfapi import ECMWFDataServer server = ECMWFDataServer() server.retrieve({ 'stream' : "enda", 'levtype' : "sfc", 'param' : "165.128/166.128/167.128", 'dataset' : "cera20c", 'step' : "0", 'grid' : "1./1.", 'time' : "00/06/12/18", 'date' : "2000-07-01/to/2000-07-31", 'type' : "an", 'class' : "ep", 'target' : "download_cera20c_ecmwfapi.grib" }) |
CDS API
Extraction of ERA5 data via CDS API might take time as currently there is a high demand for ERA5 data. Therefore, as a simple test for the API just retrieve pressure-level data (even if that is NOT what we need for FLEXPART), as they are stored on disk and don't need to be retrieved from MARS (which is the time-consuming action):
Please use the following Python code snippet to retrieve a small sample of ERA5 pressure level data:
import cdsapi c = cdsapi.Client() c.retrieve("reanalysis-era5-pressure-levels", { "variable": "temperature", "pressure_level": "1000", "product_type": "reanalysis", "year": "2008", "month": "01", "day": "01", "time": "12:00", "format": "grib" }, "download_cdsapi.grib")
If you know that your CDS API works, you can try to extract some data from MARS.
Please use the following Python code snippet to retrieve a small ERA5 data sample as a member-state user! The Public user do not have access to the full ERA5 dataset!
import cdsapi c = cdsapi.Client() c.retrieve('reanalysis-era5-complete', { 'class' : 'ea', 'expver' : '1', 'stream' : 'oper', 'type' : 'fc', 'step' : '3/to/12/by/3', 'param' : '130.128', 'levtype' : 'ml', 'levelist': '135/to/137', 'date' : '2013-01-01', 'time' : '06/18', 'area' : '50/-5/40/5', 'grid' : '1.0/1.0', 'format' : 'grib', }, 'download_era5_cdsapi.grib')
Local installation
The Fortran program called calc_etadot will be compiled during the installation process. A suitable makefile (makefile_local_gfortran) for the compilation is set by default. This may be overwritten by the MAKEFILE parameter in setup_local.sh.
However, you may have to adapt the makefile for your environment (the current default makefile works on Debian stretch and similar GNU/Linux distributions). If you use a new name for it, you will have to insert it into setup_local.sh For details on the makefile and how to adapt them, see :ref:`Fortran Makefile <ref-convert>`.
In the root directory of flex_extract, open the setup_local.sh script with an editor and adapt the installation parameters in the section labelled with "AVAILABLE COMMANDLINE ARGUMENTS TO SET" as shown below:
.. code-block:: bash :caption: 'Example settings for a local installation.' :name: setup_local.sh ... # ----------------------------------------------------------------- # AVAILABLE COMMANDLINE ARGUMENTS TO SET # # THE USER HAS TO SPECIFY THESE PARAMETER # TARGET='local' MAKEFILE=<name_of_your_makefile> ECUID=None ECGID=None GATEWAY=None DESTINATION=None INSTALLDIR=None SYSINSTALLDIR=None JOB_TEMPLATE='' CONTROLFILE='CONTROL_EA5' ...
Afterwards, type:
$ ./setup_local.sh
to start the installation. You should see the following standard output.
# Output of setup_local.sh WARNING: installdir has not been specified flex_extract will be installed in here by compiling the Fortran source in <path-to-flex_extract>/flex_extract_v7.1/Source/Fortran Install flex_extract_v7.1 software at local in directory <path-to-flex_extract>/flex_extract_v7.1 Using makefile: makefile_local_gfortran gfortran -O3 -march=native -Bstatic -leccodes_f90 -leccodes -Bdynamic -lm -lemosR64 -I. -I/usr/local/include -fdefault-real-8 -fopenmp -fconvert=big-endian -c ./rwgrib2.f90 gfortran -O3 -march=native -Bstatic -leccodes_f90 -leccodes -Bdynamic -lm -lemosR64 -I. -I/usr/local/include -fdefault-real-8 -fopenmp -fconvert=big-endian -c ./phgrreal.f90 gfortran -O3 -march=native -Bstatic -leccodes_f90 -leccodes -Bdynamic -lm -lemosR64 -I. -I/usr/local/include -fdefault-real-8 -fopenmp -fconvert=big-endian -c ./grphreal.f90 gfortran -O3 -march=native -Bstatic -leccodes_f90 -leccodes -Bdynamic -lm -lemosR64 -I. -I/usr/local/include -fdefault-real-8 -fopenmp -fconvert=big-endian -c ./ftrafo.f90 gfortran -O3 -march=native -Bstatic -leccodes_f90 -leccodes -Bdynamic -lm -lemosR64 -I. -I/usr/local/include -fdefault-real-8 -fopenmp -fconvert=big-endian -c ./calc_etadot.f90 gfortran -O3 -march=native -Bstatic -leccodes_f90 -leccodes -Bdynamic -lm -lemosR64 -I. -I/usr/local/include -fdefault-real-8 -fopenmp -fconvert=big-endian -c ./posnam.f90 gfortran rwgrib2.o calc_etadot.o ftrafo.o grphreal.o posnam.o phgrreal.o -o calc_etadot_fast.out -O3 -march=native -Bstatic -leccodes_f90 -leccodes -Bdynamic -lm -ljasper -lemosR64 -fopenmp ln -sf calc_etadot_fast.out calc_etadot lrwxrwxrwx. 1 <username> tmc 20 Aug 12 10:59 ./calc_etadot -> calc_etadot_fast.out
Local system installation
There is also the possibility to seperate the software executables from the user application files. For doing so, the TARGET parameter can be set to "syslocal" and the additional parameter SYSINSTALLDIR in the setup_local.sh has to be specified. This new path will contain the Fortran and Python executables of flex_extract. If you chose to set a system path for this, please start the script as root. The user directory for flex_extract, which basically consists of everything except the Source directory, will be installed into INSTALLDIR. In this installation mode, an extra (hidden) .setup.rc file will be generated in the Run directory. It contains the necessary settings for running the local system mode.
Despite the change in the TARGET parameter, the meaning of INSTALLDIR and the additional SYSINSTALLDIR, the steps for installation are the same as in the local installation mentioned above. For example, modify the setup_local.sh file as follows:
.. code-block:: bash :caption: 'Example settings for a local system installation.' :name: setup_local.sh ... # ----------------------------------------------------------------- # AVAILABLE COMMANDLINE ARGUMENTS TO SET # # THE USER HAS TO SPECIFY THESE PARAMETER # TARGET='syslocal' MAKEFILE=<name_of_your_makefile> ECUID=None ECGID=None GATEWAY=None DESTINATION=None INSTALLDIR=$HOME SYSINSTALLDIR=/usr/bin/ JOB_TEMPLATE='' CONTROLFILE='CONTROL_EA5' ...
Afterwards, type:
$ sudo ./setup_local.sh
to start the installation. You should see the following standard output.
Flex_extract will be installed in <$HOME> Install flex_extract_v7.1.2 software as syslocal in directory /usr/bin/flex_extract_v7.1.2 Using makefile: makefile_local_gfortran /usr/local/bin/gfortran -O3 -march=native -L/usr/local/lib64/ -leccodes_f90 -leccodes -lm -lemosR64 -I. -I/usr/local/include/ -fdefault-real-8 -fopenmp -fconvert=big-endian -c ./rwgrib2.f90 /usr/local/bin/gfortran -O3 -march=native -L/usr/local/lib64/ -leccodes_f90 -leccodes -lm -lemosR64 -I. -I/usr/local/include/ -fdefault-real-8 -fopenmp -fconvert=big-endian -c ./phgrreal.f90 /usr/local/bin/gfortran -O3 -march=native -L/usr/local/lib64/ -leccodes_f90 -leccodes -lm -lemosR64 -I. -I/usr/local/include/ -fdefault-real-8 -fopenmp -fconvert=big-endian -c ./grphreal.f90 /usr/local/bin/gfortran -O3 -march=native -L/usr/local/lib64/ -leccodes_f90 -leccodes -lm -lemosR64 -I. -I/usr/local/include/ -fdefault-real-8 -fopenmp -fconvert=big-endian -c ./ftrafo.f90 /usr/local/bin/gfortran -O3 -march=native -L/usr/local/lib64/ -leccodes_f90 -leccodes -lm -lemosR64 -I. -I/usr/local/include/ -fdefault-real-8 -fopenmp -fconvert=big-endian -c ./calc_etadot.f90 /usr/local/bin/gfortran -O3 -march=native -L/usr/local/lib64/ -leccodes_f90 -leccodes -lm -lemosR64 -I. -I/usr/local/include/ -fdefault-real-8 -fopenmp -fconvert=big-endian -c ./posnam.f90 /usr/local/bin/gfortran rwgrib2.o calc_etadot.o ftrafo.o grphreal.o posnam.o phgrreal.o -o calc_etadot_fast.out -O3 -march=native -L/usr/local/lib64/ -leccodes_f90 -leccodes -lm -lemosR64 -fopenmp ln -sf calc_etadot_fast.out calc_etadot lrwxrwxrwx. 1 <username> tmc 20 27. Okt 23:25 ./calc_etadot -> calc_etadot_fast.out SUCCESS: INSTALLATION FINISHED!
You can now change into the user directory specified in INSTALLDIR and find a flex_extract_vX.X directory there.