Gateway mode - dependencies

• Python3
• numpy
• genshi

Prepare gateway environment

In this mode, access to the ECMWF computing and archiving facilities is enabled through an ECaccess gateway server on a local member state server. The ECaccess framework is necessary to interactively submit jobs to the ECMWF batch system and to transfer files between ECMWF and local gateway server.

Please see ECMWF's instructions on gateway server to establish the gateway server if not already in place. Additionally, to be able to use the ECaccess file transfer service ectrans it is necessary to create a so-called association. The easiest way is to visit the ECaccess Member State Gateway website (e.g. https://msgatway.ecmwf.int) and follow the instructions in the short ECaccess Presentation (page 17 ff.). Additional documentation can be found on the ECMWF ectrans site.

The next step is to create an ecaccess certificate to be able to send and receive files from and to the ECMWF server. You can do this by using the ecaccess-certificate-create command on the gateway server. You will be prompted for your ECMWF member state user name and a password (generated by a token). This certificate has to be renewed periodically (every 7 days).

$ ecaccess-certificate-create
Please enter your user-id: example_username
Your passcode: ***

The easiest way to install all required packages is to use the package management system of your Linux distribution which required admin rights. The installation was tested under GNU/Linux Debian buster and Ubuntu 18.04 Bionic Beaver.

# On a Debian or Debian-derived sytem (e. g. Ubuntu) system you may use the following commands (or equivalent commands of your preferred package manager):
# (if not already available):
apt-get install python3
apt-get install python3-genshi
apt-get install python3-numpy

Test gateway environment

Check the availability of the python packages by typing python3 in a terminal window and run the import commands in the python shell. If there are no error messages, you succeeded in setting up the environment.

# check in python3 console
import genshi
import numpy

Gateway installation

Flex_extract will be run on an ECMWF server which makes the setup the same as for the remote mode. In the setup.sh script [ref], select the makefile.gfortran for the CONVERT2 Fortran program and the ECMWF server (target) you would like to use. The job script, send to the job queue via the ECaccess software, loads the required modules from the module system. For enabling the file transfer via the gateway, you have to set the ECUID, ECGID, GATEWAY and DESTINATION parameter values.

.. code-block:: bash
    :caption: 'Example settings for a gateway installation.'
    :name: setup.sh
    # -----------------------------------------------------------------
    # AVAILABLE COMMANDLINE ARGUMENTS TO SET
    #
    # THE USER HAS TO SPECIFY THESE PARAMETER
    #
    TARGET='ecgate'
    MAKEFILE='makefile.gfortran'
    ECUID='uid'
    ECGID='gid'
    GATEWAY='localserver.tld'
    DESTINATION='association_name@genericSftp'
    INSTALLDIR=None
    JOB_TEMPLATE='job.template'
    CONTROLFILE='CONTROL_EA5'

Afterwards, type:

$ ./setup.sh

to start the installation. You should see the following on standard output.

# Output of setup.sh
Create tarball ...
Job compilation script has been submitted to ecgate for installation in ${HOME}/flex_extract_vX.X
You should get an email with subject "flexcompile" within the next few minutes!

Flex_extract uses the email address connectd to the user account on ECMWF servers. The email content should look like this with a "SUCCESS" statement in the last line:

gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp phgrreal.f
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp grphreal.f
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp ftrafo.f
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I. -I/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/include -g rwGRIB2.f90
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I. -I/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/include -g posnam.f
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I. -I/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/include -g preconvert.f90
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -g -O3 -o ./CONVERT2 ftrafo.o phgrreal.o grphreal.o rwGRIB2.o posnam.o preconvert.o -L/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/lib -leccodes_f90 -leccodes -ljasper -lpthread -L/usr/local/apps/jasper/1.900.1/LP64/lib -ljasper -lm -L/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -Wl,-rpath,/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -lemos.R64.D64.I32 -L/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -lfftw3
-rwxr-x---. 1 USER at 353134 May 23 12:27 CONVERT2
SUCCESS!