Gateway mode installation

Gateway environment requirements

The environment on your local system has to provide these software packages and libraries, since flex_extract does only prepare the job script and send it to the ECMWF servers:

• Python3 or Anaconda Python3
• numpy
• genshi

Prepare gateway environment

The easiest way to install all required packages is to use the package management system of your Linux distribution. To do so, it is necessary to use a user with admin rights. The installation was tested on a GNU/Linux Debian buster and an Ubuntu 18.04 Bionic Beaver system.

# On a Linux Debian or Ubuntu system do
# (if not already available):
apt-get install python3
apt-get install pip
apt-get install genshi
apt-get install numpy

Test gateway environment

Check the availability of the python packages by typing python3 in a terminal window and run the import commands in the python shell. If there are no error messages, you succeeded in setting up the environment.

# check in python3 console
import genshi
import numpy

Gateway installation

In this mode, access to the ECMWF computing and archiving facilities is enabled through an ECaccess gateway server on a local member state server. The ECaccess framework is necessary to interactively submit jobs to the ECMWF batch system and to transfer files between ECMWF and local gateway server.

Please see ECMWF’s instructions on gateway server to establish the gateway server if not already in place. Additionally, to be able to use the Ecaccess file transfer service ectrans please also create an association. The easiest way is to visit the ECaccess Member State Gateway website (e.g. msgatway.ecmwf.int) and follow the instructions in the short ECaccess Presentation (page 17 ff.). Additional documentation can be found on the ECMWF ectrans site.

After everything is set up you have to create an ecaccess certificate to be able to send and receive files from and to the ECMWF server. You can do this by using the ecaccess-certificate-create command on the gateway server. You will be prompted for your ECMWF member state user name and a password (which will be usually generated by a Token). This certificate has to be re-newed periodically (every 7 days).

$ ecaccess-certificate-create
Please enter your user-id: example_username
Your passcode: ***

Flex_extract will be run on an ECMWF server which makes the setup the same as for the remote mode. In the setup.sh script [ref], select the Makefile.gfortran for the CONVERT2 Fortran program and the ECMWF server (target) you would like to use. The job script, send to the job queue via the ECaccess software, selects again automatically the correct libraries from the module system. For enableing the file transfer you have to set the ECUID, ECGID, GATEWAY and DESTINATION parameter values.

‘Example settings for a gateway installation.’

# -----------------------------------------------------------------
# AVAILABLE COMMANDLINE ARGUMENTS TO SET
#
# THE USER HAS TO SPECIFY THESE PARAMETER
#
TARGET='ecgate'
MAKEFILE='Makefile.gfortran'
ECUID='uid'
ECGID='gid'
GATEWAY='server.example.edu'
DESTINATION='example@genericSftp'
INSTALLDIR=None
JOB_TEMPLATE='job.template'
CONTROLFILE='CONTROL_EA5'

Afterwards, type:

$ ./setup.sh

to start the installation. You should see the following output at the command line.

# Output of setup.sh
Create tarball ...
Job compilation script has been submitted to ecgate for installation in ${HOME}/flex_extract_vX.X
You should get an email with subject "flexcompile" within the next few minutes!

The email content should look like this with a “SUCCESS” statement in the last line:

gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp phgrreal.f
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp grphreal.f
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp ftrafo.f
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I. -I/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/include -g rwGRIB2.f90
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I. -I/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/include -g posnam.f
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I. -I/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/include -g preconvert.f90
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -g -O3 -o ./CONVERT2 ftrafo.o phgrreal.o grphreal.o rwGRIB2.o posnam.o preconvert.o -L/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/eccodes/2.12.0/GNU/6.3.0/lib -leccodes_f90 -leccodes -ljasper -lpthread -L/usr/local/apps/jasper/1.900.1/LP64/lib -ljasper -lm -L/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -Wl,-rpath,/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -lemos.R64.D64.I32 -L/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -lfftw3
-rwxr-x---. 1 USER at 353134 May 23 12:27 CONVERT2
SUCCESS!