The Installation Script - `setup.sh`¶

The installation of flex_extract is done by the Shell script setup.sh which is located in the root directory of flex_extract. It calls the top-level Python script install.py which does all necessary operations to prepare the selected application environment. This includes:

preparing the file ECMWF_ENV with the user credentials for member state access to ECMWF servers (in remote and gateway mode)
preparation of a compilation Korn-shell script (in remote and gateway mode)
preparation of a job template with user credentials (in remote and gateway mode)
create a tar-ball of all necessary files
copying tar-ball to target location (depending on application mode and installation path)
submit compilation script to batch queue at ECMWF servers (in remote and gateway mode) or just untar tar-ball at target location (local mode)
compilation of the FORTRAN90 program CONVERT2

The Python installation script install.py has a couple of command line arguments which are defined in setup.sh in the section labelled with “AVAILABLE COMMANDLINE ARGUMENTS TO SET”. The user has to adapt these parameters for his personal use. The parameters are listed and described in Installation Parameter. The script also does some checks to guarantee necessary parameters were set.

After the installation process, some tests can be conducted. They are described in section Test installation.

The following diagram sketches the involved files and scripts in the installation process:

Diagram of data flow during the installation process. The trapezoids are input files with the light blue area being the template files. The edge-rounded, orange boxes are the executable files which start the installation process and reads the input files. The rectangular, green boxes are the output files. The light green files are files which are only needed in the remota and gateway mode.¶

Installation Parameter¶

Parameter for Installation¶
Parameter	Format	Possible value range	Default	Description
TARGET	String	local, ecgate, cca	None	Defines which flex_extract application mode will be used. Local = local mode; egate = Remote or Gateway mode, cca = Remote or Gateway mode. Wether the local mode is for public or member state users doesn’t matter during the installation process.
MAKEFILE	String	any Makefile present, according to environment	Makefile.gfortran	Name of the makefile in source/fortran directory to be used for compiling the Fortran program. “Makefile.gfortran” has the configuration for the ecgate environment. For local server versions there are templates “Makefile.local.gfortran” and “Makefile.local.ifort” which have to be adapted (pathes to the eccodes library) by the user.
ECUID	String	e.g. “ecname”	None	The user id at ECMWF. Necessary for access to ECMWF server. (Remote and Gateway mode)
ECGID	String	e.g. “groupname”	None	The group id at ECMWF. Necessary for access to ECMWF server. (Remote and Gateway mode)
GATEWAY	String	any server adress	None	The name of the local gateway server. Necessary for transfering files from the ECMWF servers to the local member state gateway server. (Gateway mode)
DESTINATION	String	any destination which is stored at local gateway server	None	The ecaccess association, e.g. myUser@genericSftp. Used for transfering files from the ECMWF servers to the local member state gateway server. (Gateway mode)
INSTALLDIR	String	a full path	$HOME on ECMWF server and current flex_extract root path on local server	Root path where flex_extract should be installed. If it is not set it will be set to $HOME on ECMWF server and set to the current flex_extract root path on local servers.
JOB_TEMPLATE	String	job.template	job.template	The rudimentary template file to create a batch job template for submission to ECMWF servers. Should not be changed since it is optimized for ECMWF server. (Remote and Gateway mode)
CONTROLFILE	String	any CONTROL file	CONTROL_EA5	The file with all CONTROL parameters. It does not matter which CONTROL file is used. Due to interval process it is just necessary that there is a CONTROL file in place which can be opend.

Content of `setup.sh`¶

setup.sh¶

#!/bin/bash
#
# @Author: Anne Philipp
#
# @Date: September, 10 2018
#
# @Description: 
#    This file defines the flex_extract's available installation
#    parameters and puts them together for the call of the actual 
#    python installation script. It also does some checks to 
#    guarantee necessary parameters were set.
#
# @Licence:
#    (C) Copyright 2014-2019.
#
#    SPDX-License-Identifier: CC-BY-4.0
#
#    This work is licensed under the Creative Commons Attribution 4.0
#    International License. To view a copy of this license, visit
#    http://creativecommons.org/licenses/by/4.0/ or send a letter to
#    Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.
#
# -----------------------------------------------------------------
# AVAILABLE COMMANDLINE ARGUMENTS TO SET
#
# THE USER HAS TO SPECIFY THESE PARAMETERS
#
TARGET='ecgate'
MAKEFILE='makefile_ecgate'
ECUID='km4a'
ECGID='at'
GATEWAY='srvx8.img.univie.ac.at'
DESTINATION='annep@genericSftp'
INSTALLDIR=None
JOB_TEMPLATE=''
CONTROLFILE='CONTROL_EA5'
# -----------------------------------------------------------------
#
# AFTER THIS LINE THE USER DOES NOT HAVE TO CHANGE ANYTHING !!!
#
# -----------------------------------------------------------------

# PATH TO INSTALLATION SCRIPT
script="Source/Python/install.py"

# INITIALIZE EMPTY PARAMETERLIST
parameterlist=""

# DEFAULT PARAMETERLIST
if [ -n "$TARGET" ]; then
  parameterlist=" --target=$TARGET"
else
  echo "ERROR: No installation target specified."
  echo "EXIT WITH ERROR"
  exit
fi

# CHECK FOR MORE PARAMETER 
if [ "$TARGET" == "ecgate" ] || [ "$TARGET" == "cca" ]; then
  # check if necessary Parameters are set
  if [ -z "$ECUID" ] || [ -z "$ECGID" ] || [ -z "$GATEWAY" ] || [ -z "$DESTINATION" ]; then
    echo "ERROR: At least one of the following parameters are not set: ECUID, ECGID, GATEWAY, DESTINATION!"
    echo "EXIT WITH ERROR"
    exit
  else
    parameterlist+=" --ecuid=$ECUID --ecgid=$ECGID --gateway=$GATEWAY --destination=$DESTINATION"
  fi
fi
if [ -n "$MAKEFILE" ]; then
  parameterlist+=" --makefile=$MAKEFILE"
fi
if [ -n "$FLEXPARTDIR" ]; then # not empty
  parameterlist+=" --flexpartdir=$FLEXPARTDIR"
fi
if [ -n "$JOB_TEMPLATE" ]; then
  parameterlist+=" --job_template=$JOB_TEMPLATE"
fi
if [ -n "$CONTROLFILE" ]; then
  parameterlist+=" --controlfile=$CONTROLFILE"
fi

# -----------------------------------------------------------------
# CALL INSTALLATION SCRIPT WITH DETERMINED COMMANDLINE ARGUMENTS

$script $parameterlist

Usage of `install.py` (optional)¶

It is also possible to start the installation process of flex_extract directly from command line by using the install.py script instead of the wrapping Shell script setup.sh. This top-level script is located in flex_extract_vX.X/source/python and is executable. With the help parameter we see again all possible command line parameter.

install.py --help

usage: install.py [-h] [--target INSTALL_TARGET] [--makefile MAKEFILE]
              [--ecuid ECUID] [--ecgid ECGID] [--gateway GATEWAY]
              [--destination DESTINATION] [--installdir INSTALLDIR]
              [--job_template JOB_TEMPLATE] [--controlfile CONTROLFILE]

Install flex_extract software locally or on ECMWF machines

optional arguments:
  -h, --help            show this help message and exit
  --target INSTALL_TARGET
                        Valid targets: local | ecgate | cca , the latter two
                        are at ECMWF (default: None)
  --makefile MAKEFILE   Name of Makefile to use for compiling the Fortran
                        program (default: None)
  --ecuid ECUID         The user id at ECMWF. (default: None)
  --ecgid ECGID         The group id at ECMWF. (default: None)
  --gateway GATEWAY     The name of the local gateway server. (default: None)
  --destination DESTINATION
                        The ecaccess association, e.g. myUser@genericSftp
                        (default: None)
  --installdir INSTALLDIR
                        Root directory where flex_extract will be installed
                        to. (default: None)
  --job_template JOB_TEMPLATE
                        The rudimentary template file to create a batch job
                        template for submission to ECMWF servers. (default:
                        job.template)
  --controlfile CONTROLFILE
                        The file with all CONTROL parameters. (default:
                        CONTROL_EA5)

The Installation Script - setup.sh¶

Installation Parameter¶

Content of setup.sh¶

Usage of install.py (optional)¶

The Installation Script - `setup.sh`¶

Content of `setup.sh`¶

Usage of `install.py` (optional)¶