The installation script - `setup.sh`¶

The installation of flex_extract is done by the shell script setup.sh located in the root directory of flex_extract. It calls the top-level Python script install.py which does all the necessary operations to prepare the application environment selected. This includes:

preparing the file ECMWF_ENV with the user credentials for member-state access to ECMWF servers (in remote and gateway mode)
preparation of a compilation Korn-shell script (in remote and gateway mode)
preparation of a job template with user credentials (in remote and gateway mode)
create a tarball of all necessary files
copying the tarball to the target location (depending on application mode and installation path)
submit the compilation script to the batch queue at ECMWF servers (in remote and gateway mode) or just untar the tarball at target location (local mode)
compilation of the Fortran program calc_etadot

The Python installation script install.py has several command line arguments defined in setup.sh, in the section labelled “AVAILABLE COMMANDLINE ARGUMENTS TO SET”. The user has to adapt these parameters according to his/her personal needs. The parameters are listed and described in Installation parameters. The script also does some checks to guarantee that the necessary parameters were set.

After the installation process, some tests can be conducted. They are described in section Test installation.

The following diagram sketches the files and scripts involved in the installation process:

Diagram of data flow during the installation process. Trapezoids are input files with the light blue area being the template files. Round-edge orange boxes are executable files which start the installation process and read the input files. Rectangular green boxes are output files. Light green files are needed only in the remota and gateway mode.¶

Installation parameters¶

Parameter for Installation¶
Parameter	Format	Possible values	Default	Description
TARGET	String	local, ecgate, cca, ccb	None	Defines the flex_extract application mode to be used. Local = local mode; ecgate = Remote or Gateway mode, cca or ccb = Remote or Gateway mode (HPC). Whether the local mode is for public or member state users doesn’t matter here.
MAKEFILE	String	any makefile present in the Fortran source directory	None	Name of the makefile in Source/Fortran directory to be used for compiling the Fortran program.
ECUID	String	e.g. “myecname”	None	The user name (not Linux uid!) at ECMWF. Necessary for access to ECMWF server. (Remote and Gateway mode)
ECGID	String	e.g. “mygroupname”	None	The group name (not Linux gid!) at ECMWF. Necessary for access to ECMWF server. (Remote and Gateway mode)
GATEWAY	String	any server adress	None	The name of the local gateway server. Necessary for transfering files from the ECMWF servers to the local member state gateway server. (Gateway mode)
DESTINATION	String	any destination which is stored at local gateway server	None	The ecaccess association, e.g. myUser@genericSftp. Used for transfering files from the ECMWF servers to the local member state gateway server. (Gateway mode)
INSTALLDIR	String	a full path	$HOME on ECMWF server and current flex_extract root path on local machine	Root path where flex_extract should be installed. It will always be $HOME on ECMWF servers and if not set for the local mode, it will be the current flex_extract root path on local servers.
JOB_TEMPLATE	String	installscript.template	installscript.template	The rudimentary template file to create a batch job template for submission to ECMWF servers. Should not be changed because it is already optimised for ECMWF servers. (Remote and Gateway mode)
CONTROLFILE	String	any CONTROL file	CONTROL_EA5	File containing all CONTROL parameters. Any CONTROL file may be used. Due to interval process it is just necessary that there is a CONTROL file in place which can be opend.

Content of `setup.sh`¶

setup.sh¶

#!/bin/bash
#
# @Author: Anne Philipp
#
# @Date: September, 10 2018
#
# @Description: 
#    This file defines the flex_extract's available installation
#    parameters and puts them together for the call of the actual 
#    python installation script. It also does some checks to 
#    guarantee necessary parameters were set.
#
# @Licence:
#    (C) Copyright 2014-2020.
#
#    SPDX-License-Identifier: CC-BY-4.0
#
#    This work is licensed under the Creative Commons Attribution 4.0
#    International License. To view a copy of this license, visit
#    http://creativecommons.org/licenses/by/4.0/ or send a letter to
#    Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.
#
# -----------------------------------------------------------------
# AVAILABLE COMMANDLINE ARGUMENTS TO SET
#
# THE USER HAS TO SPECIFY THESE PARAMETERS
#
TARGET='ecgate'
MAKEFILE='makefile_ecgate'
ECUID='<username>'
ECGID='<groupID>'
GATEWAY='<gatewayname>'
DESTINATION='<username>@genericSftp'
INSTALLDIR=None
JOB_TEMPLATE=''
CONTROLFILE='CONTROL_EA5'
# -----------------------------------------------------------------
#
# AFTER THIS LINE THE USER DOES NOT HAVE TO CHANGE ANYTHING !!!
#
# -----------------------------------------------------------------

# PATH TO INSTALLATION SCRIPT
script="Source/Python/install.py"

# INITIALIZE EMPTY PARAMETERLIST
parameterlist=""

# CHECK IF ON ECMWF SERVER; 
if [[ $HOST == *"ecgb"* ]] || [[ $HOST == *"cca"* ]] || [[ $HOST == *"ccb"* ]]; then
# LOAD PYTHON3 MODULE
  module load python3
fi 

# DEFAULT PARAMETERLIST
if [ -n "$TARGET" ]; then
  parameterlist=" --target=$TARGET"
else
  echo "ERROR: No installation target specified."
  echo "EXIT WITH ERROR"
  exit
fi

# CHECK FOR MORE PARAMETER 
if [ "$TARGET" == "ecgate" ] || [ "$TARGET" == "cca" ] || [ "$TARGET" == "ccb" ]; then
  # check if necessary Parameters are set
  if [ -z "$ECUID" ] || [ -z "$ECGID" ] || [ "$ECUID" == "<username>" ] || [ "$ECGID" == "<groupID>" ] ; then
    echo "ERROR: At least one of the following parameters are not properly set: ECUID or ECGID!"
    echo "EXIT WITH ERROR"
    exit
  else
    parameterlist+=" --ecuid=$ECUID --ecgid=$ECGID --gateway=$GATEWAY --destination=$DESTINATION"
  fi
  if [ -z "$GATEWAY" ] || [ -z "$DESTINATION" ] || [ "$GATEWAY" == "<gatewayname>" ] || [ "$DESTINATION" == "<username>@genericSftp" ] ; then
    echo "WARNING: Not setting parameters GATEWAY and DESTINATION means there will be no file transfer to local gateway server."
  fi
fi
if [ -n "$MAKEFILE" ]; then
  parameterlist+=" --makefile=$MAKEFILE"
fi
if [ -n "$FLEXPARTDIR" ]; then # not empty
  parameterlist+=" --flexpartdir=$FLEXPARTDIR"
fi
if [ -n "$JOB_TEMPLATE" ]; then
  parameterlist+=" --job_template=$JOB_TEMPLATE"
fi
if [ -n "$CONTROLFILE" ]; then
  parameterlist+=" --controlfile=$CONTROLFILE"
fi

# -----------------------------------------------------------------
# CALL INSTALLATION SCRIPT WITH DETERMINED COMMANDLINE ARGUMENTS

$script $parameterlist

Usage of `install.py` (optional)¶

It is also possible to start the installation process of flex_extract directly from the command line by using the install.py script instead of the wrapper shell script setup.sh. This top-level script is located in flex_extract_vX.X/Source/Python and is executable. With the --help parameter, we see again all possible command line parameters.

install.py --help

usage: install.py [-h] [--target INSTALL_TARGET] [--makefile MAKEFILE]
              [--ecuid ECUID] [--ecgid ECGID] [--gateway GATEWAY]
              [--destination DESTINATION] [--installdir INSTALLDIR]
              [--job_template JOB_TEMPLATE] [--controlfile CONTROLFILE]

Install flex_extract software locally or on ECMWF machines

optional arguments:
  -h, --help            show this help message and exit
  --target INSTALL_TARGET
                        Valid targets: local | ecgate | cca , the latter two
                        are at ECMWF (default: None)
  --makefile MAKEFILE   Name of makefile to compile the Fortran
                        code. (default depends on `target`: local - makefile_local_gfortran,
                        ecgate - makefile_ecgate, cca - makefile_cray)
  --ecuid ECUID         The user id at ECMWF. (default: None)
  --ecgid ECGID         The group id at ECMWF. (default: None)
  --gateway GATEWAY     The IP name (or IP address) of the local gateway server. (default: None)
  --destination DESTINATION
                        The ecaccess association, e.g. myUser@genericSftp
                        (default: None)
  --installdir INSTALLDIR
                        Root directory where flex_extract will be installed
                        to. (default: None)
  --job_template JOB_TEMPLATE
                        The rudimentary template file to create a batch job
                        template for submission to ECMWF servers. (default:
                        job.template)
  --controlfile CONTROLFILE
                        The file with all CONTROL parameters. (default:
                        CONTROL_EA5)

The installation script - setup.sh¶

Installation parameters¶

Content of setup.sh¶

Usage of install.py (optional)¶

The installation script - `setup.sh`¶

Content of `setup.sh`¶

Usage of `install.py` (optional)¶