Auto Generated Documentation

Porgrams

install

This script installs the flex_extract program.

Depending on the selected installation environment (locally or on the ECMWF server ecgate or cca) the program extracts the commandline arguments and the CONTROL file parameter and prepares the corresponding environment. The necessary files are collected in a tar-ball and placed at the target location. There its untared, the environment variables will be set and the Fortran code will be compiled. If the ECMWF environment is selected a job script is prepared and submitted for the remaining configurations after putting the tar-ball to the target ECMWF server.

Type: install.py –help to get information about command line parameters. Read the documentation for usage instructions.

install.check_install_conditions(c)

Checks a couple of necessary attributes and conditions for the installation such as if they exist and contain values. Otherwise set default values.

Parameters:c (ControlFile) – Contains all the parameters of CONTROL file and command line.

install.del_convert_build(src_path)

Clean up the Fortran source directory and remove all build files (e.g. *.o, *.mod and CONVERT2)

Parameters:src_path (str) – Path to the fortran source directory.

install.get_install_cmdline_args()

Decomposes the command line arguments and assigns them to variables. Apply default values for non mentioned arguments.

Returns:args – Contains the commandline arguments from script/program call. Return type:Namespace

install.install_via_gateway(c)

Perform the actual installation on local machine or prepare data transfer to remote gate and submit a job script which will install everything on the remote gate.

Parameters:c (ControlFile) – Contains all the parameters of CONTROL file and command line.

install.main()

Controls the installation process. Calls the installation function if target is specified.

install.mk_compilejob(makefile, target, ecuid, ecgid, fp_root)

Modifies the original job template file so that it is specified for the user and the environment were it will be applied. Result is stored in a new file “job.temp” in the python directory.

Parameters:

• makefile (str) – Name of the makefile which should be used to compile FORTRAN CONVERT2 program.
• target (str) – The target where the installation should be done, e.g. the queue.
• ecuid (str) – The user id on ECMWF server.
• ecgid (str) – The group id on ECMWF server.
• fp_root (str) – Path to the root directory of FLEXPART environment or flex_extract environment.

install.mk_convert_build(src_path, makefile)

Compiles the Fortran code and generates the executable.

Parameters:

• src_path (str) – Path to the fortran source directory.
• makefile (str) – The name of the makefile which should be used.

install.mk_env_vars(ecuid, ecgid, gateway, destination)

Creates a file named ECMWF_ENV which contains the necessary environmental variables at ECMWF servers. It is based on the template ECMWF_ENV.template.

Parameters:

• ecuid (str) – The user id on ECMWF server.
• ecgid (str) – The group id on ECMWF server.
• gateway (str) – The gateway server the user is using.
• destination (str) – The remote destination which is used to transfer files from ECMWF server to local gateway server.

install.mk_job_template(ecuid, ecgid, gateway, destination, fp_root)

Modifies the original job template file so that it is specified for the user and the environment were it will be applied. Result is stored in a new file.

Parameters:

• ecuid (str) – The user id on ECMWF server.
• ecgid (str) – The group id on ECMWF server.
• gateway (str) – The gateway server the user is using.
• destination (str) – The remote destination which is used to transfer files from ECMWF server to local gateway server.
• fp_root (str) – Path to the root directory of FLEXPART environment or flex_extract environment.

install.mk_tarball(tarball_path, target)

Creates a tarball with all necessary files which need to be sent to the installation directory. It does not matter if this is local or remote. Collects all python files, the Fortran source and makefiles, the ECMWF_ENV file, the CONTROL files as well as the template files.

Parameters:

• tarball_path (str) – The complete path to the tar file which will contain all relevant data for flex_extract.
• target (str) – The queue where the job is submitted to.

install.un_tarball(tarball_path)

Extracts the given tarball into current directory.

Parameters:tarball_path (str) – The complete path to the tar file which will contain all relevant data for flex_extract.

submit

This script allows the user to extract meteorological fields from the ECMWF.

It prepares the settings for retrieving the data from ECMWF servers and process the resulting files to prepare them for the use with FLEXPART or FLEXTRA.

If it is supposed to work locally then it works through the necessary functions get_mars_data and prepare_flexpart. Otherwise it prepares a job script (korn shell) which will do the necessary work on the ECMWF server. This script will de submitted via the ecaccess command ecaccess-job-submit.

This file can also be imported as a module which then contains the following functions:

• main - the main function of the script
• submit - calls mk_jobscript depending on operation mode and submits its
• mk_jobscript - creates the job script from a template

Type: submit.py –help to get information about command line parameters. Read the documentation for usage instructions.

submit.main()

Get the arguments from script call and from CONTROL file. Decides from the argument “queue” if the local version is done “queue=None” or the gateway version with “queue=ecgate” or “queue=cca”.

submit.mk_jobscript(jtemplate, job_file, clist)

Creates the job script from template.

Parameters:

• jtemplate (str) – Job template file from sub-directory “_templates” for submission to ECMWF. It contains all necessary module and variable settings for the ECMWF environment as well as the job call and mail report instructions. Default is “job.temp”.
• job_file (str) – Path to the job script file.
• clist (list of str) – Contains all necessary parameters for ECMWF CONTROL file.

submit.submit(jtemplate, c, queue)

Prepares the job script and submits it to the specified queue.

Parameters:

• jtemplate (str) – Job template file from sub-directory “_templates” for submission to ECMWF. It contains all necessary module and variable settings for the ECMWF environment as well as the job call and mail report instructions. Default is “job.temp”.
• c (ControlFile) – Contains all the parameters of CONTROL file and command line.
• queue (str) – Name of queue for submission to ECMWF (e.g. ecgate or cca )

Classes

ControlFile

class ControlFile.ControlFile(filename)

Contains the information which are stored in the CONTROL files.

The CONTROL file is the steering part of the FLEXPART extraction software. All necessary parameters needed to retrieve the data fields from the MARS archive for driving FLEXPART are set in a CONTROL file. Some specific parameters like the start and end dates can be overwritten by the command line parameters, but in generel all parameters needed for a complete set of fields for FLEXPART can be set in the CONTROL file.

controlfile

The name of the control file to be processed. Default value is the filename passed to the init function when initialised.

Type:str

start_date

The first day of the retrieval period. Default value is None.

Type:str

end_date

The last day of the retrieval period. Default value is None.

Type:str

date_chunk

Length of period for a single mars retrieval. Default value is 3.

Type:int

dtime

The time step in hours. Default value is None.

Type:str

basetime

The time for a half day retrieval. The 12 hours upfront are to be retrieved. Default value is None.

Type:str

maxstep

The maximum forecast step for non flux data. Default value is None.

Type:int

type

List of field type per retrieving hour. Default value is None.

Type:list of str

time

List of retrieving times in hours. Default valuer is None.

Type:list of str

step

List of forecast time steps in hours for non flux data. Default value is None.

Type:list of str or str

acctype

The field type for the accumulated forecast fields. Default value is None.

Type:str

acctime

The starting time of the accumulated forecasts. Default value is None.

Type:str

accmaxstep

The maximum forecast step for the accumulated forecast fields (flux data). Default value is None.

Type:int

marsclass

Characterisation of dataset. Default value is None.

Type:str

dataset

For public datasets there is the specific naming and parameter dataset which has to be used to characterize the type of data. Default value is None.

Type:str

stream

Identifies the forecasting system used to generate the data. Default value is None.

Type:str

number

Selects the member in ensemble forecast run. Default value is ‘OFF’.

Type:str

expver

The version number of the dataset. Default value is ‘1’.

Type:str

gaussian

This parameter is deprecated and should no longer be used. Specifies the desired type of Gaussian grid for the output. Default value is an empty string ‘’.

Type:str

grid

Specifies the output grid which can be either a Gaussian grid or a Latitude/Longitude grid. Default value is None.

Type:str

area

Specifies the desired sub-area of data to be extracted. Default value is None.

Type:str

left

The western most longitude of the area to be extracted. Default value is None.

Type:str

lower

The southern most latitude of the area to be extracted. Default value is None.

Type:str

upper

The northern most latitued of the area to be extracted. Default value is None.

Type:str

right

The eastern most longitude of the area to be extracted. Default value is None.

Type:str

level

Specifies the maximum level. Default value is None.

Type:str

levelist

Specifies the required level list. Default value is None.

Type:str

resol

Specifies the desired triangular truncation of retrieved data, before carrying out any other selected post-processing. Default value is None.

Type:str

gauss

Switch to select gaussian fields (1) or regular lat/lon (0). Default value is 0.

Type:int

accuracy

Specifies the number of bits per value to be used in the generated GRIB coded fields. Default value is 24.

Type:int

omega

Switch to select omega retrieval (1) or not (0). Default value is 0.

Type:int

omegadiff

Switch to decide to calculate Omega and Dps/Dt from continuity equation for diagnostic purposes (1) or not (0). Default value is 0.

Type:int

eta

Switch to select direct retrieval of etadot from MARS (1) or wether it has to be calculated (0). Then Default value is 0.

Type:int

etadiff

Switch to select calculation of etadot and Dps/Dt from continuity equation for diagnostic purposes (1) or not (0). Default value is 0.

Type:int

etapar

GRIB parameter Id for etadot fields. Default value is 77.

Type:int

dpdeta

Switch to select multiplication of etadot with dpdeta. Default value is 1.

Type:int

smooth

Spectral truncation of ETADOT after calculation on Gaussian grid. Default value is 0.

Type:int

format

The format of the GRIB data. Default value is ‘GRIB1’.

Type:str

addpar

List of additional surface level ECMWF parameter to be retrieved. Default value is None.

Type:str

prefix

Prefix string for the final FLEXPART/FLEXTRA ready input files. Default value is ‘EN’.

Type:str

cwc

Switch to select wether the sum of cloud liquid water content and cloud ice water content should be retrieved. Default value is 0.

Type:int

wrf

Switch to select further parameters for retrievment to support WRF simulations. Default value is 0.

Type:int

ecfsdir

Path to the ECMWF storage ‘ectmp:/${USER}/econdemand/’

Type:str

mailfail

Email list for sending error log files from ECMWF servers. The email addresses should be seperated by a comma. Default value is [‘${USER}’].

Type:list of str

mailops

Email list for sending operational log files from ECMWF servers. The email addresses should be seperated by a comma. Default value is [‘${USER}’].

Type:list of str

grib2flexpart

Switch to select generation of preprocessed FLEXPART files “.fp”. If it is selected, the program grib2flexpart will try to convert the flex_extract output files into “.fp” format.

Type:int 0

ecstorage

Switch to select storage of FLEXPART ready output files in the ECFS file system. Default value is 0.

Type:int

ectrans

Switch to select the transfer of FLEXPART ready output files to the gateway server. Default value is 0.

Type:int

inputdir

Path to the temporary directory for the retrieval grib files and other processing files. Default value is _config.PATH_INPUT_DIR.

Type:str

outputdir

Path to the final directory where the final FLEXPART ready input files are stored. Default value is None.

Type:str

flexextractdir

Path to the flex_extract root directory. Default value is _config.PATH_FLEXEXTRACT_DIR.

Type:str

exedir

Path to the FORTRAN executable file. Default value is _config.PATH_FORTRAN_SRC.

Type:str

flexpartdir

Path to a FLEXPART root directory. Default value is None.

Type:str

makefile

Name of the makefile to be used for the Fortran program. Default value is ‘Makefile.gfortran’.

Type:str

destination

The remote destination which is used to transfer files from ECMWF server to local gateway server. Default value is None.

Type:str

gateway

The gateway server the user is using. Default value is None.

Type:str

ecuid

The user id on ECMWF server. Default value is None.

Type:str

ecgid

The group id on ECMWF server. Default value is None.

Type:str

install_target

Defines the location where the installation is to be done. Default value is None.

Type:str

debug

Switch to keep temporary files at the end of postprocessing (1) or to delete all temporary files except the final output files (0). Default value is 0.

Type:int

request

Switch to select between just retrieving the data (0), writing the mars parameter values to a csv file (1) or doing both (2). Default value is 0.

Type:int

public

Switch to select kind of ECMWF Web Api access and the possible data sets. Public data sets (1) and Memberstate data sets (0). Default value is 0.

Type:int

ecapi

Tells wether the ECMWF Web APi was able to load or not. Default value is None.

Type:boolean

purefc

Switch to decide wether the job is a pure forecast retrieval or coupled with analysis data. Default value is 0.

Type:int

rrint

Switch to select between old precipitation disaggregation method (0) or the new IA3 disaggegration method (1). Default value is 0.

Type:int

logicals

List of the names of logical switches which controls the flow of the program. Default list is [‘gauss’, ‘omega’, ‘omegadiff’, ‘eta’, ‘etadiff’, ‘dpdeta’, ‘cwc’, ‘wrf’, ‘grib2flexpart’, ‘ecstorage’, ‘ectrans’, ‘debug’, ‘request’, ‘public’, ‘purefc’, ‘rrint’]

Type:list of str

assign_args_to_control(args)

Overwrites the existing ControlFile instance attributes with the command line arguments.

Parameters:args (Namespace) – Contains the commandline arguments from script/program call.

assign_envs_to_control(envs)

Assigns the ECMWF environment parameter.

Parameters:envs (dict of str) – Contains the ECMWF environment parameternames “ECUID”, “ECGID”, “DESTINATION” and “GATEWAY” with its corresponding values. They were read from the file “ECMWF_ENV”.

check_conditions(queue)

Checks a couple of necessary attributes and conditions, such as if they exist and contain values. Otherwise set default values.

Parameters:queue (str) – Name of the queue if submitted to the ECMWF servers. Used to check if ecuid, ecgid, gateway and destination are set correctly and are not empty.

to_list()

Just generates a list of strings containing the attributes and assigned values except the attributes “_expanded”, “exedir”, “flexextractdir” and “flexpartdir”.

Returns:l – A sorted list of the all ControlFile class attributes with their values except the attributes “_expanded”, “exedir”, “flexextractdir” and “flexpartdir”. Return type:list of *

EcFlexpart

class EcFlexpart.EcFlexpart(c, fluxes=False)

Class to represent FLEXPART specific ECMWF data.

FLEXPART needs grib files in a specifc format. All necessary data fields for one time step are stored in a single file. The class represents an instance with all the parameter and settings necessary for retrieving MARS data and modifing them so they are fitting FLEXPART needs. The class is able to disaggregate the fluxes and convert grid types to the one needed by FLEXPART, therefore using the FORTRAN program.

mreq_count

Counter for the number of generated mars requests.

Type:int

inputdir

Path to the directory where the retrieved data is stored.

Type:str

dataset

For public datasets there is the specific naming and parameter dataset which has to be used to characterize the type of data.

Type:str

basetime

The time for a half day retrieval. The 12 hours upfront are to be retrieved.

Type:str

dtime

Time step in hours.

Type:str

acctype

The field type for the accumulated forecast fields.

Type:str

acctime

The starting time from the accumulated forecasts.

Type:str

accmaxstep

The maximum forecast step for the accumulated forecast fields.

Type:str

marsclass

Characterisation of dataset.

Type:str

stream

Identifies the forecasting system used to generate the data.

Type:str

number

Selects the member in ensemble forecast run.

Type:str

resol

Specifies the desired triangular truncation of retrieved data, before carrying out any other selected post-processing.

Type:str

accuracy

Specifies the number of bits per value to be used in the generated GRIB coded fields.

Type:str

addpar

List of additional parameters to be retrieved.

Type:str

level

Specifies the maximum level.

Type:str

expver

The version of the dataset.

Type:str

levelist

Specifies the required levels.

Type:str

glevelist

Specifies the required levels for gaussian grids.

Type:str

gaussian

This parameter is deprecated and should no longer be used. Specifies the desired type of Gaussian grid for the output.

Type:str

grid

Specifies the output grid which can be either a Gaussian grid or a Latitude/Longitude grid.

Type:str

area

Specifies the desired sub-area of data to be extracted.

Type:str

purefc

Switch for definition of pure forecast mode or not.

Type:int

outputfilelist

The final list of FLEXPART ready input files.

Type:list of str

types

Determines the combination of type of fields, time and forecast step to be retrieved.

Type:dictionary

params

Collection of grid types and their corresponding parameters, levels, level types and the grid definition.

Type:dictionary

server

This is the connection to the ECMWF data servers.

Type:ECMWFService or ECMWFDataServer

public

Decides which Web API Server version is used.

Type:int

dates

Contains start and end date of the retrieval in the format “YYYYMMDD/to/YYYYMMDD”

Type:str

create(inputfiles, c)

An index file will be created which depends on the combination of “date”, “time” and “stepRange” values. This is used to iterate over all messages in each grib file which were passed through the parameter “inputfiles” to seperate specific parameters into fort.* files. Afterwards the FORTRAN program is called to convert the data fields all to the same grid and put them in one file per unique time step (combination of “date”, “time” and “stepRange”).

Note

This method is based on the ECMWF example index.py https://software.ecmwf.int/wiki/display/GRIB/index.py

Parameters:

• inputfiles (UioFiles) – Contains a list of files.
• c (ControlFile) – Contains all the parameters of CONTROL file and command line.

deacc_fluxes(inputfiles, c)

De-accumulate and disaggregate flux data.

Goes through all flux fields in ordered time and de-accumulate the fields. Afterwards the fields are disaggregated in time. Different versions of disaggregation is provided for rainfall data (darain, modified linear) and the surface fluxes and stress data (dapoly, cubic polynomial).

Parameters:

• inputfiles (UioFiles) – Contains the list of files that contain flux data.
• c (ControlFile) – Contains all the parameters of CONTROL file and command line.

prepare_fp_files(c)

Conversion of GRIB files to FLEXPART binary format.

Parameters:c (ControlFile) – Contains all the parameters of CONTROL file and command line.

process_output(c)

Postprocessing of FLEXPART input files.

The grib files are postprocessed depending on the selection in CONTROL file. The resulting files are moved to the output directory if its not equal to the input directory. The following modifications might be done if properly switched in CONTROL file: GRIB2 - Conversion to GRIB2 ECTRANS - Transfer of files to gateway server ECSTORAGE - Storage at ECMWF server

Parameters:c (ControlFile) – Contains all the parameters of CONTROL file and command line.

retrieve(server, dates, public, request, inputdir='.')

Finalizing the retrieval information by setting final details depending on grid type. Prepares MARS retrievals per grid type and submits them.

Parameters:

• server (ECMWFService or ECMWFDataServer) – The connection to the ECMWF server. This is different for member state users which have full access and non member state users which have only access to the public data sets. The decision is made from command line argument “public”; for public access its True (ECMWFDataServer) for member state users its False (ECMWFService)
• dates (str) – Contains start and end date of the retrieval in the format “YYYYMMDD/to/YYYYMMDD”
• request (int) – Selects the mode of retrieval. 0: Retrieves the data from ECMWF. 1: Prints the mars requests to an output file. 2: Retrieves the data and prints the mars request.
• inputdir (str, optional) – Path to the directory where the retrieved data is about to be stored. The default is the current directory (‘.’).

write_namelist(c)

Creates a namelist file in the temporary directory and writes the following values to it: maxl, maxb, mlevel, mlevelist, mnauf, metapar, rlo0, rlo1, rla0, rla1, momega, momegadiff, mgauss, msmooth, meta, metadiff, mdpdeta

Parameters:

• c (ControlFile) – Contains all the parameters of CONTROL file and command line.
• filename (str) – Name of the namelist file.

GribUtil

class GribUtil.GribUtil(filenames)

Class for GRIB utilities (new methods) based on GRIB API

copy_dummy_msg(filename_in, selectWhere=True, keynames=[], keyvalues=[], filemode='w')

Add the content of another input grib file to the objects file but only messages corresponding to keys/values passed to the function. The selectWhere switch decides if to copy the keys equal to (True) or different to (False) the keynames/keyvalues list passed to the function.

Parameters:

• filename_in (string) – Filename of the input file to read the grib messages from.
• selectWhere (boolean, optional) – Decides if to copy the keynames and values equal to (True) or different to (False) the keynames/keyvalues list passed to the function. Default is True.
• keynames (list of string, optional) – List of keynames. Default is an empty list.
• keyvalues (list of string, optional) – List of keyvalues. Default is an empty list.
• filemode (string, optional) – Sets the mode for the output file. Default is “w”.

get_keys(keynames, wherekeynames=[], wherekeyvalues=[])

Get keyvalues for a given list of keynames a where statement can be given (list of key and list of values)

Parameters:

• keynames (list of string) – List of keynames.
• wherekeynames (list of string, optional) – Default value is an empty list.
• wherekeyvalues (list of string, optional) – Default value is an empty list.

Returns:

return_list – List of keyvalues for given keynames.

Return type:

list of string

index(index_keys=['mars'], index_file='my.idx')

Create index file from a list of files if it does not exist or read an index file.

Parameters:

• index_keys (list of string, optional) – Contains the list of key parameter names from which the index is to be created. Default is a list with a single entry string “mars”.
• index_file (string, optional) – Filename where the indices are stored. Default is “my.idx”.

Returns:

iid – Grib index id.

Return type:

integer

set_keys(fromfile, keynames, keyvalues, wherekeynames=[], wherekeyvalues=[], strict=False, filemode='w')

Opens the file to read the grib messages and then write the selected messages (with wherekeys) to a new output file. Also, the keyvalues of the passed list of keynames are set.

Parameters:

• fromfile (string) – Filename of the input file to read the grib messages from.
• keynames (list of string) – List of keynames to set in the selected messages. Default is an empty list.
• keyvalues (list of string) – List of keyvalues to set in the selected messages. Default is an empty list.
• wherekeynames (list of string, optional) – List of keynames to select correct message. Default value is an empty list.
• wherekeyvalues (list of string, optional) – List of keyvalues for keynames to select correct message. Default value is an empty list.
• strict (boolean, optional) – Decides if everything from keynames and keyvalues is written out the grib file (False) or only those meeting the where statement (True). Default is False.
• filemode (string, optional) – Sets the mode for the output file. Default is “w”.

MarsRetrieval

class MarsRetrieval.MarsRetrieval(server, public, marsclass='ei', dataset='', type='', levtype='', levelist='', repres='', date='', resol='', stream='', area='', time='', step='', expver='1', number='', accuracy='', grid='', gaussian='', target='', param='')

Specific syntax and content for submission of MARS retrievals.

A MARS revtrieval has a specific syntax with a selection of keywords and their corresponding values. This class provides the necessary functions by displaying the selected parameters and their values and the actual retrievement of the data through a mars request or a Python web api interface. The initialization already expects all the keyword values.

A description of MARS keywords/arguments and examples of their values can be found here: https://software.ecmwf.int/wiki/display/UDOC/ Identification+keywords#Identificationkeywords-class

server

This is the connection to the ECMWF data servers.

Type:ECMWFService or ECMWFDataServer

public

Decides which Web API Server version is used.

Type:int

marsclass

Characterisation of dataset.

Type:str, optional

dataset

For public datasets there is the specific naming and parameter dataset which has to be used to characterize the type of data.

Type:str, optional

type

Determines the type of fields to be retrieved.

Type:str, optional

levtype

Denotes type of level.

Type:str, optional

levelist

Specifies the required levels.

Type:str, optional

repres

Selects the representation of the archived data.

Type:str, optional

date

Specifies the Analysis date, the Forecast base date or Observations date.

Type:str, optional

resol

Specifies the desired triangular truncation of retrieved data, before carrying out any other selected post-processing.

Type:str, optional

stream

Identifies the forecasting system used to generate the data.

Type:str, optional

area

Specifies the desired sub-area of data to be extracted.

Type:str, optional

time

Specifies the time of the data in hours and minutes.

Type:str, optional

step

Specifies the forecast time step from forecast base time.

Type:str, optional

expver

The version of the dataset.

Type:str, optional

number

Selects the member in ensemble forecast run.

Type:str, optional

accuracy

Specifies the number of bits per value to be used in the generated GRIB coded fields.

Type:str, optional

grid

Specifies the output grid which can be either a Gaussian grid or a Latitude/Longitude grid.

Type:str, optional

gaussian

This parameter is deprecated and should no longer be used. Specifies the desired type of Gaussian grid for the output.

Type:str, optional

target

Specifies a file into which data is to be written after retrieval or manipulation.

Type:str, optional

param

Specifies the meteorological parameter.

Type:str, optional

data_retrieve()

Submits a MARS retrieval. Depending on the existence of ECMWF Web-API it is submitted via Python or a subprocess in the Shell. The parameter for the mars retrieval are taken from the defined class attributes.

display_info()

Prints all class attributes and their values to the standard output.

print_infodata_csv(inputdir, request_number)

Write all request parameter in alpabetical order into a “csv” file.

Parameters:

• inputdir (str) – The path where all data from the retrievals are stored.
• request_number (int) – Number of mars requests for flux and non-flux data.

UioFiles

class UioFiles.UioFiles(path, pattern)

Collection of files matching a specific pattern.

The pattern can contain regular expressions for the files. The files are listed and can be transformed to a single string or they can be deleted.

path

Directory where to list the files.

Type:str

pattern

Regular expression pattern. For example: ‘*.grb’

Type:str

files

List of files matching the pattern in the path.

Type:list of str

delete_files()

Deletes the files.

Modules

get_mars_data

This script extracts MARS data from ECMWF servers.

At first, the necessary parameters from command line and CONTROL files are extracted. They define the data set to be extracted from MARS.

This file can also be imported as a module and contains the following functions:

• main - the main function of the script
• get_mars_data - overall control of ECMWF data retrievment
• write_reqheader - writes the header into the mars_request file
• mk_server - creates the server connection to ECMWF servers
• mk_dates - defines the start and end date
• remove_old - deletes old retrieved grib files
• do_retrievement - creates individual retrievals

Type: get_mars_data.py –help to get information about command line parameters. Read the documentation for usage instructions.

get_mars_data.do_retrievement(c, server, start, end, delta_t, fluxes=False)

Divides the complete retrieval period in smaller chunks and retrieves the data from MARS.

Parameters:

• c (ControlFile) – Contains all the parameters of CONTROL file and command line.
• server (ECMWFService or ECMWFDataServer) – The server connection to ECMWF.
• start (datetime) – The start date of the retrieval.
• end (datetime) – The end date of the retrieval.
• delta_t (datetime) – Delta_t + 1 is the maximal time period of a single retrieval.
• fluxes (boolean, optional) – Decides if the flux parameters are to be retrieved or the rest of the parameter list. Default value is False.

get_mars_data.get_mars_data(c)

Retrieves the EC data needed for a FLEXPART simulation.

Start and end dates for retrieval period is set. Retrievals are divided into smaller periods if necessary and datechunk parameter is set.

Parameters:c (ControlFile) – Contains all the parameters of CONTROL file and command line.

get_mars_data.main()

Controls the program to get data out of mars.

This is done if it is called directly from command line. Then it also takes program call arguments and control file input.

get_mars_data.mk_dates(c, fluxes)

Prepares start and end date depending on flux or non flux data.

If forecast for maximum one day (upto 24h) are to be retrieved, then collect accumulation data (flux data) with additional days in the beginning and at the end (used for complete disaggregation of original period)

If forecast data longer than 24h are to be retrieved, then collect accumulation data (flux data) with the exact start and end date (disaggregation will be done for the exact time period with boundary conditions)

Since for basetime the extraction contains the 12 hours upfront, if basetime is 0, the starting date has to be the day before and

Parameters:

• c (ControlFile) – Contains all the parameters of CONTROL file and command line.
• fluxes (boolean, optional) – Decides if the flux parameter settings are stored or the rest of the parameter list. Default value is False.

Returns:

• start (datetime) – The start date of the retrieving data set.
• end (datetime) – The end date of the retrieving data set.
• chunk (datetime) – Time period in days for one single mars retrieval.

get_mars_data.mk_server(c)

Creates server connection if ECMWF WebAPI is available.

Parameters:c (ControlFile) – Contains all the parameters of CONTROL file and command line. Returns:server – Connection to ECMWF server via python interface ECMWF WebAPI. Return type:ECMWFDataServer or ECMWFService

get_mars_data.remove_old(pattern, inputdir)

Deletes old retrieval files from current input directory matching the pattern.

Parameters:

• pattern (str) – The sub string pattern which identifies the files to be deleted.
• inputdir (str, optional) – Path to the directory where the retrieved data is stored.

get_mars_data.write_reqheader(marsfile)

Writes header with column names into mars request file.

Parameters:marsfile (str) – Path to the mars request file.

prepare_flexpart

This script prepares the final version of the grib files which are then used by FLEXPART.

It converts the bunch of grib files extracted via get_mars_data before, by doing the necessary conversion to get consistent grids or the disaggregation of flux data. Finally, the data fields are combined in files per available hour with the naming convention xxYYMMDDHH, where xx should be 2 arbitrary letters (mostly xx is chosen to be “EN”).

This file can also be imported as a module which then contains the following functions:

• main
• prepare_flexpart

Type: prepare_flexpart.py –help to get information about command line parameters. Read the documentation for usage instructions.

prepare_flexpart.main()

Controls the program to prepare flexpart input files from mars data.

This is done if it is called directly from command line. Then it also takes program call arguments and control file input.

prepare_flexpart.prepare_flexpart(ppid, c)

Converts the mars data into flexpart ready input files.

Specific data fields are converted to a different grid and the flux data are going to be disaggregated. The data fields are collected by hour and stored in a file with a specific FLEXPART relevant naming convention.

Parameters:

• ppid (int) – Contains the ppid number of the current ECMWF job. It will be None if the method was called within this module.
• c (ControlFile) – Contains all the parameters of CONTROL file and command line.

tools

This module contains a collection of diverse tasks within flex_extract.

tools.clean_up(c)

Remove files from the intermediate directory (inputdir).

It keeps the final FLEXPART input files if program runs without ECMWF Api and keywords “ectrans” or “ecstorage” are set to “1”.

Parameters:c (ControlFile) – Contains all the parameters of CONTROL file and command line.

tools.execute_subprocess(cmd_list, error_msg='SUBPROCESS FAILED!')

Executes a command line instruction via a subprocess.

Error handling is done if an error occures.

Parameters:cmd_list (list of str) – A list of the components for the command line execution. Each list entry is a single part of the command which is seperated from the rest by a blank space. E.g. [‘mv’, file1, file2] Returns:error_msg – The possible error message if the subprocess failed. By default it will just tell “SUBPROCESS FAILED!”. Return type:str, optional

tools.get_cmdline_args()

Decomposes the command line arguments and assigns them to variables. Apply default values for non mentioned arguments.

Returns:args – Contains the commandline arguments from script/program call. Return type:Namespace

tools.get_dimensions(info, purefc, dtime, index_vals, start_date, end_date)

This function specifies the correct dimensions for x, y and t.

Parameters:

• info (dict) – Contains basic informations of the ECMWF grib files, e.g. ‘Ni’, ‘Nj’, ‘latitudeOfFirstGridPointInDegrees’, ‘longitudeOfFirstGridPointInDegrees’, ‘latitudeOfLastGridPointInDegrees’, ‘longitudeOfLastGridPointInDegrees’, ‘jDirectionIncrementInDegrees’, ‘iDirectionIncrementInDegrees’, ‘missingValue’
• purefc (int) – Switch for definition of pure forecast mode or not.
• dtime (str) – Time step in hours.
• index_vals (list of list of str) – Contains the values from the keys used for a distinct selection of grib messages in processing the grib files. Content looks like e.g.: index_vals[0]: (‘20171106’, ‘20171107’, ‘20171108’) ; date index_vals[1]: (‘0’, ‘1200’, ‘1800’, ‘600’) ; time index_vals[2]: (‘0’, ‘12’, ‘3’, ‘6’, ‘9’) ; stepRange
• start_date (str) – The start date of the retrieval job.
• end_date (str) – The end date of the retrieval job.

Returns:

(ix, jy, it) – Dimension in x-direction, y-direction and in time.

Return type:

tuple of int

tools.get_informations(filename)

Gets basic information from an example grib file.

These information are important for later use and the initialization of numpy arrays for data storing.

Parameters:filename (str) – Name of the file which will be opened to extract basic information. Returns:data – Contains basic informations of the ECMWF grib files, e.g. ‘Ni’, ‘Nj’, ‘latitudeOfFirstGridPointInDegrees’, ‘longitudeOfFirstGridPointInDegrees’, ‘latitudeOfLastGridPointInDegrees’, ‘longitudeOfLastGridPointInDegrees’, ‘jDirectionIncrementInDegrees’, ‘iDirectionIncrementInDegrees’, ‘missingValue’ Return type:dict

tools.get_list_as_string(list_obj, concatenate_sign=', ')

Converts a list of arbitrary content into a single string.

Parameters:

• list_obj (list of *) – A list with arbitrary content.
• concatenate_sign (str, optional) – A string which is used to concatenate the single list elements. Default value is “, “.

Returns:

str_of_list – The content of the list as a single string.

Return type:

str

tools.init128(filepath)

Opens and reads the grib file with table 128 information.

Parameters:filepath (str) – Path to file of ECMWF grib table number 128. Returns:table128 – Contains the ECMWF grib table 128 information. The key is the parameter number and the value is the short name of the parameter. Return type:dict

tools.make_dir(directory)

Creates a directory.

It gives a warning if the directory already exists and skips process. The program stops only if there is another problem.

Parameters:directory (str) – The path to directory which should be created.

tools.my_error(users, message='ERROR')

Prints a specified error message which can be passed to the function before exiting the program.

Parameters:

• user (list of str) – Contains all email addresses which should be notified. It might also contain just the ecmwf user name which wil trigger mailing to the associated email address for this user.
• message (str, optional) – Error message. Default value is “ERROR”.

tools.none_or_int(value)

Converts the input string into pythons None-type if the string contains string “None”. Otherwise it is converted to an integer value.

Parameters:value (str) – String to be checked for the “None” word. Returns:Return depends on the content of the input value. If it was “None”, then the python type None is returned. Otherwise the string is converted into an integer value. Return type:None or int(value)

tools.none_or_str(value)

Converts the input string into pythons None-type if the string contains string “None”.

Parameters:value (str) – String to be checked for the “None” word. Returns:Return depends on the content of the input value. If it was “None”, then the python type None is returned. Otherwise the string itself. Return type:None or value

tools.normal_exit(message='Done!')

Prints a specific exit message which can be passed to the function.

Parameters:message (str, optional) – Message for exiting program. Default value is “Done!”.

tools.product(*args, **kwds)

Creates combinations of all passed arguments.

This method combines the single characters of the passed arguments with each other. So that each character of each argument value will be combined with each character of the other arguments as a tuple.

Note

This method is taken from an example at the ECMWF wiki website. https://software.ecmwf.int/wiki/display/GRIB/index.py; 2018-03-16

Example

product(‘ABCD’, ‘xy’) –> Ax Ay Bx By Cx Cy Dx Dy

product(range(2), repeat = 3) –> 000 001 010 011 100 101 110 111

Parameters:

• *args (list or str) – Positional arguments (arbitrary number).
• **kwds (dict) – Contains all the keyword arguments from *args.

Returns:

prod – Return will be done with “yield”. A tuple of combined arguments. See example in description above.

Return type:

tuple

tools.put_file_to_ecserver(ecd, filename, target, ecuid, ecgid)

Uses the ecaccess-file-put command to send a file to the ECMWF servers.

Note

The return value is just for testing reasons. It does not have to be used from the calling function since the whole error handling is done in here.

Parameters:

• ecd (str) – The path were the file is stored.
• filename (str) – The name of the file to send to the ECMWF server.
• target (str) – The target queue where the file should be sent to.
• ecuid (str) – The user id on ECMWF server.
• ecgid (str) – The group id on ECMWF server.

tools.read_ecenv(filepath)

Reads the file into a dictionary where the key values are the parameter names.

Parameters:filepath (str) – Path to file where the ECMWF environment parameters are stored. Returns:envs – Contains the environment parameter ecuid, ecgid, gateway and destination for ECMWF server environments. Return type:dict

tools.send_mail(users, success_mode, message)

Prints a specific exit message which can be passed to the function.

Parameters:

• users (list of str) – Contains all email addresses which should be notified. It might also contain just the ecmwf user name which wil trigger mailing to the associated email address for this user.
• success_mode (str) – States the exit mode of the program to put into the mail subject line.
• message (str, optional) – Message for exiting program. Default value is “Done!”.

tools.silent_remove(filename)

Remove file if it exists. The function does not fail if the file does not exist.

Parameters:filename (str) – The name of the file to be removed without notification.

tools.submit_job_to_ecserver(target, jobname)

Uses ecaccess-job-submit command to submit a job to the ECMWF server.

Note

The return value is just for testing reasons. It does not have to be used from the calling function since the whole error handling is done in here.

Parameters:

• target (str) – The target where the file should be sent to, e.g. the queue.
• jobname (str) – The name of the jobfile to be submitted to the ECMWF server.

Returns:

job_id – The id number of the job as a reference at the ecmwf server.

Return type:

int

tools.to_param_id(pars, table)

Transform parameter names to parameter ids with ECMWF grib table 128.

Parameters:

• pars (str) – Addpar argument from CONTROL file in the format of parameter names instead of ids. The parameter short names are sepearted with “/” and they are passed as one single string.
• table (dict) – Contains the ECMWF grib table 128 information. The key is the parameter number and the value is the short name of the parameter.

Returns:

ipar – List of addpar parameters from CONTROL file transformed to parameter ids in the format of integer.

Return type:

list of int

disaggregation

Disaggregation of deaccumulated flux data from an ECMWF model FG field.

Initially the flux data to be concerned are:

• large-scale precipitation
• convective precipitation
• surface sensible heat flux
• surface solar radiation
• u stress
• v stress

Different versions of disaggregation is provided for rainfall data (darain, modified linear) and the surface fluxes and stress data (dapoly, cubic polynomial).

disaggregation.IA3(g)

Interpolation with a non-negative geometric mean based algorithm.

The original grid is reconstructed by adding two sampling points in each data series interval. This subgrid is used to keep all information during the interpolation within the associated interval. Additionally, an advanced monotonicity filter is applied to improve the monotonicity properties of the series.

Note

(C) Copyright 2017-2019 Sabine Hittmeir, Anne Philipp, Petra Seibert

This work is licensed under the Creative Commons Attribution 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ or send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.

Parameters:g (list of float) – Complete data series that will be interpolated having the dimension of the original raw series. Returns:f – The interpolated data series with additional subgrid points. Its dimension is equal to the length of the input data series times three. Return type:list of float

References

For more information see article: Hittmeir, S.; Philipp, A.; Seibert, P. (2017): A conservative interpolation scheme for extensive quantities with application to the Lagrangian particle dispersion model FLEXPART., Geoscientific Model Development

disaggregation.dapoly(alist)

Cubic polynomial interpolation of deaccumulated fluxes.

Interpolation of deaccumulated fluxes of an ECMWF model FG field using a cubic polynomial solution which conserves the integrals of the fluxes within each timespan. Disaggregation is done for 4 accumluated timespans which generates a new, disaggregated value which is output at the central point of the 4 accumulation timespans. This new point is used for linear interpolation of the complete timeseries afterwards.

Parameters:alist (list of array of float) – List of 4 timespans as 2-dimensional, horizontal fields. E.g. [[array_t1], [array_t2], [array_t3], [array_t4]] Returns:nfield – Interpolated flux at central point of accumulation timespan. Return type:array of float

Note

March 2000 : P. JAMES

Original author

June 2003 : A. BECK

Adaptations

November 2015 : Leopold Haimberger (University of Vienna)

Migration from Fortran to Python

disaggregation.darain(alist)

Linear interpolation of deaccumulated fluxes.

Interpolation of deaccumulated fluxes of an ECMWF model FG rainfall field using a modified linear solution which conserves the integrals of the fluxes within each timespan. Disaggregation is done for 4 accumluated timespans which generates a new, disaggregated value which is output at the central point of the 4 accumulation timespans. This new point is used for linear interpolation of the complete timeseries afterwards.

Parameters:alist (list of array of float) – List of 4 timespans as 2-dimensional, horizontal fields. E.g. [[array_t1], [array_t2], [array_t3], [array_t4]] Returns:nfield – Interpolated flux at central point of accumulation timespan. Return type:array of float

Note

March 2000 : P. JAMES

Original author

June 2003 : A. BECK

Adaptations

November 2015 : Leopold Haimberger (University of Vienna)

Migration from Fortran to Python