Auto Generated Documentation

Porgrams

install

install.delete_convert_build(src_path)

Clean up the Fortran source directory and remove all build files (e.g. *.o, *.mod and CONVERT2)

Parameters:src_path (string) – Path to the fortran source directory.

install.get_install_cmdline_arguments()

Decomposes the command line arguments and assigns them to variables. Apply default values for non mentioned arguments.

Returns:args – Contains the commandline arguments from script/program call. Return type:Namespace

install.install_via_gateway(c)

Perform the actual installation on local machine or prepare data transfer to remote gate and submit a job script which will install everything on the remote gate.

Parameters:c (ControlFile) – Contains all the parameters of CONTROL file and command line.

install.main()

Controls the installation process. Calls the installation function if target is specified.

install.make_convert_build(src_path, makefile)

Compiles the Fortran code and generates the executable.

Parameters:

• src_path (string) – Path to the fortran source directory.
• makefile (string) – The name of the makefile which should be used.

install.mk_compilejob(makefile, target, ecuid, ecgid, fp_root)

Modifies the original job template file so that it is specified for the user and the environment were it will be applied. Result is stored in a new file “job.temp” in the python directory.

Parameters:

• makefile (string) – Name of the makefile which should be used to compile FORTRAN CONVERT2 program.
• target (string) – The target where the installation should be done, e.g. the queue.
• ecuid (string) – The user id on ECMWF server.
• ecgid (string) – The group id on ECMWF server.
• fp_root (string) – Path to the root directory of FLEXPART environment or flex_extract environment.

install.mk_env_vars(ecuid, ecgid, gateway, destination)

Creates a file named ECMWF_ENV which contains the necessary environmental variables at ECMWF servers. It is based on the template ECMWF_ENV.template.

Parameters:

• ecuid (string) – The user id on ECMWF server.
• ecgid (string) – The group id on ECMWF server.
• gateway (string) – The gateway server the user is using.
• destination (string) – The remote destination which is used to transfer files from ECMWF server to local gateway server.

install.mk_job_template(ecuid, ecgid, gateway, destination, fp_root)

Modifies the original job template file so that it is specified for the user and the environment were it will be applied. Result is stored in a new file.

Parameters:

• ecuid (string) – The user id on ECMWF server.
• ecgid (string) – The group id on ECMWF server.
• gateway (string) – The gateway server the user is using.
• destination (string) – The remote destination which is used to transfer files from ECMWF server to local gateway server.
• fp_root (string) – Path to the root directory of FLEXPART environment or flex_extract environment.

install.mk_tarball(tarball_path, target)

Creates a tarball with all necessary files which need to be sent to the installation directory. It does not matter if this is local or remote. Collects all python files, the Fortran source and makefiles, the ECMWF_ENV file, the CONTROL files as well as the template files.

Parameters:

• tarball_path (string) – The complete path to the tar file which will contain all relevant data for flex_extract.
• target (string) – The queue where the job is submitted to.

install.un_tarball(tarball_path)

Extracts the given tarball into current directory.

Parameters:tarball_path (string) – The complete path to the tar file which will contain all relevant data for flex_extract.

submit

submit.main()

Get the arguments from script call and from CONTROL file. Decides from the argument “queue” if the local version is done “queue=None” or the gateway version with “queue=ecgate” or “queue=cca”.

submit.submit(jtemplate, c, queue)

Prepares the job script and submit it to the specified queue.

Parameters:

• jtemplate (string) – Job template file from sub-directory “_templates” for submission to ECMWF. It contains all necessary module and variable settings for the ECMWF environment as well as the job call and mail report instructions. Default is “job.temp”.
• c (ControlFile) – Contains all the parameters of CONTROL file and command line.
• queue (string) – Name of queue for submission to ECMWF (e.g. ecgate or cca )

Classes

ControlFile

class ControlFile.ControlFile(filename)

Contains the information which are stored in the CONTROL files.

assign_args_to_control(args)

Overwrites the existing ControlFile instance attributes with the command line arguments.

Parameters:args (Namespace) – Contains the commandline arguments from script/program call.

assign_envs_to_control(envs)

Assigns the ECMWF environment parameter.

Parameters:envs (dictionary of strings) – Contains the ECMWF environment parameternames “ECUID”, “ECGID”, “DESTINATION” and “GATEWAY” with its corresponding values. They were read from the file “ECMWF_ENV”.

check_conditions(queue)

Checks a couple of necessary attributes and conditions, such as if they exist and contain values. Otherwise set default values.

Parameters:queue (string) – Name of the queue if submitted to the ECMWF servers. Used to check if ecuid, ecgid, gateway and destination are set correctly and are not empty.

check_install_conditions()

Checks a couple of necessary attributes and conditions for the installation such as if they exist and contain values. Otherwise set default values.

to_list()

Just generates a list of strings containing the attributes and assigned values except the attributes “_expanded”, “exedir”, “ecmwfdatadir” and “flexpart_root_scripts”.

Returns:l – A sorted list of the all ControlFile class attributes with their values except the attributes “_expanded”, “exedir”, “ecmwfdatadir” and “flexpart_root_scripts”. Return type:list

EcFlexpart

class EcFlexpart.EcFlexpart(c, fluxes=False)

Class to retrieve FLEXPART specific ECMWF data.

create(inputfiles, c)

An index file will be created which depends on the combination of “date”, “time” and “stepRange” values. This is used to iterate over all messages in each grib file which were passed through the parameter “inputfiles” to seperate specific parameters into fort.* files. Afterwards the FORTRAN program is called to convert the data fields all to the same grid and put them in one file per unique time step (combination of “date”, “time” and “stepRange”).

Note

This method is based on the ECMWF example index.py https://software.ecmwf.int/wiki/display/GRIB/index.py

Parameters:

• inputfiles (UioFiles) – Contains a list of files.
• c (ControlFile) – Contains all the parameters of CONTROL file and command line.

deacc_fluxes(inputfiles, c)

Goes through all flux fields in ordered time and de-accumulate the fields. Afterwards the fields are disaggregated in time. Different versions of disaggregation is provided for rainfall data (darain, modified linear) and the surface fluxes and stress data (dapoly, cubic polynomial).

Parameters:

• inputfiles (UioFiles) – Contains a list of files.
• c (ControlFile) – Contains all the parameters of CONTROL file and command line.

prepare_fp_files(c)

Conversion of GRIB files to FLEXPART binary format.

Parameters:c (ControlFile) – Contains all the parameters of CONTROL file and command line.

process_output(c)

The grib files are postprocessed depending on the selection in CONTROL file. The resulting files are moved to the output directory if its not equal to the input directory. The following modifications might be done if properly switched in CONTROL file: GRIB2 - Conversion to GRIB2 ECTRANS - Transfer of files to gateway server ECSTORAGE - Storage at ECMWF server

Parameters:c (ControlFile) – Contains all the parameters of CONTROL file and command line.

retrieve(server, dates, public, request, inputdir='.')

Finalizing the retrieval information by setting final details depending on grid type. Prepares MARS retrievals per grid type and submits them.

Parameters:

• server (ECMWFService or ECMWFDataServer) – The connection to the ECMWF server. This is different for member state users which have full access and non member state users which have only access to the public data sets. The decision is made from command line argument “public”; for public access its True (ECMWFDataServer) for member state users its False (ECMWFService)
• dates (string) – Contains start and end date of the retrieval in the format “YYYYMMDD/to/YYYYMMDD”
• request (integer) – Selects the mode of retrieval. 0: Retrieves the data from ECMWF. 1: Prints the mars requests to an output file. 2: Retrieves the data and prints the mars request.
• inputdir (string, optional) – Path to the directory where the retrieved data is about to be stored. The default is the current directory (‘.’).

write_namelist(c)

Creates a namelist file in the temporary directory and writes the following values to it: maxl, maxb, mlevel, mlevelist, mnauf, metapar, rlo0, rlo1, rla0, rla1, momega, momegadiff, mgauss, msmooth, meta, metadiff, mdpdeta

Parameters:

• c (ControlFile) – Contains all the parameters of CONTROL file and command line.
• filename (string) – Name of the namelist file.

GribTools

class GribTools.GribTools(filenames)

Class for GRIB utilities (new methods) based on GRIB API

copy(filename_in, selectWhere=True, keynames=[], keyvalues=[], filemode='w')

Add the content of another input grib file to the objects file but only messages corresponding to keys/values passed to the function. The selectWhere switch decides if to copy the keys equal to (True) or different to (False) the keynames/keyvalues list passed to the function.

Parameters:

• filename_in (string) – Filename of the input file to read the grib messages from.
• selectWhere (boolean, optional) – Decides if to copy the keynames and values equal to (True) or different to (False) the keynames/keyvalues list passed to the function. Default is True.
• keynames (list of string, optional) – List of keynames. Default is an empty list.
• keyvalues (list of string, optional) – List of keynames. Default is an empty list.
• filemode (string, optional) – Sets the mode for the output file. Default is “w”.

get_keys(keynames, wherekeynames=[], wherekeyvalues=[])

Get keyvalues for a given list of keynames a where statement can be given (list of key and list of values)

Parameters:

• keynames (list of string) – List of keynames.
• wherekeynames (list of string, optional) – Default value is an empty list.
• wherekeyvalues (list of string, optional) – Default value is an empty list.

Returns:

return_list – List of keyvalues for given keynames.

Return type:

list of string

index(index_keys=['mars'], index_file='my.idx')

Create index file from a list of files if it does not exist or read an index file.

Parameters:

• index_keys (list of string, optional) – Contains the list of key parameter names from which the index is to be created. Default is a list with a single entry string “mars”.
• index_file (string, optional) – Filename where the indices are stored. Default is “my.idx”.

Returns:

iid – Grib index id.

Return type:

integer

set_keys(fromfile, keynames, keyvalues, wherekeynames=[], wherekeyvalues=[], strict=False, filemode='w')

Opens the file to read the grib messages and then write them to a new output file. By default all messages are written out. Also, the keyvalues of the passed list of keynames are set or only those meeting the where statement. (list of key and list of values).

Parameters:

• fromfile (string) – Filename of the input file to read the grib messages from.
• keynames (list of string) – List of keynames. Default is an empty list.
• keyvalues (list of string) – List of keynames. Default is an empty list.
• wherekeynames (list of string, optional) – Default value is an empty list.
• wherekeyvalues (list of string, optional) – Default value is an empty list.
• strict (boolean, optional) – Decides if everything from keynames and keyvalues is written out the grib file (False) or only those meeting the where statement (True). Default is False.
• filemode (string, optional) – Sets the mode for the output file. Default is “w”.

MarsRetrieval

class MarsRetrieval.MarsRetrieval(server, public, marsclass='ei', dataset='', type='', levtype='', levelist='', repres='', date='', resol='', stream='', area='', time='', step='', expver='1', number='', accuracy='', grid='', gaussian='', target='', param='')

Class for submitting MARS retrievals.

A description of MARS keywords/arguments and examples of their values can be found here: https://software.ecmwf.int/wiki/display/UDOC/ Identification+keywords#Identificationkeywords-class

data_retrieve()

Submits a MARS retrieval. Depending on the existence of ECMWF Web-API it is submitted via Python or a subprocess in the Shell. The parameter for the mars retrieval are taken from the defined class attributes.

display_info()

Prints all class attributes and their values to the standard output.

print_info(inputdir, request_number)

Prints all mars requests to an extra file for debugging and information.

Parameters:

• inputdir (string) – The path where all data from the retrievals are stored.
• request_number (integer) – Number of mars requests for flux and non-flux data.

print_infodata_csv(inputdir, request_number)

Write all request parameter in alpabetical order into a “csv” file.

Parameters:

• inputdir (string) – The path where all data from the retrievals are stored.
• request_number (integer) – Number of mars requests for flux and non-flux data.

UioFiles

class UioFiles.UioFiles(path, pattern)

Class to manipulate files. At initialisation it has the pattern which stores a regular expression pattern for the files, the path to the files and the files already.

delete_files()

Deletes the files.

Modules

get_mars_data

get_mars_data.do_retrievement(c, server, start, end, delta_t, fluxes=False)

Divides the complete retrieval period in smaller chunks and retrieves the data from MARS.

Parameters:

• c (ControlFile) – Contains all the parameters of CONTROL file and command line.
• server (ECMWFService) – The server connection to ECMWF.
• start (datetime) – The start date of the retrieval.
• end (datetime) – The end date of the retrieval.
• delta_t (datetime) – Delta_t + 1 is the maximal time period of a single retrieval.
• fluxes (boolean, optional) – Decides if the flux parameters are to be retrieved or the rest of the parameter list. Default value is False.

get_mars_data.get_mars_data(c)

Retrieves the EC data needed for a FLEXPART simulation.

Start and end dates for retrieval period is set. Retrievals are divided into smaller periods if necessary and datechunk parameter is set.

Parameters:c (ControlFile) – Contains all the parameters of CONTROL file and command line.

get_mars_data.main()

Controls the program to get data out of mars.

This is done if it is called directly from command line. Then it also takes program call arguments and control file input.

prepare_flexpart

prepare_flexpart.main()

Controls the program to prepare flexpart input files from mars data.

This is done if it is called directly from command line. Then it also takes program call arguments and control file input.

prepare_flexpart.prepare_flexpart(ppid, c)

Converts the mars data into flexpart ready input files.

Specific data fields are converted to a different grid and the flux data are going to be disaggregated. The data fields are collected by hour and stored in a file with a specific FLEXPART relevant naming convention.

Parameters:

• ppid (int) – Contains the ppid number of the current ECMWF job. It will be None if the method was called within this module.
• c (ControlFile) – Contains all the parameters of CONTROL file and command line.

tools

tools.clean_up(c)

Remove all files from intermediate directory (inputdir).

Parameters:c (ControlFile) – Contains all the parameters of CONTROL file and command line.

tools.get_cmdline_arguments()

Decomposes the command line arguments and assigns them to variables. Apply default values for non mentioned arguments.

Returns:args – Contains the commandline arguments from script/program call. Return type:Namespace

tools.get_list_as_string(list_obj, concatenate_sign=', ')

Converts a list of arbitrary content into a single string.

Parameters:

• list_obj (list) – A list with arbitrary content.
• concatenate_sign (string, optional) – A string which is used to concatenate the single list elements. Default value is “, “.

Returns:

str_of_list – The content of the list as a single string.

Return type:

string

tools.init128(filepath)

Opens and reads the grib file with table 128 information.

Parameters:filepath (string) – Path to file of ECMWF grib table number 128. Returns:table128 – Contains the ECMWF grib table 128 information. The key is the parameter number and the value is the short name of the parameter. Return type:dictionary

tools.make_dir(directory)

Creates a directory and gives a warning if the directory already exists. The program stops only if there is another problem.

Parameters:directory (string) – The directory name including the path which should be created.

tools.my_error(users, message='ERROR')

Prints a specified error message which can be passed to the function before exiting the program.

Parameters:

• user (list of string) – Contains all email addresses which should be notified. It might also contain just the ecmwf user name which wil trigger mailing to the associated email address for this user.
• message (string, optional) – Error message. Default value is “ERROR”.

tools.none_or_int(value)

Converts the input string into pythons None-type if the string contains string “None”. Otherwise it is converted to an integer value.

Parameters:value (string) – String to be checked for the “None” word. Returns:Return depends on the content of the input value. If it was “None”, then the python type None is returned. Otherwise the string is converted into an integer value. Return type:None or int(value)

tools.none_or_str(value)

Converts the input string into pythons None-type if the string contains string “None”.

Parameters:value (string) – String to be checked for the “None” word. Returns:Return depends on the content of the input value. If it was “None”, then the python type None is returned. Otherwise the string itself. Return type:None or value

tools.normal_exit(users, message='Done!')

Prints a specific exit message which can be passed to the function.

Parameters:

• user (list of string) – Contains all email addresses which should be notified. It might also contain just the ecmwf user name which wil trigger mailing to the associated email address for this user.
• message (string, optional) – Message for exiting program. Default value is “Done!”.

tools.product(*args, **kwds)

This method combines the single characters of the passed arguments with each other. So that each character of each argument value will be combined with each character of the other arguments as a tuple.

Note

This method is taken from an example at the ECMWF wiki website. https://software.ecmwf.int/wiki/display/GRIB/index.py; 2018-03-16

Example

product(‘ABCD’, ‘xy’) –> Ax Ay Bx By Cx Cy Dx Dy

product(range(2), repeat = 3) –> 000 001 010 011 100 101 110 111

Parameters:

• *args (tuple) – Positional arguments (arbitrary number).
• **kwds (dictionary) – Contains all the keyword arguments from *args.

Returns:

prod – Return will be done with “yield”. A tuple of combined arguments. See example in description above.

Return type:

tuple

tools.put_file_to_ecserver(ecd, filename, target, ecuid, ecgid)

Uses the ecaccess-file-put command to send a file to the ECMWF servers.

Note

The return value is just for testing reasons. It does not have to be used from the calling function since the whole error handling is done in here.

Parameters:

• ecd (string) – The path were the file is stored.
• filename (string) – The name of the file to send to the ECMWF server.
• target (string) – The target queue where the file should be sent to.
• ecuid (string) – The user id on ECMWF server.
• ecgid (string) – The group id on ECMWF server.

Returns:

rcode – Resulting code of command execution. If successful the string will be empty.

Return type:

string

tools.read_ecenv(filename)

Reads the file into a dictionary where the key values are the parameter names.

Parameters:filename (string) – Path to file where the ECMWF environment parameters are stored. Returns:envs – Contains the environment parameter ecuid, ecgid, gateway and destination for ECMWF server environments. Return type:dictionary

tools.silent_remove(filename)

Remove file if it exists. The function does not fail if the file does not exist.

Parameters:filename (string) – The name of the file to be removed without notification.

tools.submit_job_to_ecserver(target, jobname)

Uses ecaccess-job-submit command to submit a job to the ECMWF server.

Note

The return value is just for testing reasons. It does not have to be used from the calling function since the whole error handling is done in here.

Parameters:

• target (string) – The target where the file should be sent to, e.g. the queue.
• jobname (string) – The name of the jobfile to be submitted to the ECMWF server.

Returns:

rcode – Resulting code of command execution. If successful the string will contain an integer number, representing the id of the job at the ecmwf server.

Return type:

string

tools.to_param_id(pars, table)

Transform parameter names to parameter ids with ECMWF grib table 128.

Parameters:

• pars (string) – Addpar argument from CONTROL file in the format of parameter names instead of ids. The parameter short names are sepearted with “/” and they are passed as one single string.
• table (dictionary) – Contains the ECMWF grib table 128 information. The key is the parameter number and the value is the short name of the parameter.

Returns:

ipar – List of addpar parameters from CONTROL file transformed to parameter ids in the format of integer.

Return type:

list of integer

disaggregation

disaggregation.IA3(g)

Interpolation with a non-negative geometric mean based algorithm.

The original grid is reconstructed by adding two sampling points in each data series interval. This subgrid is used to keep all information during the interpolation within the associated interval. Additionally, an advanced monotonicity filter is applied to improve the monotonicity properties of the series.

Note

Copyright 2017 Sabine Hittmeir, Anne Philipp, Petra Seibert

This work is licensed under the Creative Commons Attribution 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ or send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.

Parameters:g (list of float) – Complete data series that will be interpolated having the dimension of the original raw series. Returns:f – The interpolated data series with additional subgrid points. Its dimension is equal to the length of the input data series times three. Return type:list of float

References

For more information see article: Hittmeir, S.; Philipp, A.; Seibert, P. (2017): A conservative interpolation scheme for extensive quantities with application to the Lagrangian particle dispersion model FLEXPART., Geoscientific Model Development

disaggregation.dapoly(alist)

Cubic polynomial interpolation of deaccumulated fluxes.

Interpolation of deaccumulated fluxes of an ECMWF model FG field using a cubic polynomial solution which conserves the integrals of the fluxes within each timespan. Disaggregation is done for 4 accumluated timespans which generates a new, disaggregated value which is output at the central point of the 4 accumulation timespans. This new point is used for linear interpolation of the complete timeseries afterwards.

Parameters:alist (list of array of float) – List of 4 timespans as 2-dimensional, horizontal fields. E.g. [[array_t1], [array_t2], [array_t3], [array_t4]] Returns:nfield – Interpolated flux at central point of accumulation timespan. Return type:array of float

Note

March 2000 : P. JAMES

Original author

June 2003 : A. BECK

Adaptations

November 2015 : Leopold Haimberger (University of Vienna)

Migration from Fortran to Python

disaggregation.darain(alist)

Linear interpolation of deaccumulated fluxes.

Interpolation of deaccumulated fluxes of an ECMWF model FG rainfall field using a modified linear solution which conserves the integrals of the fluxes within each timespan. Disaggregation is done for 4 accumluated timespans which generates a new, disaggregated value which is output at the central point of the 4 accumulation timespans. This new point is used for linear interpolation of the complete timeseries afterwards.

Parameters:alist (list of array of float) – List of 4 timespans as 2-dimensional, horizontal fields. E.g. [[array_t1], [array_t2], [array_t3], [array_t4]] Returns:nfield – Interpolated flux at central point of accumulation timespan. Return type:array of float

Note

March 2000 : P. JAMES

Original author

June 2003 : A. BECK

Adaptations

November 2015 : Leopold Haimberger (University of Vienna)

Migration from Fortran to Python