Changeset 6f951ca in flex_extract.git


Ignore:
Timestamp:
Jan 15, 2019, 1:05:10 AM (5 years ago)
Author:
Anne Philipp <anne.philipp@…>
Branches:
master, ctbto, dev
Children:
d4696e0
Parents:
2625ca8
Message:

new style of docstring params and updates in docstrings

Location:
source/python
Files:
12 edited

Legend:

Unmodified
Added
Removed
  • source/python/_config.py

    r1eca806 r6f951ca  
    11#!/usr/bin/env python
    22# -*- coding: utf-8 -*-
     3#*******************************************************************************
     4# @Author: Anne Philipp (University of Vienna)
     5#
     6# @Date: August 2018
     7#
     8# @Change History:
     9#
     10# @License:
     11#    (C) Copyright 2014-2019.
     12#    Anne Philipp, Leopold Haimberger
     13#
     14#    This work is licensed under the Creative Commons Attribution 4.0
     15#    International License. To view a copy of this license, visit
     16#    http://creativecommons.org/licenses/by/4.0/ or send a letter to
     17#    Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.
     18#*******************************************************************************
     19'''Configurations for flex_extract.
     20
     21Defines a couple of global constant parameters as well as filenames
     22and pathes.
    323'''
    4 *******************************************************************************
    5  @Author: Anne Philipp (University of Vienna)
    6 
    7  @Date: August 2018
    8 
    9  @Change History:
    10 
    11  @License:
    12     (C) Copyright 2014-2018.
    13 
    14     This software is licensed under the terms of the Apache Licence Version 2.0
    15     which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
    16 
    17  @Description:
    18     Contains constant value parameter for flex_extract.
    19 
    20 *******************************************************************************
    21 '''
    22 
    2324# ------------------------------------------------------------------------------
    2425# MODULES
     
    9596#
    9697# ------------------------------------------------------------------------------
    97 # for making the installation tar ball, the relative pathes to the
    98 # flex_extract root directory are needed
     98# RELATIVE PATHES FOR INSTALLATION TAR-BALL
    9999# ------------------------------------------------------------------------------
    100100
  • source/python/classes/ControlFile.py

    rf2616a3 r6f951ca  
    2121#
    2222# @License:
    23 #    (C) Copyright 2015-2018.
     23#    (C) Copyright 2014-2019.
     24#    Anne Philipp, Leopold Haimberger
    2425#
    25 #    This software is licensed under the terms of the Apache Licence Version 2.0
    26 #    which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
     26#    This work is licensed under the Creative Commons Attribution 4.0
     27#    International License. To view a copy of this license, visit
     28#    http://creativecommons.org/licenses/by/4.0/ or send a letter to
     29#    Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.
    2730#
    28 # @Class Description:
    29 #    The CONTROL file is the steering part of the FLEXPART extraction
    30 #    software. All necessary parameters needed to retrieve the data fields
    31 #    from the MARS archive for driving FLEXPART are set in a CONTROL file.
    32 #    Some specific parameters like the start and end dates can be overwritten
    33 #    by the command line parameters, but in generel all parameters needed
    34 #    for a complete set of fields for FLEXPART can be set in the CONTROL file.
    35 #
    36 # @Class Content:
    37 #    - __init__
    38 #    - __read_controlfile__
    39 #    - __str__
    40 #    - assign_args_to_control
    41 #    - assign_envs_to_control
    42 #    - check_conditions
    43 #    - check_install_conditions
    44 #    - to_list
    45 #
    46 # @Class Attributes:
    47 #
    48 #
     31# @Class Methods:
     32#    __init__
     33#    _read_controlfile
     34#    __str__
     35#    assign_args_to_control
     36#    assign_envs_to_control
     37#    check_conditions
     38#    check_install_conditions
     39#    to_list
    4940#*******************************************************************************
    5041
     
    7667    '''
    7768    Contains the information which are stored in the CONTROL files.
     69
     70    The CONTROL file is the steering part of the FLEXPART extraction
     71    software. All necessary parameters needed to retrieve the data fields
     72    from the MARS archive for driving FLEXPART are set in a CONTROL file.
     73    Some specific parameters like the start and end dates can be overwritten
     74    by the command line parameters, but in generel all parameters needed
     75    for a complete set of fields for FLEXPART can be set in the CONTROL file.
     76
     77    Attributes
     78    ----------
     79    controlfile : str
     80        The name of the control file to be processed. Default value is the
     81        filename passed to the init function when initialised.
     82
     83    start_date : str
     84        The first day of the retrieval period. Default value is None.
     85
     86    end_date :str
     87        The last day of the retrieval period. Default value is None.
     88
     89    date_chunk : int
     90        Length of period for a single mars retrieval. Default value is 3.
     91
     92    dtime :str
     93        The time step in hours. Default value is None.
     94
     95    basetime : str
     96        The time for a half day retrieval. The 12 hours upfront are to be
     97        retrieved. Default value is None.
     98
     99    maxstep : int
     100        The maximum forecast step for non flux data. Default value is None.
     101
     102    type : list of str
     103        List of field type per retrieving hour. Default value is None.
     104
     105    time : list of str
     106        List of retrieving times in hours. Default valuer is None.
     107
     108    step : list of str or str
     109        List of forecast time steps in hours for non flux data.
     110        Default value is None.
     111
     112    acctype : str
     113        The field type for the accumulated forecast fields.
     114        Default value is None.
     115
     116    acctime : str
     117        The starting time of the accumulated forecasts. Default value is None.
     118
     119    accmaxstep : int
     120        The maximum forecast step for the accumulated forecast fields
     121        (flux data). Default value is None.
     122
     123    marsclass : str
     124        Characterisation of dataset. Default value is None.
     125
     126    dataset : str
     127        For public datasets there is the specific naming and parameter
     128        dataset which has to be used to characterize the type of
     129        data. Default value is None.
     130
     131    stream : str
     132        Identifies the forecasting system used to generate the data.
     133        Default value is None.
     134
     135    number : str
     136        Selects the member in ensemble forecast run. Default value is 'OFF'.
     137
     138    expver : str
     139        The version number of the dataset. Default value is '1'.
     140
     141    gaussian : str
     142        This parameter is deprecated and should no longer be used.
     143        Specifies the desired type of Gaussian grid for the output.
     144        Default value is an empty string ''.
     145
     146    grid : str
     147        Specifies the output grid which can be either a Gaussian grid
     148        or a Latitude/Longitude grid. Default value is None.
     149
     150    area : str
     151        Specifies the desired sub-area of data to be extracted.
     152        Default value is None.
     153
     154    left : str
     155        The western most longitude of the area to be extracted.
     156        Default value is None.
     157
     158    lower : str
     159        The southern most latitude of the area to be extracted.
     160        Default value is None.
     161
     162    upper : str
     163        The northern most latitued of the area to be extracted.
     164        Default value is None.
     165
     166    right : str
     167        The eastern most longitude of the area to be extracted.
     168        Default value is None.
     169
     170    level : str
     171        Specifies the maximum level. Default value is None.
     172
     173    levelist : str
     174        Specifies the required level list. Default value is None.
     175
     176    resol : str
     177        Specifies the desired triangular truncation of retrieved data,
     178        before carrying out any other selected post-processing.
     179        Default value is None.
     180
     181    gauss : int
     182        Switch to select gaussian fields (1) or regular lat/lon (0).
     183        Default value is 0.
     184
     185    accuracy : int
     186        Specifies the number of bits per value to be used in the
     187        generated GRIB coded fields. Default value is 24.
     188
     189    omega : int
     190       Switch to select omega retrieval (1) or not (0). Default value is 0.
     191
     192    omegadiff : int
     193        Switch to decide to calculate Omega and Dps/Dt from continuity
     194        equation for diagnostic purposes (1) or not (0). Default value is 0.
     195
     196    eta : int
     197        Switch to select direct retrieval of etadot from MARS (1) or
     198        wether it has to be calculated (0). Then Default value is 0.
     199
     200    etadiff : int
     201        Switch to select calculation of etadot and Dps/Dt from continuity
     202        equation for diagnostic purposes (1) or not (0). Default value is 0.
     203
     204    etapar : int
     205        GRIB parameter Id for etadot fields. Default value is 77.
     206
     207    dpdeta : int
     208        Switch to select multiplication of etadot with dpdeta.
     209        Default value is 1.
     210
     211    smooth : int
     212        Spectral truncation of ETADOT after calculation on Gaussian grid.
     213        Default value is 0.
     214
     215    format : str
     216        The format of the GRIB data. Default value is 'GRIB1'.
     217
     218    addpar : str
     219        List of additional surface level ECMWF parameter to be retrieved.
     220        Default value is None.
     221
     222    prefix : str
     223        Prefix string for the final FLEXPART/FLEXTRA ready input files.
     224        Default value is 'EN'.
     225
     226    cwc : int
     227        Switch to select wether the sum of cloud liquid water content and
     228        cloud ice water content should be retrieved. Default value is 0.
     229
     230    wrf : int
     231        Switch to select further parameters for retrievment to support
     232        WRF simulations. Default value is 0.
     233
     234    ecfsdir : str
     235        Path to the ECMWF storage  'ectmp:/${USER}/econdemand/'
     236
     237    mailfail : list of str
     238        Email list for sending error log files from ECMWF servers.
     239        The email addresses should be seperated by a comma.
     240        Default value is ['${USER}'].
     241
     242    mailops : list of str
     243        Email list for sending operational log files from ECMWF servers.
     244        The email addresses should be seperated by a comma.
     245        Default value is ['${USER}'].
     246
     247    grib2flexpart : int 0
     248        Switch to select generation of preprocessed FLEXPART files ".fp".
     249        If it is selected, the program grib2flexpart will try
     250        to convert the flex_extract output files into ".fp" format.
     251
     252    ecstorage : int
     253        Switch to select storage of FLEXPART ready output files
     254        in the ECFS file system. Default value is 0.
     255
     256    ectrans : int
     257        Switch to select the transfer of FLEXPART ready output files
     258        to the gateway server. Default value is 0.
     259
     260    inputdir : str
     261        Path to the temporary directory for the retrieval grib files and
     262        other processing files. Default value is _config.PATH_INPUT_DIR.
     263
     264    outputdir : str
     265        Path to the final directory where the final FLEXPART ready input
     266        files are stored. Default value is None.
     267
     268    flexextractdir : str
     269        Path to the flex_extract root directory. Default value is
     270        _config.PATH_FLEXEXTRACT_DIR.
     271
     272    exedir : str
     273        Path to the FORTRAN executable file. Default value is
     274        _config.PATH_FORTRAN_SRC.
     275
     276    flexpartdir : str
     277        Path to a FLEXPART root directory. Default value is None.
     278
     279    makefile : str
     280        Name of the makefile to be used for the Fortran program.
     281        Default value is 'Makefile.gfortran'.
     282
     283    destination : str
     284        The remote destination which is used to transfer files
     285        from ECMWF server to local gateway server. Default value is None.
     286
     287    gateway : str
     288        The gateway server the user is using. Default value is None.
     289
     290    ecuid : str
     291        The user id on ECMWF server. Default value is None.
     292
     293    ecgid : str
     294        The group id on ECMWF server. Default value is None.
     295
     296    install_target : str
     297        Defines the location where the installation is to be done.
     298        Default value is None.
     299
     300    debug : int
     301        Switch to keep temporary files at the end of postprocessing (1) or
     302        to delete all temporary files except the final output files (0).
     303        Default value is 0.
     304
     305    request : int
     306        Switch to select between just retrieving the data (0), writing the mars
     307        parameter values to a csv file (1) or doing both (2).
     308        Default value is 0.
     309
     310    public : int
     311        Switch to select kind of ECMWF Web Api access and the
     312        possible data sets. Public data sets (1) and Memberstate data sets (0).
     313        Default value is 0.
     314
     315    ecapi : boolean
     316        Tells wether the ECMWF Web APi was able to load or not.
     317        Default value is None.
     318
     319    purefc : int
     320        Switch to decide wether the job is a pure forecast retrieval or
     321        coupled with analysis data. Default value is 0.
     322
     323    rrint: int
     324        Switch to select between old precipitation disaggregation method (0)
     325        or the new IA3 disaggegration method (1). Default value is 0.
     326
     327    logicals : list of str
     328        List of the names of logical switches which controls the flow
     329        of the program. Default list is ['gauss', 'omega', 'omegadiff', 'eta',
     330        'etadiff', 'dpdeta', 'cwc', 'wrf', 'grib2flexpart', 'ecstorage',
     331        'ectrans', 'debug', 'request', 'public', 'purefc', 'rrint']
    78332    '''
    79333
     
    85339        Parameters
    86340        ----------
    87         filename : :obj:`string`
     341        filename : str
    88342            Name of CONTROL file.
    89343
     
    98352        self.end_date = None
    99353        self.date_chunk = 3
     354        self.job_chunk = None
    100355        self.dtime = None
    101356        self.basetime = None
     
    273528        Parameters
    274529        ----------
    275         args : :obj:`Namespace`
     530        args : Namespace
    276531            Contains the commandline arguments from script/program call.
    277532
     
    298553        Parameters
    299554        ----------
    300         envs : :obj:`dictionary` of :obj:`strings`
     555        envs : dict of str
    301556            Contains the ECMWF environment parameternames "ECUID", "ECGID",
    302557            "DESTINATION" and "GATEWAY" with its corresponding values.
     
    320575        Parameters
    321576        ----------
    322         queue : :obj:`string`
     577        queue : str
    323578            Name of the queue if submitted to the ECMWF servers.
    324579            Used to check if ecuid, ecgid, gateway and destination
     
    397652        Return
    398653        ------
    399         l : :obj:`list`
     654        l : list of *
    400655            A sorted list of the all ControlFile class attributes with
    401656            their values except the attributes "_expanded", "exedir",
  • source/python/classes/EcFlexpart.py

    rbf48c8a r6f951ca  
    4141#
    4242# @License:
    43 #    (C) Copyright 2014-2018.
     43#    (C) Copyright 2014-2019.
     44#    Anne Philipp, Leopold Haimberger
    4445#
    45 #    This software is licensed under the terms of the Apache Licence Version 2.0
    46 #    which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
    47 #
    48 # @Class Description:
    49 #    FLEXPART needs grib files in a specifc format. All necessary data fields
    50 #    for one time step are stored in a single file. The class represents an
    51 #    instance with all the parameter and settings necessary for retrieving
    52 #    MARS data and modifing them so they are fitting FLEXPART need. The class
    53 #    is able to disaggregate the fluxes and convert grid types to the one needed
    54 #    by FLEXPART, therefore using the FORTRAN program.
    55 #
    56 # @Class Content:
    57 #    - __init__
    58 #    - write_namelist
    59 #    - retrieve
    60 #    - process_output
    61 #    - create
    62 #    - deacc_fluxes
    63 #
    64 # @Class Attributes:
    65 #
    66 #  TODO
    67 #
     46#    This work is licensed under the Creative Commons Attribution 4.0
     47#    International License. To view a copy of this license, visit
     48#    http://creativecommons.org/licenses/by/4.0/ or send a letter to
     49#    Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.
    6850#*******************************************************************************
    6951#pylint: disable=unsupported-assignment-operation
     
    10284class EcFlexpart(object):
    10385    '''
    104     Class to retrieve FLEXPART specific ECMWF data.
     86    Class to represent FLEXPART specific ECMWF data.
     87
     88    FLEXPART needs grib files in a specifc format. All necessary data fields
     89    for one time step are stored in a single file. The class represents an
     90    instance with all the parameter and settings necessary for retrieving
     91    MARS data and modifing them so they are fitting FLEXPART needs. The class
     92    is able to disaggregate the fluxes and convert grid types to the one needed
     93    by FLEXPART, therefore using the FORTRAN program.
     94
     95    Attributes
     96    ----------
     97    mreq_count : int
     98        Counter for the number of generated mars requests.
     99
     100    inputdir : str
     101        Path to the directory where the retrieved data is stored.
     102
     103    dataset : str
     104        For public datasets there is the specific naming and parameter
     105        dataset which has to be used to characterize the type of
     106        data.
     107
     108    basetime : str
     109        The time for a half day retrieval. The 12 hours upfront are to be
     110        retrieved.
     111
     112    dtime : str
     113        Time step in hours.
     114
     115    acctype : str
     116        The field type for the accumulated forecast fields.
     117
     118    acctime : str
     119        The starting time from the accumulated forecasts.
     120
     121    accmaxstep : str
     122        The maximum forecast step for the accumulated forecast fields.
     123
     124    marsclass : str
     125        Characterisation of dataset.
     126
     127    stream : str
     128        Identifies the forecasting system used to generate the data.
     129
     130    number : str
     131        Selects the member in ensemble forecast run.
     132
     133    resol : str
     134        Specifies the desired triangular truncation of retrieved data,
     135        before carrying out any other selected post-processing.
     136
     137    accuracy : str
     138        Specifies the number of bits per value to be used in the
     139        generated GRIB coded fields.
     140
     141    addpar : str
     142        List of additional parameters to be retrieved.
     143
     144    level : str
     145        Specifies the maximum level.
     146
     147    expver : str
     148        The version of the dataset.
     149
     150    levelist : str
     151        Specifies the required levels.
     152
     153    glevelist : str
     154        Specifies the required levels for gaussian grids.
     155
     156    gaussian : str
     157        This parameter is deprecated and should no longer be used.
     158        Specifies the desired type of Gaussian grid for the output.
     159
     160    grid : str
     161        Specifies the output grid which can be either a Gaussian grid
     162        or a Latitude/Longitude grid.
     163
     164    area : str
     165        Specifies the desired sub-area of data to be extracted.
     166
     167    purefc : int
     168        Switch for definition of pure forecast mode or not.
     169
     170    outputfilelist : list of str
     171        The final list of FLEXPART ready input files.
     172
     173    types : dictionary
     174        Determines the combination of type of fields, time and forecast step
     175        to be retrieved.
     176
     177    params : dictionary
     178        Collection of grid types and their corresponding parameters,
     179        levels, level types and the grid definition.
     180
     181    server : ECMWFService or ECMWFDataServer
     182        This is the connection to the ECMWF data servers.
     183
     184    public : int
     185        Decides which Web API Server version is used.
     186
     187    dates : str
     188        Contains start and end date of the retrieval in the format
     189        "YYYYMMDD/to/YYYYMMDD"
    105190    '''
     191
    106192    # --------------------------------------------------------------------------
    107193    # CLASS FUNCTIONS
     
    113199        Parameters:
    114200        -----------
    115         c : :obj:`ControlFile`
     201        c : ControlFile
    116202            Contains all the parameters of CONTROL file and
    117203            command line.
    118204
    119         fluxes : :obj:`boolean`, optional
     205        fluxes : boolean, optional
    120206            Decides if the flux parameter settings are stored or
    121207            the rest of the parameter list.
     
    174260        Parameters:
    175261        -----------
    176         ftype : :obj:`list` of :obj:`string`
     262        ftype : list of str
    177263            List of field types.
    178264
    179         ftime : :obj:`list` of :obj:`string`
     265        ftime : list of str
    180266            The time in hours of the field.
    181267
    182         fstep : :obj:`string`
     268        fstep : str
    183269            Specifies the forecast time step from forecast base time.
    184270            Valid values are hours (HH) from forecast base time.
     
    267353        Parameters:
    268354        -----------
    269         gauss : :obj:`integer`
     355        gauss : int
    270356            Gaussian grid is retrieved.
    271357
    272         eta : :obj:`integer`
     358        eta : int
    273359            Etadot parameter will be directly retrieved.
    274360
    275         omega : :obj:`integer`
     361        omega : int
    276362            The omega paramterwill be retrieved.
    277363
    278         cwc : :obj:`integer`
     364        cwc : int
    279365            The cloud liquid and ice water content will be retrieved.
    280366
    281         wrf : :obj:`integer`
     367        wrf : int
    282368            Additional model level and surface level data will be retrieved for
    283369            WRF/FLEXPART-WRF simulations.
     
    374460        Parameters
    375461        ----------
    376         ftype : :obj:`string`
     462        ftype : str
    377463            Shortcut name of the type of the field. E.g. AN, FC, PF, ...
    378464
    379         param : :obj:`string`
     465        param : str
    380466            Shortcut of the grid type. E.g. SH__ML, SH__SL, GG__ML,
    381467            GG__SL, OG__ML, OG__SL, OG_OROLSM_SL, OG_acc_SL
    382468
    383         date : :obj:`string`
     469        date : str
    384470            The date period of the grib data to be stored in this file.
    385471
    386472        Return
    387473        ------
    388         targetname : :obj:`string`
     474        targetname : str
    389475            The target filename for the grib data.
    390476        '''
     
    401487        Parameters
    402488        ----------
    403         request : :obj:`integer`
     489        request : int
    404490            Selects the mode of retrieval.
    405491            0: Retrieves the data from ECMWF.
     
    407493            2: Retrieves the data and prints the mars request.
    408494
    409         par_dict : :obj:`dictionary`
     495        par_dict : dictionary
    410496            Contains all parameter which have to be set for creating the
    411497            Mars Retrievals. The parameter are:
     
    463549        Parameters
    464550        ----------
    465         keys : :obj:`dictionary`
     551        keys : dictionary
    466552            List of parameter names which serves as index.
    467553
    468         inputfiles : :obj:`UioFiles`
     554        inputfiles : UioFiles
    469555            Contains a list of files.
    470556
    471557        Return
    472558        ------
    473         iid : :obj:`codes_index`
     559        iid : codes_index
    474560            This is a grib specific index structure to access
    475561            messages in a file.
    476562
    477         index_vals : :obj:`list`
     563        index_vals : list of list  of str
    478564            Contains the values from the keys used for a distinct selection
    479565            of grib messages in processing  the grib files.
     
    517603        Parameters
    518604        ----------
    519         server : :obj:`ECMWFService` or :obj:`ECMWFDataServer`
     605        server : ECMWFService or ECMWFDataServer
    520606            The connection to the ECMWF server. This is different
    521607            for member state users which have full access and non
     
    525611            for member state users its False (ECMWFService)
    526612
    527         dates : :obj:`string`
     613        dates : str
    528614            Contains start and end date of the retrieval in the format
    529615            "YYYYMMDD/to/YYYYMMDD"
    530616
    531         request : :obj:`integer`
     617        request : int
    532618            Selects the mode of retrieval.
    533619            0: Retrieves the data from ECMWF.
     
    535621            2: Retrieves the data and prints the mars request.
    536622
    537         inputdir : :obj:`string`, optional
     623        inputdir : str, optional
    538624            Path to the directory where the retrieved data is about
    539625            to be stored. The default is the current directory ('.').
     
    723809        Parameters
    724810        ----------
    725         c : :obj:`ControlFile`
     811        c : ControlFile
    726812            Contains all the parameters of CONTROL file and
    727813            command line.
    728814
    729         filename : :obj:`string`
     815        filename : str
    730816                Name of the namelist file.
    731817
     
    810896        Parameters
    811897        ----------
    812         inputfiles : :obj:`UioFiles`
     898        inputfiles : UioFiles
    813899            Contains the list of files that contain flux data.
    814900
    815         c : :obj:`ControlFile`
     901        c : ControlFile
    816902            Contains all the parameters of CONTROL file and
    817903            command line.
     
    837923        # index_vals[0]: ('20171106', '20171107', '20171108') ; date
    838924        # index_vals[1]: ('0', '600', '1200', '1800') ; time
    839         # index_vals[2]: ('0', '12', '3', '6', '9') ; stepRange
     925        # index_vals[2]: ('0', '3', '6', '9', '12') ; stepRange
    840926
    841927        if c.rrint:
     
    10961182        Parameters
    10971183        ----------
    1098         ni : :obj:`integer`
     1184        ni : int
    10991185            Amount of zonal grid points.
    11001186
    1101         nj : :obj:`integer`
     1187        nj : int
    11021188            Amount of meridional grid points.
    11031189
    1104         nt : :obj:`integer`
     1190        nt : int
    11051191            Number of time steps.
    11061192
    1107         lsp_np : :obj:`numpy array` of :obj:`float`
     1193        lsp_np : numpy array of float
    11081194            The large scale precipitation fields for each time step.
    11091195            Shape (ni * nj, nt).
    11101196
    1111         cp_np : :obj:`numpy array` of :obj:`float`
     1197        cp_np : numpy array of float
    11121198            The convective precipitation fields for each time step.
    11131199            Shape (ni * nj, nt).
    11141200
    1115         date_list : :obj:`list` of :obj:`datetime`
     1201        date_list : list of datetime
    11161202            The list of dates for which the disaggregation is to be done.
    11171203
    1118         step_list : :obj:`list` of :obj:`integer`
     1204        step_list : list of int
    11191205            The list of steps for a single forecast time.
    11201206            Only necessary for pure forecasts.
    11211207
    1122         c : :obj:`ControlFile`
     1208        c : ControlFile
    11231209            Contains all the parameters of CONTROL file and
    11241210            command line.
     
    12171303        Parameters
    12181304        ----------
    1219         ifile : :obj:`string`
     1305        ifile : str
    12201306            Filename of the input file to read the grib messages from.
    12211307
    1222         inputdir : :obj:`string`, optional
     1308        inputdir : str, optional
    12231309            Path to the directory where the retrieved data is stored.
    12241310
     
    12551341        Parameters
    12561342        ----------
    1257         inputfiles : :obj:`UioFiles`
     1343        inputfiles : UioFiles
    12581344            Contains a list of files.
    12591345
    1260         c : :obj:`ControlFile`
     1346        c : ControlFile
    12611347            Contains all the parameters of CONTROL file and
    12621348            command line.
     
    14861572        Parameters
    14871573        ----------
    1488         c : :obj:`ControlFile`
     1574        c : ControlFile
    14891575            Contains all the parameters of CONTROL file and
    14901576            command line.
     
    15451631        Parameters
    15461632        ----------
    1547         c : :obj:`ControlFile`
     1633        c : ControlFile
    15481634            Contains all the parameters of CONTROL file and
    15491635            command line.
  • source/python/classes/MarsRetrieval.py

    rbc1720f r6f951ca  
    1717#        - added documentation
    1818#        - applied some minor modifications in programming style/structure
     19#        - added writing of mars request attributes to a csv file
    1920#
    2021# @License:
    21 #    (C) Copyright 2015-2018.
     22#    (C) Copyright 2014-2019.
     23#    Anne Philipp, Leopold Haimberger
    2224#
    23 #    This software is licensed under the terms of the Apache Licence Version 2.0
    24 #    which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
    25 #
    26 # @Class Description:
    27 #    A MARS revtrieval has a specific syntax with a selection of keywords and
    28 #    their corresponding values. This class provides the necessary functions
    29 #    by displaying the selected parameters and their values and the actual
    30 #    retrievement of the data through a mars request or a Python web api
    31 #    interface. The initialization already expects all the keyword values.
    32 #
    33 # @Class Content:
    34 #    - __init__
    35 #    - display_info
    36 #    - data_retrieve
    37 #
    38 # @Class Attributes:
    39 #    - server
    40 #    - marsclass
    41 #    - dtype
    42 #    - levtype
    43 #    - levelist
    44 #    - repres
    45 #    - date
    46 #    - resol
    47 #    - stream
    48 #    - area
    49 #    - time
    50 #    - step
    51 #    - expver
    52 #    - number
    53 #    - accuracy
    54 #    - grid
    55 #    - gaussian
    56 #    - target
    57 #    - param
    58 #
     25#    This work is licensed under the Creative Commons Attribution 4.0
     26#    International License. To view a copy of this license, visit
     27#    http://creativecommons.org/licenses/by/4.0/ or send a letter to
     28#    Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.
    5929#*******************************************************************************
    6030
     
    7343# ------------------------------------------------------------------------------
    7444class MarsRetrieval(object):
    75     '''Class for submitting MARS retrievals.
     45    '''Specific syntax and content for submission of MARS retrievals.
     46
     47    A MARS revtrieval has a specific syntax with a selection of keywords and
     48    their corresponding values. This class provides the necessary functions
     49    by displaying the selected parameters and their values and the actual
     50    retrievement of the data through a mars request or a Python web api
     51    interface. The initialization already expects all the keyword values.
    7652
    7753    A description of MARS keywords/arguments and examples of their
     
    7955    https://software.ecmwf.int/wiki/display/UDOC/\
    8056                   Identification+keywords#Identificationkeywords-class
     57
     58    Attributes
     59    ----------
     60    server : ECMWFService or ECMWFDataServer
     61        This is the connection to the ECMWF data servers.
     62
     63    public : int
     64        Decides which Web API Server version is used.
     65
     66    marsclass : str, optional
     67        Characterisation of dataset.
     68
     69    dataset : str, optional
     70        For public datasets there is the specific naming and parameter
     71        dataset which has to be used to characterize the type of
     72        data.
     73
     74    type : str, optional
     75        Determines the type of fields to be retrieved.
     76
     77    levtype : str, optional
     78        Denotes type of level.
     79
     80    levelist : str, optional
     81        Specifies the required levels.
     82
     83    repres : str, optional
     84        Selects the representation of the archived data.
     85
     86    date : str, optional
     87        Specifies the Analysis date, the Forecast base date or
     88        Observations date.
     89
     90    resol : str, optional
     91        Specifies the desired triangular truncation of retrieved data,
     92        before carrying out any other selected post-processing.
     93
     94    stream : str, optional
     95        Identifies the forecasting system used to generate the data.
     96
     97    area : str, optional
     98        Specifies the desired sub-area of data to be extracted.
     99
     100    time : str, optional
     101        Specifies the time of the data in hours and minutes.
     102
     103    step : str, optional
     104        Specifies the forecast time step from forecast base time.
     105
     106    expver : str, optional
     107        The version of the dataset.
     108
     109    number : str, optional
     110        Selects the member in ensemble forecast run.
     111
     112    accuracy : str, optional
     113        Specifies the number of bits per value to be used in the
     114        generated GRIB coded fields.
     115
     116    grid : str, optional
     117        Specifies the output grid which can be either a Gaussian grid
     118        or a Latitude/Longitude grid.
     119
     120    gaussian : str, optional
     121        This parameter is deprecated and should no longer be used.
     122        Specifies the desired type of Gaussian grid for the output.
     123
     124    target : str, optional
     125        Specifies a file into which data is to be written after
     126        retrieval or manipulation.
     127
     128    param : str, optional
     129        Specifies the meteorological parameter.
    81130    '''
    82131
     
    96145        Parameters
    97146        ----------
    98         server : :obj:`ECMWFService`
     147        server : ECMWFService or ECMWFDataServer
    99148            This is the connection to the ECMWF data servers.
    100149            It is needed for the pythonic access of ECMWF data.
    101150
    102         public : :obj:`integer`
     151        public : int
    103152            Decides which Web API version is used:
    104153            0: member-state users and full archive access
     
    107156            Default is "0" and for member-state users.
    108157
    109         marsclass : :obj:`string`, optional
     158        marsclass : str, optional
    110159            Characterisation of dataset. E.g. EI (ERA-Interim),
    111160            E4 (ERA40), OD (Operational archive), ea (ERA5).
    112161            Default is the ERA-Interim dataset "ei".
    113162
    114         dataset : :obj:`string`, optional
     163        dataset : str, optional
    115164            For public datasets there is the specific naming and parameter
    116165            dataset which has to be used to characterize the type of
     
    119168            Default is an empty string.
    120169
    121         type : :obj:`string`, optional
     170        type : str, optional
    122171            Determines the type of fields to be retrieved.
    123172            Selects between observations, images or fields.
     
    126175            Default is an empty string.
    127176
    128         levtype : :obj:`string`, optional
     177        levtype : str, optional
    129178            Denotes type of level. Has a direct implication on valid
    130179            levelist values!
     
    134183            Default is an empty string.
    135184
    136         levelist : :obj:`string`, optional
     185        levelist : str, optional
    137186            Specifies the required levels. It has to have a valid
    138187            correspondence to the selected levtype.
     
    140189            Default is an empty string.
    141190
    142         repres : :obj:`string`, optional
     191        repres : str, optional
    143192            Selects the representation of the archived data.
    144193            E.g. sh - spherical harmonics, gg - Gaussian grid,
     
    146195            Default is an empty string.
    147196
    148         date : :obj:`string`, optional
     197        date : str, optional
    149198            Specifies the Analysis date, the Forecast base date or
    150199            Observations date. Valid formats are:
     
    152201            Default is an empty string.
    153202
    154         resol : :obj:`string`, optional
     203        resol : str, optional
    155204            Specifies the desired triangular truncation of retrieved data,
    156205            before carrying out any other selected post-processing.
     
    169218            Default is an empty string.
    170219
    171         stream : :obj:`string`, optional
     220        stream : str, optional
    172221            Identifies the forecasting system used to generate the data.
    173222            E.g. oper (Atmospheric model), enfo (Ensemble forecats), ...
    174223            Default is an empty string.
    175224
    176         area : :obj:`string`, optional
     225        area : str, optional
    177226            Specifies the desired sub-area of data to be extracted.
    178227            Areas can be defined to wrap around the globe.
     
    192241            Default is an empty string.
    193242
    194         time : :obj:`string`, optional
     243        time : str, optional
    195244            Specifies the time of the data in hours and minutes.
    196245            Valid values depend on the type of data: Analysis time,
     
    202251            Default is an empty string.
    203252
    204         step : :obj:`string`, optional
     253        step : str, optional
    205254            Specifies the forecast time step from forecast base time.
    206255            Valid values are hours (HH) from forecast base time. It also
     
    210259            Default is an empty string.
    211260
    212         expver : :obj:`string`, optional
     261        expver : str, optional
    213262            The version of the dataset. Each experiment is assigned a
    214263            unique code (version). Production data is assigned 1 or 2,
     
    218267            Default is "1".
    219268
    220         number : :obj:`string`, optional
     269        number : str, optional
    221270            Selects the member in ensemble forecast run. (Only then it
    222271            is necessary.) It has a different meaning depending on
     
    225274            Default is an empty string.
    226275
    227         accuracy : :obj:`string`, optional
     276        accuracy : str, optional
    228277            Specifies the number of bits per value to be used in the
    229278            generated GRIB coded fields.
     
    236285            Default is an empty string.
    237286
    238         grid : :obj:`string`, optional
     287        grid : str, optional
    239288            Specifies the output grid which can be either a Gaussian grid
    240289            or a Latitude/Longitude grid. MARS requests specifying
     
    257306            Default is an empty string.
    258307
    259         gaussian : :obj:`string`, optional
     308        gaussian : str, optional
    260309            This parameter is deprecated and should no longer be used.
    261310            Specifies the desired type of Gaussian grid for the output.
     
    265314            Default is an empty string.
    266315
    267         target : :obj:`string`, optional
     316        target : str, optional
    268317            Specifies a file into which data is to be written after
    269318            retrieval or manipulation. Path names should always be
     
    279328            Default is an empty string.
    280329
    281         param : :obj:`string`, optional
     330        param : str, optional
    282331            Specifies the meteorological parameter.
    283332            The list of meteorological parameters in MARS is extensive.
     
    352401        Parameters
    353402        ----------
    354         inputdir : :obj:`string`
     403        inputdir : str
    355404            The path where all data from the retrievals are stored.
    356405
    357         request_number : :obj:`integer`
     406        request_number : int
    358407            Number of mars requests for flux and non-flux data.
    359408
  • source/python/classes/UioFiles.py

    r5cb0eaa r6f951ca  
    1212#        - added pattern search in method list_files
    1313#
    14 #    February 2018 - Anne Philipp (University of Vienna):
     14#    February - December 2018 - Anne Philipp (University of Vienna):
    1515#        - applied PEP8 style guide
    1616#        - added documentation
     
    2020#        - modified the class so that it is initiated with a pattern instead
    2121#          of suffixes. Gives more precision in selection of files.
     22#        - added delete method
    2223#
    2324# @License:
    24 #    (C) Copyright 2014-2018.
     25#    (C) Copyright 2014-2019.
     26#    Anne Philipp, Leopold Haimberger
    2527#
    26 #    This software is licensed under the terms of the Apache Licence Version 2.0
    27 #    which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
    28 #
    29 # @Class Decription:
    30 #    The class is for file manipulation. It is initiated with a regular
    31 #    expression pattern for this instance and can produce a list of Files
    32 #    from the given file pattern. These files can be deleted.
    33 #
    34 # @Class Content:
    35 #    - __init__
    36 #    - __str__
    37 #    - __list_files__
    38 #    - delete_files
    39 #
    40 # @Class Attributes:
    41 #    - pattern
    42 #    - files
    43 #
     28#    This work is licensed under the Creative Commons Attribution 4.0
     29#    International License. To view a copy of this license, visit
     30#    http://creativecommons.org/licenses/by/4.0/ or send a letter to
     31#    Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.
    4432#*******************************************************************************
    4533
     
    5139import fnmatch
    5240
    53 # software specific module from flex_extract
     41# software specific modules from flex_extract
    5442sys.path.append('../')
    55 #import profiling
    5643from mods.tools import silent_remove, get_list_as_string
    5744
     
    6148
    6249class UioFiles(object):
    63     '''Class to manipulate files. At initialisation it has the pattern
    64     which stores a regular expression pattern for the files, the path
    65     to the files and the files already.
     50    '''Collection of files matching a specific pattern.
     51
     52    The pattern can contain regular expressions for the files.
     53    The files are listed and can be transformed to a single string or
     54    they can be deleted.
     55
     56    Attributes
     57    ----------
     58    path : str
     59        Directory where to list the files.
     60
     61    pattern : str
     62        Regular expression pattern. For example: '\*.grb'
     63
     64    files : list of str
     65        List of files matching the pattern in the path.
    6666    '''
    6767    # --------------------------------------------------------------------------
    68     # CLASS FUNCTIONS
     68    # CLASS METHODS
    6969    # --------------------------------------------------------------------------
    7070    def __init__(self, path, pattern):
     
    7373        Parameters
    7474        ----------
    75         path : :obj:`string`
     75        path : str
    7676            Directory where to list the files.
    7777
    78         pattern : :obj:`string`
     78        pattern : str
    7979            Regular expression pattern. For example: '\*.grb'
    8080
     
    9292        return
    9393
    94     #@profiling.timefn
     94
    9595    def _list_files(self, path):
    9696        '''Lists all files in the directory with the matching
     
    9999        Parameters
    100100        ----------
    101         path : :obj:`string`
     101        path : str
    102102            Path to the files.
    103103
     
    116116        return
    117117
     118
    118119    def __str__(self):
    119120        '''Converts the list of files into a single string.
     
    125126        Return
    126127        ------
    127         files_string : :obj:`string`
     128        files_string : str
    128129            The content of the list as a single string.
    129130        '''
     
    133134
    134135        return files_string
     136
    135137
    136138    def delete_files(self):
  • source/python/install.py

    r1eca806 r6f951ca  
    1414#        - splitted code in smaller functions
    1515#        - delete convert build files in here instead of compile job script
    16 #        - changed static path names to Variables from config file
     16#        - changed static path names to variables from config file
     17#        - splitted install function into several smaller pieces
     18#        - use of tarfile package in python
    1719#
    1820# @License:
    19 #    (C) Copyright 2015-2018.
     21#    (C) Copyright 2014-2019.
     22#    Anne Philipp, Leopold Haimberger
    2023#
    21 #    This software is licensed under the terms of the Apache Licence Version 2.0
    22 #    which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
     24#    This work is licensed under the Creative Commons Attribution 4.0
     25#    International License. To view a copy of this license, visit
     26#    http://creativecommons.org/licenses/by/4.0/ or send a letter to
     27#    Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.
    2328#
    24 # @Program Functionality:
    25 #    Depending on the selected installation environment (locally or on the
    26 #    ECMWF server ecgate or cca) the program extracts the commandline
    27 #    arguments and the CONTROL file parameter and prepares the corresponding
    28 #    environment. The necessary files are collected in a tar-ball and placed
    29 #    at the target location. There its untared, the environment variables will
    30 #    be set and the Fortran code will be compiled. If the ECMWF environment is
    31 #    selected a job script is prepared and submitted for the remaining
    32 #    configurations after putting the tar-ball to the target ECMWF server.
    33 #
    34 # @Program Content:
    35 #    - main
    36 #    - get_install_cmdline_args
    37 #    - install_via_gateway
    38 #    - mk_tarball
    39 #    - un_tarball
    40 #    - mk_env_vars
    41 #    - mk_compilejob
    42 #    - mk_job_template
    43 #    - del_convert_build
    44 #    - mk_convert_build
     29# @Methods:
     30#    main
     31#    get_install_cmdline_args
     32#    install_via_gateway
     33#    check_install_conditions
     34#    mk_tarball
     35#    un_tarball
     36#    mk_env_vars
     37#    mk_compilejob
     38#    mk_job_template
     39#    del_convert_build
     40#    mk_convert_build
    4541#
    4642#*******************************************************************************
     43'''This script installs the flex_extract program.
     44
     45Depending on the selected installation environment (locally or on the
     46ECMWF server ecgate or cca) the program extracts the commandline
     47arguments and the CONTROL file parameter and prepares the corresponding
     48environment.
     49The necessary files are collected in a tar-ball and placed
     50at the target location. There its untared, the environment variables will
     51be set and the Fortran code will be compiled.
     52If the ECMWF environment is selected a job script is prepared and submitted
     53for the remaining configurations after putting the tar-ball to the
     54target ECMWF server.
     55
     56Type: install.py --help
     57to get information about command line parameters.
     58Read the documentation for usage instructions.
     59'''
    4760
    4861# ------------------------------------------------------------------------------
     
    96109    Return
    97110    ------
    98     args : :obj:`Namespace`
     111    args : Namespace
    99112        Contains the commandline arguments from script/program call.
    100113    '''
     
    144157    Parameters
    145158    ----------
    146     c : :obj:`ControlFile`
     159    c : ControlFile
    147160        Contains all the parameters of CONTROL file and
    148161        command line.
     
    229242    Parameters
    230243    ----------
    231     c : :obj:`ControlFile`
     244    c : ControlFile
    232245        Contains all the parameters of CONTROL file and
    233246        command line.
     
    283296    Parameters
    284297    ----------
    285     tarball_path : :obj:`string`
     298    tarball_path : str
    286299        The complete path to the tar file which will contain all
    287300        relevant data for flex_extract.
    288301
    289     target : :obj:`string`
     302    target : str
    290303        The queue where the job is submitted to.
    291304
     
    374387    Parameters
    375388    ----------
    376     tarball_path : :obj:`string`
     389    tarball_path : str
    377390        The complete path to the tar file which will contain all
    378391        relevant data for flex_extract.
     
    407420    Parameters
    408421    ----------
    409     ecuid : :obj:`string`
     422    ecuid : str
    410423        The user id on ECMWF server.
    411424
    412     ecgid : :obj:`string`
     425    ecgid : str
    413426        The group id on ECMWF server.
    414427
    415     gateway : :obj:`string`
     428    gateway : str
    416429        The gateway server the user is using.
    417430
    418     destination : :obj:`string`
     431    destination : str
    419432        The remote destination which is used to transfer files
    420433        from ECMWF server to local gateway server.
     
    469482    Parameters
    470483    ----------
    471     makefile : :obj:`string`
     484    makefile : str
    472485        Name of the makefile which should be used to compile FORTRAN
    473486        CONVERT2 program.
    474487
    475     target : :obj:`string`
     488    target : str
    476489        The target where the installation should be done, e.g. the queue.
    477490
    478     ecuid : :obj:`string`
     491    ecuid : str
    479492        The user id on ECMWF server.
    480493
    481     ecgid : :obj:`string`
     494    ecgid : str
    482495        The group id on ECMWF server.
    483496
    484     fp_root : :obj:`string`
     497    fp_root : str
    485498       Path to the root directory of FLEXPART environment or flex_extract
    486499       environment.
     
    544557    Parameters
    545558    ----------
    546     ecuid : :obj:`string`
     559    ecuid : str
    547560        The user id on ECMWF server.
    548561
    549     ecgid : :obj:`string`
     562    ecgid : str
    550563        The group id on ECMWF server.
    551564
    552     gateway : :obj:`string`
     565    gateway : str
    553566        The gateway server the user is using.
    554567
    555     destination : :obj:`string`
     568    destination : str
    556569        The remote destination which is used to transfer files
    557570        from ECMWF server to local gateway server.
    558571
    559     fp_root : :obj:`string`
     572    fp_root : str
    560573       Path to the root directory of FLEXPART environment or flex_extract
    561574       environment.
     
    622635    Parameters
    623636    ----------
    624     src_path : :obj:`string`
     637    src_path : str
    625638        Path to the fortran source directory.
    626639
     
    645658    Parameters
    646659    ----------
    647     src_path : :obj:`string`
     660    src_path : str
    648661        Path to the fortran source directory.
    649662
    650     makefile : :obj:`string`
     663    makefile : str
    651664        The name of the makefile which should be used.
    652665
  • source/python/mods/checks.py

    rf2616a3 r6f951ca  
    11#!/usr/bin/env python
    22# -*- coding: utf-8 -*-
    3 ##*******************************************************************************
     3#*******************************************************************************
    44# @Author: Anne Philipp (University of Vienna)
    55#
     
    99#
    1010# @License:
    11 #    (C) Copyright 2014-2018.
     11#    (C) Copyright 2014-2019.
     12#    Anne Philipp, Leopold Haimberger
    1213#
    13 #    This software is licensed under the terms of the Apache Licence Version 2.0
    14 #    which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
    15 #
    16 # @Modul Description:
    17 #
    18 #
    19 # @Module Content:
    20 
    21 #
     14#    This work is licensed under the Creative Commons Attribution 4.0
     15#    International License. To view a copy of this license, visit
     16#    http://creativecommons.org/licenses/by/4.0/ or send a letter to
     17#    Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.
    2218#*******************************************************************************
     19'''This module contains check methods for the CONTROL paramaters.
     20'''
    2321
    2422# ------------------------------------------------------------------------------
     
    2725
    2826import os
     27import sys
    2928import _config
    3029import exceptions
     
    4039    Parameters
    4140    ----------
    42     c : :obj:`ControlFile`
     41    c : ControlFile
    4342        Contains all the parameters of CONTROL file and
    4443        command line.
    4544
    46     logicals : :obj:`list` of (:obj:`string` or :obj:`integer`)
     45    logicals : list of (str or int)
    4746        Names of the switches that are used to control the flow of the
    4847        program.
     
    6766    Parameters
    6867    ----------
    69     grid : :obj:`string`
     68    grid : str
    7069        Contains grid information
    7170
    7271    Return
    7372    ------
    74     grid : :obj:``string`
     73    grid : str
    7574        Contains grid in format Lat/lon. E.g. 0.1/0.1
    7675    '''
     
    108107    Parameters
    109108    ----------
    110     grid : :obj:`string`
     109    grid : str
    111110        Contains grid information.
    112111
    113     area : :obj:`string`
     112    area : str
    114113        Contains area informtion.
    115114
    116     upper : :obj:`string`
     115    upper : str
    117116        The northern most latitude.
    118117
    119     lower : :obj:`string`
     118    lower : str
    120119        The souther most latitude.
    121120
    122     left : :obj:`string`
     121    left : str
    123122        The western most longitude.
    124123
    125     right : :obj:`string`
     124    right : str
    126125        The eastern most longiude.
    127126
    128127    Return
    129128    ------
    130     grid : :obj:``string`
     129    grid : str
    131130        Contains grid in format Lat/lon. E.g. 0.1/0.1
    132131    '''
     
    173172    Parameters
    174173    ----------
    175     levelist : :obj:`string`
     174    levelist : str
    176175        Specifies the level list.
    177176        Examples: model level: 1/to/137, pressure levels: 500/to/1000
    178177
    179     level : :obj:`string`
     178    level : str
    180179        Specifies the maximum level.
    181180
    182181    Return
    183182    ------
    184     levelist : :obj:`string`
     183    levelist : str
    185184        Specifies the required levels. It has to have a valid
    186185        correspondence to the selected levtype.
    187186        Examples: model level: 1/to/137, pressure levels: 500/to/1000
    188187
    189     level : :obj:`string`
     188    level : str
    190189        Specifies the maximum level. It has to be one of the
    191190        available maximum level number as contained in the variable
     
    221220    Parameters
    222221    ----------
    223     c : :obj:`ControlFile`
     222    c : ControlFile
    224223            Contains all the parameters of CONTROL file and
    225224            command line.
    226225
    227     ppid : :obj:`int` or :obj:`None`
     226    ppid : int or None
    228227        Contains the ppid number provided by the command line parameter
    229228        of is None otherwise.
     
    247246    Parameters
    248247    ----------
    249     ftype : :obj:`list` of :obj:`string`
     248    ftype : list of str
    250249        List of field types.
    251250
     
    273272    Parameters
    274273    ----------
    275     step : :obj:`list` of :obj:`string` or :obj:`string`
     274    step : list of str or str
    276275        Specifies the forecast time step from forecast base time.
    277276        Valid values are hours (HH) from forecast base time.
    278277
    279     mailfail : :obj:`list` of :obj:``string`
     278    mailfail : list of str
    280279        Contains all email addresses which should be notified.
    281280        It might also contain just the ecmwf user name which will trigger
     
    284283    Return
    285284    ------
    286     step : :obj:`list` of :obj:`string`
     285    step : list of str
    287286        List of forecast steps in format e.g. [001, 002, ...]
    288287    '''
     
    313312    Parameters
    314313    ----------
    315     ftype : :obj:`list` of :obj:`string` or :obj:`string`
     314    ftype : list of str or str
    316315        List of field types.
    317316
    318     steps : :obj:`string`
     317    steps : str
    319318        Specifies the forecast time step from forecast base time.
    320319        Valid values are hours (HH) from forecast base time.
     
    322321    Return
    323322    ------
    324     ftype : :obj:`list` of :obj:`string`
     323    ftype : list of str
    325324        List of field types.
    326325    '''
     
    340339    Parameters
    341340    ----------
    342     ftime : :obj:`list` of :obj:`string` or :obj:`string`
     341    ftime : list of str or str
    343342        The time in hours of the field.
    344343
    345344    Return
    346345    ------
    347     ftime : :obj:`list` of :obj:`string`
     346    ftime : list of str
    348347        The time in hours of the field.
    349348    '''
     
    358357    Parameters
    359358    ----------
    360     ftype : :obj:`list` of :obj:`string`
     359    ftype : list of str
    361360        List of field types.
    362361
    363     ftime : :obj:`list` of :obj:`string` or :obj:`string`
     362    ftime : list of str or str
    364363        The time in hours of the field.
    365364
    366     steps : :obj:`string`
     365    steps : str
    367366        Specifies the forecast time step from forecast base time.
    368367        Valid values are hours (HH) from forecast base time.
    369368
    370     maxstep : :obj:`integer`
     369    maxstep : int
    371370        The maximum forecast time step in hours from the forecast base time.
    372371        This is the maximum step for non flux (accumulated) forecast data.
    373372
    374     purefc : :obj:`integer`
     373    purefc : int
    375374        Switch for definition of pure forecast mode or not.
    376375
    377376    Return
    378377    ------
    379     ftype : :obj:`list` of :obj:`string`
     378    ftype : list of str
    380379        List of field types.
    381380
    382     ftime : :obj:`list` of :obj:`string`
     381    ftime : list of str
    383382        The time in hours of the field.
    384383
    385     steps : :obj:`string`
     384    steps : str
    386385        Specifies the forecast time step from forecast base time.
    387386        Valid values are hours (HH) from forecast base time.
     
    410409    Parameters
    411410    ----------
    412     mail : :obj:`list` of :obj:`string` or :obj:`string`
     411    mail : list of str or str
    413412        Contains email addresses for notifications.
    414413        It might also contain just the ecmwf user name which will trigger
     
    417416    Return
    418417    ------
    419     mail : :obj:`list` of :obj:``string`
     418    mail : list of str
    420419        Contains email addresses for notifications.
    421420        It might also contain just the ecmwf user name which will trigger
     
    439438    Parameters
    440439    ----------
    441     queue : :obj:`string`
     440    queue : str
    442441        Name of the queue if submitted to the ECMWF servers.
    443442        Used to check if ecuid, ecgid, gateway and destination
    444443        are set correctly and are not empty.
    445444
    446     gateway : :obj:`string`
     445    gateway : str
    447446        The address of the gateway server.
    448447
    449     destination : :obj:`string`
     448    destination : str
    450449        The name of the destination of the gateway server for data
    451450        transfer through ectrans. E.g. name@genericSftp
    452451
    453     ecuid : :obj:`string`
     452    ecuid : str
    454453        ECMWF user id.
    455454
    456     ecgid : :obj:`string`
     455    ecgid : str
    457456        ECMWF group id.
    458457
     
    475474    Parameters
    476475    ----------
    477     idir : :obj:`string`
     476    idir : str
    478477        Path to the temporary directory for MARS retrieval data.
    479478
    480     odir : :obj:`string`
     479    odir : str
    481480        Path to the final output directory where the FLEXPART input files
    482481        will be stored.
    483482
    484     fpdir : :obj:`string`
     483    fpdir : str
    485484        Path to FLEXPART root directory.
    486485
    487     fedir : :obj:`string`
     486    fedir : str
    488487        Path to flex_extract root directory.
    489488
    490489    Return
    491490    ------
    492     odir : :obj:`string`
     491    odir : str
    493492        Path to the final output directory where the FLEXPART input files
    494493        will be stored.
    495494
    496     fpdir : :obj:`string`
     495    fpdir : str
    497496        Path to FLEXPART root directory.
    498497
     
    513512    Parameters
    514513    ----------
    515     start : :obj:`string`
     514    start : str
    516515        The start date of the retrieval job.
    517516
    518     end : :obj:`string`
     517    end : str
    519518        The end date of the retrieval job.
    520519
    521520    Return
    522521    ------
    523     start : :obj:`string`
     522    start : str
    524523        The start date of the retrieval job.
    525524
    526     end : :obj:`string`
     525    end : str
    527526        The end date of the retrieval job.
    528527
     
    555554    Parameters
    556555    ----------
    557     maxstep : :obj:`string`
     556    maxstep : str
    558557        The maximum forecast time step in hours from the forecast base time.
    559558        This is the maximum step for non flux (accumulated) forecast data.
    560559
    561     steps : :obj:`string`
     560    steps : str
    562561        Specifies the forecast time step from forecast base time.
    563562        Valid values are hours (HH) from forecast base time.
     
    565564    Return
    566565    ------
    567     maxstep : :obj:`integer`
     566    maxstep : int
    568567        The maximum forecast time step in hours from the forecast base time.
    569568        This is the maximum step for non flux (accumulated) forecast data.
     
    588587    Parameters
    589588    ----------
    590     basetime : :obj:``
     589    basetime : str
    591590        The time for a half day retrieval. The 12 hours upfront are to be
    592591        retrieved.
     
    607606    Parameters
    608607    ----------
    609     request : :obj:`integer`
     608    request : int
    610609        Selects the mode of retrieval.
    611610        0: Retrieves the data from ECMWF.
     
    613612        2: Retrieves the data and prints the mars request.
    614613
    615     marsfile : :obj:`string`
     614    marsfile : str
    616615        Path to the mars request file.
    617616
     
    631630    Parameters
    632631    ----------
    633     public : :obj:`ìnteger`
     632    public : int
    634633        Specifies if public data are to be retrieved or not.
    635634
    636     dataset : :obj:`string`
     635    dataset : str
    637636        Specific name which identifies the public dataset.
    638637
     
    653652    Parameters
    654653    ----------
    655     acctype : :obj:`string`
     654    acctype : str
    656655        The field type for the accumulated forecast fields.
    657656
    658     ftype : :obj:`list` of :obj:`string`
     657    ftype : list of str
    659658        List of field types.
    660659
    661660    Return
    662661    ------
    663     acctype : :obj:`string`
     662    acctype : str
    664663        The field type for the accumulated forecast fields.
    665664    '''
     
    690689    Parameters
    691690    ----------
    692     acctime : :obj:`string`
     691    acctime : str
    693692        The starting time from the accumulated forecasts.
    694693
    695     acctype : :obj:`string`
     694    acctype : str
    696695        The field type for the accumulated forecast fields.
    697696
    698     purefc : :obj:`integer`
     697    purefc : int
    699698        Switch for definition of pure forecast mode or not.
    700699
    701700    Return
    702701    ------
    703     acctime : :obj:`string`
     702    acctime : str
    704703        The starting time from the accumulated forecasts.
    705704    '''
    706705    if not acctime:
    707706        print('... Control parameter ACCTIME was not defined.')
    708         print('... Value will be set depending on field type: '
     707        print('... Value will be set depending on field type:\n '
    709708              '\t\t EA=06/18\n\t\t EI/OD=00/12\n\t\t  EP=18')
    710709        if acctype.upper() == 'EA': # Era 5
     
    726725    Parameters
    727726    ----------
    728     accmaxstep : :obj:`string`
     727    accmaxstep : str
    729728        The maximum forecast step for the accumulated forecast fields.
    730729
    731     acctype : :obj:`string`
     730    acctype : str
    732731        The field type for the accumulated forecast fields.
    733732
    734     purefc : :obj:`integer`
     733    purefc : int
    735734        Switch for definition of pure forecast mode or not.
    736735
    737     maxstep : :obj:`string`
     736    maxstep : str
    738737        The maximum forecast time step in hours from the forecast base time.
    739738        This is the maximum step for non flux (accumulated) forecast data.
     
    741740    Return
    742741    ------
    743     accmaxstep : :obj:`string`
     742    accmaxstep : str
    744743        The maximum forecast step for the accumulated forecast fields.
    745744    '''
     
    776775    Parameters
    777776    ----------
    778     addpar : :obj:`string` or :obj:'list' of :obj:'string'
     777    addpar : str or list of str
    779778        List of additional parameters to be retrieved.
    780779
    781780    Return
    782781    ------
    783     addpar : :obj:'string'
     782    addpar : str
    784783        List of additional parameters to be retrieved.
    785784    '''
     
    798797
    799798def check_job_chunk(job_chunk):
    800     '''Checks that the job chunk number is positive and non zero.
    801 
    802     Parameters
    803     ----------
    804     job_chunk : :obj:`integer`
     799    '''Checks that if job chunk is set, the number is positive and non zero.
     800
     801    Parameters
     802    ----------
     803    job_chunk : int
    805804        The number of days for a single job script.
    806805
    807806    Return
    808807    ------
    809     job_chunk : :obj:`integer`
     808    job_chunk : int
    810809        The number of days for a single job script.
    811810    '''
     811    if not job_chunk:
     812        return job_chunk
     813
    812814    if job_chunk < 0:
    813815        raise ValueError('ERROR: The number of job chunk is negative!\n'
  • source/python/mods/disaggregation.py

    r0e2f93e r6f951ca  
    1717#        - outsourced the disaggregation functions dapoly and darain
    1818#          to a new module named disaggregation
     19#        - added the new disaggregation method for precipitation
    1920#
    2021# @License:
    21 #    (C) Copyright 2015-2018.
    22 #
    23 #    This software is licensed under the terms of the Apache Licence Version 2.0
    24 #    which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
    25 #
    26 # @Module Description:
    27 #    disaggregation of deaccumulated flux data from an ECMWF model FG field.
    28 #    Initially the flux data to be concerned are:
    29 #    - large-scale precipitation
    30 #    - convective precipitation
    31 #    - surface sensible heat flux
    32 #    - surface solar radiation
    33 #    - u stress
    34 #    - v stress
    35 #    Different versions of disaggregation is provided for rainfall
    36 #    data (darain, modified linear) and the surface fluxes and
    37 #    stress data (dapoly, cubic polynomial).
    38 #
    39 # @Module Content:
     22#    (C) Copyright 2014-2019.
     23#    Anne Philipp, Leopold Haimberger
     24#
     25#    This work is licensed under the Creative Commons Attribution 4.0
     26#    International License. To view a copy of this license, visit
     27#    http://creativecommons.org/licenses/by/4.0/ or send a letter to
     28#    Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.
     29#
     30# @Methods:
    4031#    - dapoly
    4132#    - darain
    4233#    - IA3
    43 #
    4434#*******************************************************************************
     35'''Disaggregation of deaccumulated flux data from an ECMWF model FG field.
     36
     37Initially the flux data to be concerned are:
     38    - large-scale precipitation
     39    - convective precipitation
     40    - surface sensible heat flux
     41    - surface solar radiation
     42    - u stress
     43    - v stress
     44
     45Different versions of disaggregation is provided for rainfall
     46data (darain, modified linear) and the surface fluxes and
     47stress data (dapoly, cubic polynomial).
     48'''
    4549
    4650# ------------------------------------------------------------------------------
     
    6569    Parameters
    6670    ----------
    67     alist : :obj:`list` of :obj:`array` of :obj:`float`
     71    alist : list of array of float
    6872        List of 4 timespans as 2-dimensional, horizontal fields.
    6973        E.g. [[array_t1], [array_t2], [array_t3], [array_t4]]
     
    7175    Return
    7276    ------
    73     nfield : :obj:`array` of :obj:`float`
     77    nfield : array of float
    7478        Interpolated flux at central point of accumulation timespan.
    7579
     
    109113    Parameters
    110114    ----------
    111     alist : :obj:`list` of :obj:`array` of :obj:`float`
     115    alist : list of array of float
    112116        List of 4 timespans as 2-dimensional, horizontal fields.
    113117        E.g. [[array_t1], [array_t2], [array_t3], [array_t4]]
     
    115119    Return
    116120    ------
    117     nfield : :obj:`array` of :obj:`float`
     121    nfield : array of float
    118122        Interpolated flux at central point of accumulation timespan.
    119123
     
    160164    Note
    161165    ----
    162     Copyright 2017
     166    (C) Copyright 2017-2019
    163167    Sabine Hittmeir, Anne Philipp, Petra Seibert
    164168
     
    170174    Parameters
    171175    ----------
    172     g : :obj:`list` of :obj:`float`
     176    g : list of float
    173177        Complete data series that will be interpolated having
    174178        the dimension of the original raw series.
     
    176180    Return
    177181    ------
    178     f : :obj:`list` of :obj:`float`
     182    f : list of float
    179183        The interpolated data series with additional subgrid points.
    180184        Its dimension is equal to the length of the input data series
  • source/python/mods/get_mars_data.py

    r9aefaad r6f951ca  
    2222#          online documentation)
    2323#        - use of UIFiles class for file selection and deletion
    24 #
     24#        - seperated get_mars_data function into several smaller pieces:
     25#          write_reqheader, mk_server, mk_dates, remove_old, do_retrievment
    2526#
    2627# @License:
    27 #    (C) Copyright 2014-2018.
    28 #
    29 #    This software is licensed under the terms of the Apache Licence Version 2.0
    30 #    which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
    31 #
    32 # @Program Functionality:
    33 #    This program can be used as a module in the whole flex_extract process
    34 #    or can be run by itself to just extract MARS data from ECMWF. To do so,
    35 #    a couple of necessary parameters has to be passed with the program call.
    36 #    See documentation for more details.
    37 #
    38 # @Program Content:
    39 #    - main
    40 #    - get_mars_data
    41 #    - do_retrievement
    42 #
     28#    (C) Copyright 2014-2019.
     29#    Anne Philipp, Leopold Haimberger
     30#
     31#    This work is licensed under the Creative Commons Attribution 4.0
     32#    International License. To view a copy of this license, visit
     33#    http://creativecommons.org/licenses/by/4.0/ or send a letter to
     34#    Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.
    4335#*******************************************************************************
    44 """ToDo: Name of litte program
    45 
    46 ToDo: Add desccription
    47 
    48 ToDo: Add Conditions
    49 
    50 This script requires that `...` be installed within the Python
    51 environment you are running this script in.
     36'''This script extracts MARS data from ECMWF servers.
     37
     38At first, the necessary parameters from command line and CONTROL files are
     39extracted. They define the data set to be extracted from MARS.
    5240
    5341This file can also be imported as a module and contains the following
    5442functions:
    5543
    56     * get_mars_data -
    57     * do_retrievement -
    5844    * main - the main function of the script
    59 """
     45    * get_mars_data - overall control of ECMWF data retrievment
     46    * write_reqheader - writes the header into the mars_request file
     47    * mk_server - creates the server connection to ECMWF servers
     48    * mk_dates - defines the start and end date
     49    * remove_old - deletes old retrieved grib files
     50    * do_retrievement - creates individual retrievals
     51
     52Type: get_mars_data.py --help
     53to get information about command line parameters.
     54Read the documentation for usage instructions.
     55'''
    6056# ------------------------------------------------------------------------------
    6157# MODULES
     
    6763
    6864# software specific classes and modules from flex_extract
     65# add path to local main python path for flex_extract to get full access
    6966sys.path.append(os.path.dirname(os.path.abspath(
    7067    inspect.getfile(inspect.currentframe()))) + '/../')
     
    120117    Parameters
    121118    ----------
    122     c : :obj:`ControlFile`
     119    c : ControlFile
    123120        Contains all the parameters of CONTROL file and
    124121        command line.
     
    166163    Parameters
    167164    ----------
    168     marsfile : :obj:`string`
     165    marsfile : str
    169166        Path to the mars request file.
    170167
     
    189186    Parameters
    190187    ----------
    191     c : :obj:`ControlFile`
     188    c : ControlFile
    192189        Contains all the parameters of CONTROL file and
    193190        command line.
     
    195192    Return
    196193    ------
    197     server : :obj:`ECMWFDataServer` or :obj:`ECMWFService`
     194    server : ECMWFDataServer or ECMWFService
    198195        Connection to ECMWF server via python interface ECMWF WebAPI.
    199196
     
    230227    Parameters
    231228    ----------
    232     c : :obj:`ControlFile`
     229    c : ControlFile
    233230        Contains all the parameters of CONTROL file and
    234231        command line.
    235232
    236     fluxes : :obj:`boolean`, optional
     233    fluxes : boolean, optional
    237234        Decides if the flux parameter settings are stored or
    238235        the rest of the parameter list.
     
    241238    Return
    242239    ------
    243     start : :obj:`datetime`
     240    start : datetime
    244241        The start date of the retrieving data set.
    245242
    246     end : :obj:`datetime`
     243    end : datetime
    247244        The end date of the retrieving data set.
    248245
    249     chunk : :obj:`datetime`
     246    chunk : datetime
    250247        Time period in days for one single mars retrieval.
    251248
     
    271268    Parameters
    272269    ----------
    273     pattern : :obj:`string`
     270    pattern : str
    274271        The sub string pattern which identifies the files to be deleted.
    275272
    276     inputdir : :obj:`string`, optional
     273    inputdir : str, optional
    277274        Path to the directory where the retrieved data is stored.
    278275
     
    295292    Parameters
    296293    ----------
    297     c : :obj:`ControlFile`
     294    c : ControlFile
    298295        Contains all the parameters of CONTROL file and
    299296        command line.
    300297
    301     server : :obj:`ECMWFService`
     298    server : ECMWFService or ECMWFDataServer
    302299            The server connection to ECMWF.
    303300
    304     start : :obj:`datetime`
     301    start : datetime
    305302        The start date of the retrieval.
    306303
    307     end : :obj:`datetime`
     304    end : datetime
    308305        The end date of the retrieval.
    309306
    310     delta_t : :obj:`datetime`
     307    delta_t : datetime
    311308        Delta_t + 1 is the maximal time period of a single
    312309        retrieval.
    313310
    314     fluxes : :obj:`boolean`, optional
     311    fluxes : boolean, optional
    315312        Decides if the flux parameters are to be retrieved or
    316313        the rest of the parameter list.
  • source/python/mods/prepare_flexpart.py

    r9aefaad r6f951ca  
    2222#        - added documentation
    2323#        - minor changes in programming style for consistence
    24 #        - BUG: removed call of clean_up-Function after call of
     24#        - BUGFIX: removed call of clean_up-Function after call of
    2525#               prepareFlexpart in main since it is already called in
    2626#               prepareFlexpart at the end!
     
    2929#
    3030# @License:
    31 #    (C) Copyright 2014-2018.
     31#    (C) Copyright 2014-2019.
     32#    Anne Philipp, Leopold Haimberger
    3233#
    33 #    This software is licensed under the terms of the Apache Licence Version 2.0
    34 #    which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
    35 #
    36 # @Program Functionality:
    37 #    This program prepares the final version of the grib files which are
    38 #    then used by FLEXPART. It converts the bunch of grib files extracted
    39 #    via get_mars_data by doing for example the necessary conversion to get
    40 #    consistent grids or the disaggregation of flux data. Finally, the
    41 #    program combines the data fields in files per available hour with the
    42 #    naming convention xxYYMMDDHH, where xx should be 2 arbitrary letters
    43 #    (mostly xx is chosen to be "EN").
    44 #
    45 # @Program Content:
    46 #    - main
    47 #    - prepare_flexpart
    48 #
     34#    This work is licensed under the Creative Commons Attribution 4.0
     35#    International License. To view a copy of this license, visit
     36#    http://creativecommons.org/licenses/by/4.0/ or send a letter to
     37#    Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.
    4938#*******************************************************************************
     39'''This script prepares the final version of the grib files which are
     40then used by FLEXPART.
     41
     42It converts the bunch of grib files extracted via get_mars_data before,
     43by doing the necessary conversion to get consistent grids or the
     44disaggregation of flux data. Finally, the data fields are combined
     45in files per available hour with the naming convention xxYYMMDDHH,
     46where xx should be 2 arbitrary letters (mostly xx is chosen to be "EN").
     47
     48This file can also be imported as a module which then contains the following
     49functions:
     50
     51    * main
     52    * prepare_flexpart
     53
     54Type: prepare_flexpart.py --help
     55to get information about command line parameters.
     56Read the documentation for usage instructions.
     57'''
    5058
    5159# ------------------------------------------------------------------------------
     
    5967
    6068# software specific classes and modules from flex_extract
    61 
     69# add path to local main python path for flex_extract to get full access
    6270sys.path.append(os.path.dirname(os.path.abspath(
    6371    inspect.getfile(inspect.currentframe()))) + '/../')
     
    115123    Parameters
    116124    ----------
    117     ppid : :obj:`int`
     125    ppid : int
    118126        Contains the ppid number of the current ECMWF job. It will be None if
    119127        the method was called within this module.
    120128
    121     c : :obj:`ControlFile`
     129    c : ControlFile
    122130        Contains all the parameters of CONTROL file and
    123131        command line.
  • source/python/mods/tools.py

    rf2616a3 r6f951ca  
    1414#          my_error, normal_exit, init128, to_param_id
    1515#
    16 #    April 2018 - Anne Philipp (University of Vienna):
     16#    April - December 2018 - Anne Philipp (University of Vienna):
    1717#        - applied PEP8 style guide
    1818#        - added documentation
    19 #        - moved all functions from file Flexparttools to this file tools
    20 #        - added function get_list_as_string
     19#        - moved all non class methods from former file Flexparttools in here
    2120#        - seperated args and control interpretation
     21#        - added functions get_list_as_string, read_ecenv, send_mail, make_dir,
     22#          put_file_to_ecserver, submit_job_to_ecserver, get_informations,
     23#          get_dimensions, execute_subprocess, none_or_int, none_or_str
    2224#
    2325# @License:
    24 #    (C) Copyright 2014-2018.
     26#    (C) Copyright 2014-2019.
     27#    Anne Philipp, Leopold Haimberger
    2528#
    26 #    This software is licensed under the terms of the Apache Licence Version 2.0
    27 #    which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
     29#    This work is licensed under the Creative Commons Attribution 4.0
     30#    International License. To view a copy of this license, visit
     31#    http://creativecommons.org/licenses/by/4.0/ or send a letter to
     32#    Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.
    2833#
    29 # @Modul Description:
    30 #    This module contains a couple of helpful functions which are
    31 #    used in different places in flex_extract.
    32 #
    33 # @Module Content:
    34 #    - get_cmdline_args
    35 #    - clean_up
    36 #    - my_error
    37 #    - normal_exit
    38 #    - product
    39 #    - silent_remove
    40 #    - init128
    41 #    - to_param_id
    42 #    - get_list_as_string
    43 #    - make_dir
    44 #
     34# @Methods:
     35#    none_or_str
     36#    none_or_int
     37#    get_cmdline_args
     38#    read_ecenv
     39#    clean_up
     40#    my_error
     41#    send_mail
     42#    normal_exit
     43#    product
     44#    silent_remove
     45#    init128
     46#    to_param_id
     47#    get_list_as_string
     48#    make_dir
     49#    put_file_to_ecserver
     50#    submit_job_to_ecserver
     51#    get_informations
     52#    get_dimensions
     53#    execute_subprocess
    4554#*******************************************************************************
     55'''This module contains a collection of diverse tasks within flex_extract.
     56'''
    4657
    4758# ------------------------------------------------------------------------------
     
    5970
    6071# ------------------------------------------------------------------------------
    61 # FUNCTIONS
     72# METHODS
    6273# ------------------------------------------------------------------------------
    6374
     
    6879    Parameters
    6980    ----------
    70     value : :obj:`string`
     81    value : str
    7182        String to be checked for the "None" word.
    7283
     
    8798    Parameters
    8899    ----------
    89     value : :obj:`string`
     100    value : str
    90101        String to be checked for the "None" word.
    91102
     
    110121    Return
    111122    ------
    112     args : :obj:`Namespace`
     123    args : Namespace
    113124        Contains the commandline arguments from script/program call.
    114125    '''
     
    201212    Parameters
    202213    ----------
    203     filepath : :obj:`string`
     214    filepath : str
    204215        Path to file where the ECMWF environment parameters are stored.
    205216
    206217    Return
    207218    ------
    208     envs : :obj:`dictionary`
     219    envs : dict
    209220        Contains the environment parameter ecuid, ecgid, gateway
    210221        and destination for ECMWF server environments.
     
    233244    Parameters
    234245    ----------
    235     c : :obj:`ControlFile`
     246    c : ControlFile
    236247        Contains all the parameters of CONTROL file and
    237248        command line.
     
    265276    Parameters
    266277    ----------
    267     user : :obj:`list` of :obj:`string`
     278    user : list of str
    268279        Contains all email addresses which should be notified.
    269280        It might also contain just the ecmwf user name which wil trigger
    270281        mailing to the associated email address for this user.
    271282
    272     message : :obj:`string`, optional
     283    message : str, optional
    273284        Error message. Default value is "ERROR".
    274285
     
    295306    Parameters
    296307    ----------
    297     users : :obj:`list` of :obj:`string`
     308    users : list of str
    298309        Contains all email addresses which should be notified.
    299310        It might also contain just the ecmwf user name which wil trigger
    300311        mailing to the associated email address for this user.
    301312
    302     success_mode : :obj:``string`
     313    success_mode : str
    303314        States the exit mode of the program to put into
    304315        the mail subject line.
    305316
    306     message : :obj:`string`, optional
     317    message : str, optional
    307318        Message for exiting program. Default value is "Done!".
    308319
     
    341352    Parameters
    342353    ----------
    343     message : :obj:`string`, optional
     354    message : str, optional
    344355        Message for exiting program. Default value is "Done!".
    345356
     
    374385    Parameters
    375386    ----------
    376     \*args : :obj:`list` or :obj:`string`
     387    \*args : list or str
    377388        Positional arguments (arbitrary number).
    378389
    379     \*\*kwds : :obj:`dictionary`
     390    \*\*kwds : dict
    380391        Contains all the keyword arguments from \*args.
    381392
     
    405416    Parameters
    406417    ----------
    407     filename : :obj:`string`
     418    filename : str
    408419        The name of the file to be removed without notification.
    409420
     
    429440    Parameters
    430441    ----------
    431     filepath : :obj:`string`
     442    filepath : str
    432443        Path to file of ECMWF grib table number 128.
    433444
    434445    Return
    435446    ------
    436     table128 : :obj:`dictionary`
     447    table128 : dict
    437448        Contains the ECMWF grib table 128 information.
    438449        The key is the parameter number and the value is the
     
    462473    Parameters
    463474    ----------
    464     pars : :obj:`string`
     475    pars : str
    465476        Addpar argument from CONTROL file in the format of
    466477        parameter names instead of ids. The parameter short
     
    468479        one single string.
    469480
    470     table : :obj:`dictionary`
     481    table : dict
    471482        Contains the ECMWF grib table 128 information.
    472483        The key is the parameter number and the value is the
     
    475486    Return
    476487    ------
    477     ipar : :obj:`list` of :obj:`integer`
     488    ipar : list of int
    478489        List of addpar parameters from CONTROL file transformed to
    479490        parameter ids in the format of integer.
     
    501512    Parameters
    502513    ----------
    503     list_obj : :obj:`list`
     514    list_obj : list of *
    504515        A list with arbitrary content.
    505516
    506     concatenate_sign : :obj:`string`, optional
     517    concatenate_sign : str, optional
    507518        A string which is used to concatenate the single
    508519        list elements. Default value is ", ".
     
    510521    Return
    511522    ------
    512     str_of_list : :obj:`string`
     523    str_of_list : str
    513524        The content of the list as a single string.
    514525    '''
     
    528539    Parameters
    529540    ----------
    530     directory : :obj:`string`
     541    directory : str
    531542        The path to directory which should be created.
    532543
     
    557568    Parameters
    558569    ----------
    559     ecd : :obj:`string`
     570    ecd : str
    560571        The path were the file is stored.
    561572
    562     filename : :obj:`string`
     573    filename : str
    563574        The name of the file to send to the ECMWF server.
    564575
    565     target : :obj:`string`
     576    target : str
    566577        The target queue where the file should be sent to.
    567578
    568     ecuid : :obj:`string`
     579    ecuid : str
    569580        The user id on ECMWF server.
    570581
    571     ecgid : :obj:`string`
     582    ecgid : str
    572583        The group id on ECMWF server.
    573584
     
    610621    Parameters
    611622    ----------
    612     target : :obj:`string`
     623    target : str
    613624        The target where the file should be sent to, e.g. the queue.
    614625
    615     jobname : :obj:`string`
     626    jobname : str
    616627        The name of the jobfile to be submitted to the ECMWF server.
    617628
    618629    Return
    619630    ------
    620     job_id : :obj:`int`
     631    job_id : int
    621632        The id number of the job as a reference at the ecmwf server.
    622633    '''
     
    650661    Parameters
    651662    ----------
    652     filename : :obj:`string`
     663    filename : str
    653664            Name of the file which will be opened to extract basic information.
    654665
    655666    Return
    656667    ------
    657     data : :obj:`dictionary`
     668    data : dict
    658669        Contains basic informations of the ECMWF grib files, e.g.
    659670        'Ni', 'Nj', 'latitudeOfFirstGridPointInDegrees',
     
    702713    Parameters
    703714    ----------
    704     info : :obj:`dictionary`
     715    info : dict
    705716        Contains basic informations of the ECMWF grib files, e.g.
    706717        'Ni', 'Nj', 'latitudeOfFirstGridPointInDegrees',
     
    709720        'iDirectionIncrementInDegrees', 'missingValue'
    710721
    711     purefc : :obj:`integer`
     722    purefc : int
    712723        Switch for definition of pure forecast mode or not.
    713724
    714     dtime : :obj:`string`
     725    dtime : str
    715726        Time step in hours.
    716727
    717     index_vals : :obj:`list`
     728    index_vals : list of list of str
    718729        Contains the values from the keys used for a distinct selection
    719730        of grib messages in processing  the grib files.
     
    723734        index_vals[2]: ('0', '12', '3', '6', '9') ; stepRange
    724735
    725     start_date : :obj:`string`
     736    start_date : str
    726737        The start date of the retrieval job.
    727738
    728     end_date : :obj:`string`
     739    end_date : str
    729740        The end date of the retrieval job.
    730741
    731742    Return
    732743    ------
    733     (ix, jy, it) : :obj:`tuple` of :obj:`integer`
     744    (ix, jy, it) : tuple of int
    734745        Dimension in x-direction, y-direction and in time.
    735746    '''
     
    755766    Parameters
    756767    ----------
    757     cmd_list : :obj:`list` of `:obj:`string`
     768    cmd_list : list of str
    758769        A list of the components for the command line execution. Each
    759770        list entry is a single part of the command which is seperated from
     
    763774    Return
    764775    ------
    765     error_msg : :obj:`string`, optional
     776    error_msg : str, optional
    766777        The possible error message if the subprocess failed.
    767778        By default it will just tell "SUBPROCESS FAILED!".
  • source/python/submit.py

    r0629ba8 r6f951ca  
    1818#        - changed path names to variables from config file
    1919#        - added option for writing mars requests to extra file
    20 #          additionally,as option without submitting the mars jobs
     20#          additionally, as option without submitting the mars jobs
     21#        - splitted submit function to use genshi templates for the
     22#          job script and avoid code duplication
    2123#
    2224# @License:
    23 #    (C) Copyright 2014-2018.
    24 #
    25 #    This software is licensed under the terms of the Apache Licence Version 2.0
    26 #    which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
    27 #
    28 # @Program Functionality:
    29 #    This program is the main program of flex_extract and controls the
    30 #    program flow.
    31 #    If it is supposed to work locally then it works through the necessary
    32 #    functions get_mars_data and prepareFlexpart. Otherwise it prepares
    33 #    a shell job script which will do the necessary work on the
    34 #    ECMWF server and is submitted via ecaccess-job-submit.
    35 #
    36 # @Program Content:
    37 #    - main
    38 #    - submit
    39 #
     25#    (C) Copyright 2014-2019.
     26#    Anne Philipp, Leopold Haimberger
     27#
     28#    This work is licensed under the Creative Commons Attribution 4.0
     29#    International License. To view a copy of this license, visit
     30#    http://creativecommons.org/licenses/by/4.0/ or send a letter to
     31#    Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.
    4032#*******************************************************************************
     33'''This script allows the user to extract meteorological fields from the ECMWF.
     34
     35It prepares the settings for retrieving the data from ECMWF servers and
     36process the resulting files to prepare them for the use with FLEXPART or
     37FLEXTRA.
     38
     39If it is supposed to work locally then it works through the necessary
     40functions get_mars_data and prepare_flexpart. Otherwise it prepares
     41a job script (korn shell) which will do the necessary work on the
     42ECMWF server. This script will de submitted via the ecaccess command
     43ecaccess-job-submit.
     44
     45This file can also be imported as a module which then contains the following
     46functions:
     47
     48    * main - the main function of the script
     49    * submit - calls mk_jobscript depending on operation mode and submits its
     50    * mk_jobscript - creates the job script from a template
     51
     52Type: submit.py --help
     53to get information about command line parameters.
     54Read the documentation for usage instructions.
     55'''
    4156
    4257# ------------------------------------------------------------------------------
     
    5974
    6075# ------------------------------------------------------------------------------
    61 # FUNCTIONS
     76# METHODS
    6277# ------------------------------------------------------------------------------
    6378
     
    112127    Parameters
    113128    ----------
    114     jtemplate : :obj:`string`
     129    jtemplate : str
    115130        Job template file from sub-directory "_templates" for
    116131        submission to ECMWF. It contains all necessary
     
    119134        Default is "job.temp".
    120135
    121     c : :obj:`ControlFile`
     136    c : ControlFile
    122137        Contains all the parameters of CONTROL file and
    123138        command line.
    124139
    125     queue : :obj:`string`
     140    queue : str
    126141        Name of queue for submission to ECMWF (e.g. ecgate or cca )
    127142
     
    201216    Parameters
    202217    ----------
    203     jtemplate : :obj:`string`
     218    jtemplate : str
    204219        Job template file from sub-directory "_templates" for
    205220        submission to ECMWF. It contains all necessary
     
    208223        Default is "job.temp".
    209224
    210     job_file : :obj:`string`
     225    job_file : str
    211226        Path to the job script file.
    212227
    213     clist : :obj:`list` of :obj:`string`
     228    clist : list of str
    214229        Contains all necessary parameters for ECMWF CONTROL file.
    215230
Note: See TracChangeset for help on using the changeset viewer.
hosted by ZAMG