Changeset 991df6a in flex_extract.git for python/getMARSdata.py


Ignore:
Timestamp:
May 14, 2018, 10:11:29 PM (6 years ago)
Author:
Anne Philipp <anne.philipp@…>
Branches:
master, ctbto, dev
Children:
812283d
Parents:
efdb01a
Message:

finished documentation (except plot_retrieved)

File:
1 edited

Legend:

Unmodified
Added
Removed

  • python/getMARSdata.py

    refdb01a r991df6a  
    33#************************************************************************
    44# TODO AP
    5 # - Change History ist nicht angepasst ans File!
    6 # - add file description
     5# - add function docstrings!!!!
    76#************************************************************************
    8 """
    9 @Author: Anne Fouilloux (University of Oslo)
    10 
    11 @Date: October 2014
    12 
    13 @ChangeHistory:
    14     November 2015 - Leopold Haimberger (University of Vienna):
    15         - using the WebAPI also for general MARS retrievals
    16         - job submission on ecgate and cca
    17         - job templates suitable for twice daily operational dissemination
    18         - dividing retrievals of longer periods into digestable chunks
    19         - retrieve also longer term forecasts, not only analyses and
    20           short term forecast data
    21         - conversion into GRIB2
    22         - conversion into .fp format for faster execution of FLEXPART
    23 
    24     February 2018 - Anne Philipp (University of Vienna):
    25         - applied PEP8 style guide
    26         - added documentation
    27         - minor changes in programming style for consistence
    28 
    29 @License:
    30     (C) Copyright 2014-2018.
    31 
    32     This software is licensed under the terms of the Apache Licence Version 2.0
    33     which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
    34 
    35 @Requirements:
    36     - A standard python 2.6 or 2.7 installation
    37     - dateutils
    38     - ECMWF specific packages, all available from https://software.ecmwf.int/
    39         ECMWF WebMARS, gribAPI with python enabled, emoslib and
    40         ecaccess web toolkit
    41 
    42 @Description:
    43     Further documentation may be obtained from www.flexpart.eu.
    44 
    45 """
     7#*******************************************************************************
     8# @Author: Anne Fouilloux (University of Oslo)
     9#
     10# @Date: October 2014
     11#
     12# @Change History:
     13#
     14#    November 2015 - Leopold Haimberger (University of Vienna):
     15#        - moved the getEIdata program into a function "getMARSdata"
     16#        - moved the AgurmentParser into a seperate function
     17#        - adatpted the function for the use in flex_extract
     18#        - renamed file to getMARSdata
     19#
     20#    February 2018 - Anne Philipp (University of Vienna):
     21#        - applied PEP8 style guide
     22#        - added structured documentation
     23#        - minor changes in programming style for consistence
     24#        - added function main and moved function calls vom __main__ there
     25#          (necessary for better documentation with docstrings for later
     26#          online documentation)
     27#        - use of UIFiles class for file selection and deletion
     28
     29#
     30# @License:
     31#    (C) Copyright 2014-2018.
     32#
     33#    This software is licensed under the terms of the Apache Licence Version 2.0
     34#    which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
     35#
     36# @Program Functionality:
     37#    This program can be used as a module in the whole flex_extract process
     38#    or can be run by itself to just extract MARS data from ECMWF. To do so,
     39#    a couple of necessary parameters has to be passed with the program call.
     40#    See documentation for more details.
     41#
     42# @Program Content:
     43#    - main
     44#    - getMARSdata
     45#
     46#*******************************************************************************
     47
    4648# ------------------------------------------------------------------------------
    4749# MODULES
    4850# ------------------------------------------------------------------------------
     51import os
     52import sys
     53import datetime
     54import inspect
    4955try:
    5056    ecapi=True
    … …  
    5359    ecapi=False
    5460
    55 import calendar
    56 import shutil
    57 import datetime
    58 import time
    59 import os
    60 import glob
    61 import sys
    62 import inspect
    63 # add path to submit.py to pythonpath so that python finds its buddies
    64 localpythonpath=os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
     61# add path to pythonpath so that python finds its buddies
     62localpythonpath = os.path.dirname(os.path.abspath(
     63    inspect.getfile(inspect.currentframe())))
    6564if localpythonpath not in sys.path:
    6665    sys.path.append(localpythonpath)
    6766
    68 from Control import Control
     67# software specific classes and modules from flex_extract
     68from ControlFile import ControlFile
    6969from Tools import myerror, normalexit, \
    7070                          interpret_args_and_control
    7171from ECFlexpart import ECFlexpart
     72from UIOFiles import UIOFiles
    7273
    7374# ------------------------------------------------------------------------------
    7475# FUNCTION
    7576# ------------------------------------------------------------------------------
     77def main():
     78    '''
     79    @Description:
     80        If getMARSdata is called from command line, this function controls
     81        the program flow and calls the argumentparser function and
     82        the getMARSdata function for retrieving EC data.
     83
     84    @Input:
     85        <nothing>
     86
     87    @Return:
     88        <nothing>
     89    '''
     90    args, c = interpret_args_and_control()
     91    getMARSdata(args, c)
     92    normalexit(c)
     93
     94    return
     95
    7696def getMARSdata(args, c):
    77 
     97    '''
     98    @Description:
     99        Retrieves the EC data needed for a FLEXPART simulation.
     100        Start and end dates for retrieval period is set. Retrievals
     101        are divided into smaller periods if necessary and datechunk parameter
     102        is set.
     103
     104    @Input:
     105        args: instance of ArgumentParser
     106            Contains the commandline arguments from script/program call.
     107
     108        c: instance of class ControlFile
     109            Contains all the parameters of CONTROL file, which are e.g.:
     110            DAY1(start_date), DAY2(end_date), DTIME, MAXSTEP, TYPE, TIME,
     111            STEP, CLASS(marsclass), STREAM, NUMBER, EXPVER, GRID, LEFT,
     112            LOWER, UPPER, RIGHT, LEVEL, LEVELIST, RESOL, GAUSS, ACCURACY,
     113            OMEGA, OMEGADIFF, ETA, ETADIFF, DPDETA, SMOOTH, FORMAT,
     114            ADDPAR, WRF, CWC, PREFIX, ECSTORAGE, ECTRANS, ECFSDIR,
     115            MAILOPS, MAILFAIL, GRIB2FLEXPART, FLEXPARTDIR, BASETIME
     116            DATE_CHUNK, DEBUG, INPUTDIR, OUTPUTDIR, FLEXPART_ROOT_SCRIPTS
     117
     118            For more information about format and content of the parameter
     119            see documentation.
     120
     121    @Return:
     122        <nothing>
     123    '''
    78124
    79125    if not os.path.exists(c.inputdir):
    80126        os.makedirs(c.inputdir)
     127
     128    print("Retrieving EC data!")
    81129    print("start date %s " % (c.start_date))
    82130    print("end date %s " % (c.end_date))
    … …  
    88136
    89137    c.ecapi = ecapi
    90     print 'ecapi:', c.ecapi
    91 
    92 # Retrieve EC data for running flexpart
    93 #AP change this variant to correct format conversion with datetime
    94 #AP import datetime and timedelta explicitly
    95     syear = int(c.start_date[:4])
    96     smonth = int(c.start_date[4:6])
    97     sday = int(c.start_date[6:])
    98     start = datetime.date(year=syear, month=smonth, day=sday)
     138    print 'ecapi: ', c.ecapi
     139
     140    # set start date of retrieval period
     141    start = datetime.date(year=int(c.start_date[:4]),
     142                          month=int(c.start_date[4:6]),
     143                          day=int(c.start_date[6:]))
    99144    startm1 = start - datetime.timedelta(days=1)
    100145    if c.basetime == '00':
    101146        start = startm1
    102147
    103     eyear = int(c.end_date[:4])
    104     emonth = int(c.end_date[4:6])
    105     eday = int(c.end_date[6:])
    106     end = datetime.date(year=eyear, month=emonth, day=eday)
     148    # set end date of retrieval period
     149    end = datetime.date(year=int(c.end_date[:4]),
     150                        month=int(c.end_date[4:6]),
     151                        day=int(c.end_date[6:]))
    107152    if c.basetime == '00' or c.basetime == '12':
    108153        endp1 = end + datetime.timedelta(days=1)
    … …  
    110155        endp1 = end + datetime.timedelta(days=2)
    111156
     157    # set time period of one single retrieval
    112158    datechunk = datetime.timedelta(days=int(c.date_chunk))
    113159
    114     # retrieving of accumulated data fields (flux data), (maximum one month)
    115 
    116     # remove old files
    117     print 'removing content of ' + c.inputdir
    118     tobecleaned = glob.glob(c.inputdir + '/*_acc_*.' + \
    119                             str(os.getppid()) + '.*.grb')
    120     for f in tobecleaned:
    121         os.remove(f)
    122 
    123     times = None
     160    # --------------  flux data ------------------------------------------------
     161    print 'removing old flux content of ' + c.inputdir
     162    tobecleaned = UIOFiles('*_acc_*.' + str(os.getppid()) + '.*.grb')
     163    tobecleaned.listFiles(c.inputdir)
     164    tobecleaned.deleteFiles()
     165
    124166    # if forecast for maximum one day (upto 23h) are to be retrieved,
    125167    # collect accumulation data (flux data)
    … …  
    129171        day = startm1
    130172        while day < endp1:
    131                 # retrieve MARS data for the whole period
    132                 flexpart = ECFlexpart(c, fluxes=True)
    133                 tmpday = day + datechunk - datetime.timedelta(days=1)
    134                 if tmpday < endp1:
    135                     dates = day.strftime("%Y%m%d") + "/to/" + \
    136                             tmpday.strftime("%Y%m%d")
    137                 else:
    138                     dates = day.strftime("%Y%m%d") + "/to/" + \
    139                             end.strftime("%Y%m%d")
    140 
    141                 print "retrieve " + dates + " in dir " + c.inputdir
    142 
    143                 try:
    144                     flexpart.retrieve(server, dates, times, c.inputdir)
    145                 except IOError:
    146                     myerror(c,'MARS request failed')
    147 
    148                 day += datechunk
     173            # retrieve MARS data for the whole period
     174            flexpart = ECFlexpart(c, fluxes=True)
     175            tmpday = day + datechunk - datetime.timedelta(days=1)
     176            if tmpday < endp1:
     177                dates = day.strftime("%Y%m%d") + "/to/" + \
     178                        tmpday.strftime("%Y%m%d")
     179            else:
     180                dates = day.strftime("%Y%m%d") + "/to/" + \
     181                        end.strftime("%Y%m%d")
     182
     183            print "retrieve " + dates + " in dir " + c.inputdir
     184
     185            try:
     186                flexpart.retrieve(server, dates, c.inputdir)
     187            except IOError:
     188                myerror(c, 'MARS request failed')
     189
     190            day += datechunk
    149191
    150192    # if forecast data longer than 24h are to be retrieved,
    … …  
    156198        day = start
    157199        while day <= end:
    158                 # retrieve MARS data for the whole period
    159                 flexpart = ECFlexpart(c, fluxes=True)
    160                 tmpday = day + datechunk - datetime.timedelta(days=1)
    161                 if tmpday < end:
    162                     dates = day.strftime("%Y%m%d") + "/to/" + \
    163                             tmpday.trftime("%Y%m%d")
    164                 else:
    165                     dates = day.strftime("%Y%m%d") + "/to/" + \
    166                             end.strftime("%Y%m%d")
    167 
    168                 print "retrieve " + dates + " in dir " + c.inputdir
    169 
    170                 try:
    171                     flexpart.retrieve(server, dates, times, c.inputdir)
    172                 except IOError:
    173                     myerror(c, 'MARS request failed')
    174 
    175                 day += datechunk
    176 
    177     # retrieving of normal data fields (non flux data), (maximum one month)
    178 
    179     # remove old *__* files
    180     tobecleaned = glob.glob(c.inputdir + '/*__*.' +
    181                             str(os.getppid()) + '.*.grb')
    182     for f in tobecleaned:
    183         os.remove(f)
     200            # retrieve MARS data for the whole period
     201            flexpart = ECFlexpart(c, fluxes=True)
     202            tmpday = day + datechunk - datetime.timedelta(days=1)
     203            if tmpday < end:
     204                dates = day.strftime("%Y%m%d") + "/to/" + \
     205                        tmpday.trftime("%Y%m%d")
     206            else:
     207                dates = day.strftime("%Y%m%d") + "/to/" + \
     208                        end.strftime("%Y%m%d")
     209
     210            print "retrieve " + dates + " in dir " + c.inputdir
     211
     212            try:
     213                flexpart.retrieve(server, dates, c.inputdir)
     214            except IOError:
     215                myerror(c, 'MARS request failed')
     216
     217            day += datechunk
     218
     219    # --------------  non flux data --------------------------------------------
     220    print 'removing old non flux content of ' + c.inputdir
     221    tobecleaned = UIOFiles('*__*.' + str(os.getppid()) + '.*.grb')
     222    tobecleaned.listFiles(c.inputdir)
     223    tobecleaned.deleteFiles()
     224
    184225    day = start
    185     times = None
    186226    while day <= end:
    187             # retrieve MARS data for the whole period
     227            # retrieve all non flux MARS data for the whole period
    188228            flexpart = ECFlexpart(c, fluxes=False)
    189             tmpday = day+datechunk-datetime.timedelta(days=1)
     229            tmpday = day + datechunk - datetime.timedelta(days=1)
    190230            if tmpday < end:
    191231                dates = day.strftime("%Y%m%d") + "/to/" + \
    … …  
    198238
    199239            try:
    200                 flexpart.retrieve(server, dates, times, c.inputdir)
     240                flexpart.retrieve(server, dates, c.inputdir)
    201241            except IOError:
    202242                myerror(c, 'MARS request failed')
    … …  
    207247
    208248if __name__ == "__main__":
    209 
    210     args, c = interpret_args_and_control()
    211     getMARSdata(args, c)
    212     normalexit(c)
     249    main()
     250
Note: See TracChangeset for help on using the changeset viewer.
hosted by ZAMG