Changeset 991df6a in flex_extract.git


Ignore:
Timestamp:
May 14, 2018, 10:11:29 PM (6 years ago)
Author:
Anne Philipp <anne.philipp@…>
Branches:
master, ctbto, dev
Children:
812283d
Parents:
efdb01a
Message:

finished documentation (except plot_retrieved)

Files:
2 deleted
19 edited

Legend:

Unmodified
Added
Removed

  • python/Disagg.py

    refdb01a r991df6a  
    33#************************************************************************
    44# TODO AP
    5 # - make a class out of this ???
    6 # - write a test class
     5# - check alist of size 4 ?
     6# - write a test, IMPORTANT
    77#************************************************************************
    8 """
    9 @Author: Anne Philipp (University of Vienna)
     8#*******************************************************************************
     9# @Author: Anne Philipp (University of Vienna)
     10#
     11# @Date: March 2018
     12#
     13# @Change History:
     14#    November 2015 - Leopold Haimberger (University of Vienna):
     15#        - migration of the methods dapoly and darain from Fortran
     16#          (flex_extract_v6 and earlier) to Python
     17#
     18#    April 2018 - Anne Philipp (University of Vienna):
     19#        - applied PEP8 style guide
     20#        - added structured documentation
     21#        - outsourced the disaggregation functions dapoly and darain
     22#          to a new module named Disagg
     23#
     24# @License:
     25#    (C) Copyright 2015-2018.
     26#
     27#    This software is licensed under the terms of the Apache Licence Version 2.0
     28#    which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
     29#
     30# @Module Description:
     31#    Disaggregation of deaccumulated flux data from an ECMWF model FG field.
     32#    Initially the flux data to be concerned are:
     33#    - large-scale precipitation
     34#    - convective precipitation
     35#    - surface sensible heat flux
     36#    - surface solar radiation
     37#    - u stress
     38#    - v stress
     39#    Different versions of disaggregation is provided for rainfall
     40#    data (darain, modified linear) and the surface fluxes and
     41#    stress data (dapoly, cubic polynomial).
     42#
     43# @Module Content:
     44#    - dapoly
     45#    - darain
     46#
     47#*******************************************************************************
    1048
    11 @Date: March 2018
    12 
    13 @ChangeHistory:
    14     November 2015 - Leopold Haimberger (University of Vienna):
    15         - integrated methods dapoly and darain from Fortran to Python
    16 
    17     April 2018 - Anne Philipp (University of Vienna):
    18         - applied PEP8 style guide
    19         - added structured documentation
    20         - outsourced the disaggregation functions dapoly and darain
    21           to a new module named Disagg
    22 
    23 @License:
    24     (C) Copyright 2015-2018.
    25 
    26     This software is licensed under the terms of the Apache Licence Version 2.0
    27     which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
    28 
    29 @Requirements:
    30     A standard python 2.6 or 2.7 installation
    31 
    32 @Description:
    33     Further documentation may be obtained from www.flexpart.eu.
    34 
    35     Functionality provided:
    36         Disaggregation of deaccumulated flux data from an ECMWF model FG field.
    37         Initially the flux data to be concerned are:
    38         - large-scale precipitation
    39         - convective precipitation
    40         - surface sensible heat flux
    41         - surface solar radiation
    42         - u stress
    43         - v stress
    44 
    45         Different versions of disaggregation is provided for rainfall
    46         data (darain, modified linear) and the surface fluxes and
    47         stress data (dapoly, cubic polynomial).
    48 
    49 """
    5049# ------------------------------------------------------------------------------
    5150# MODULES

  • python/ECFlexpart.py

    refdb01a r991df6a  
    33#************************************************************************
    44# TODO AP
    5 #AP
    65# - specifiy file header documentation
     6# - add class description in header information
    77# - apply classtests
    88# - add references to ECMWF specific software packages
     9# - add describtion of deacc_fluxes
     10# - change name of func deacc ( weil disagg )
     11# - add desc of retrieve function
    912#************************************************************************
    10 """
    11 @Author: Anne Fouilloux (University of Oslo)
    12 
    13 @Date: October 2014
    14 
    15 @ChangeHistory:
    16     November 2015 - Leopold Haimberger (University of Vienna):
    17         - extended with Class Control
    18         - removed functions mkdir_p, daterange, years_between, months_between
    19         - added functions darain, dapoly, toparamId, init128, normalexit,
    20           myerror, cleanup, install_args_and_control,
    21           interpret_args_and_control,
    22         - removed function __del__ in class EIFLexpart
    23         - added the following functions in EIFlexpart:
    24             - create_namelist
    25             - process_output
    26             - deacc_fluxes
    27         - modified existing EIFlexpart - functions for the use in
    28           flex_extract
    29         - retrieve also longer term forecasts, not only analyses and
    30           short term forecast data
    31         - added conversion into GRIB2
    32         - added conversion into .fp format for faster execution of FLEXPART
    33           (see https://www.flexpart.eu/wiki/FpCtbtoWo4FpFormat)
    34 
    35     February 2018 - Anne Philipp (University of Vienna):
    36         - applied PEP8 style guide
    37         - added documentation
    38         - outsourced class Control
    39         - outsourced class MarsRetrieval
    40         - changed class name from EIFlexpart to ECFlexpart
    41         - applied minor code changes (style)
    42 
    43 @License:
    44     (C) Copyright 2014-2018.
    45 
    46     This software is licensed under the terms of the Apache Licence Version 2.0
    47     which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
    48 
    49 @Requirements:
    50     - A standard python 2.6 or 2.7 installation
    51     - dateutils
    52     - matplotlib (optional, for debugging)
    53     - ECMWF specific packages, all available from https://software.ecmwf.int/
    54         ECMWF WebMARS, gribAPI with python enabled, emoslib and
    55         ecaccess web toolkit
    56 
    57 @Description:
    58     Further documentation may be obtained from www.flexpart.eu.
    59 
    60     Functionality provided:
    61         Prepare input 3D-wind fields in hybrid coordinates +
    62         surface fields for FLEXPART runs
    63 """
     13#*******************************************************************************
     14# @Author: Anne Fouilloux (University of Oslo)
     15#
     16# @Date: October 2014
     17#
     18# @Change History:
     19#
     20#    November 2015 - Leopold Haimberger (University of Vienna):
     21#        - extended with class Control
     22#        - removed functions mkdir_p, daterange, years_between, months_between
     23#        - added functions darain, dapoly, toparamId, init128, normalexit,
     24#          myerror, cleanup, install_args_and_control,
     25#          interpret_args_and_control,
     26#        - removed function __del__ in class EIFLexpart
     27#        - added the following functions in EIFlexpart:
     28#            - create_namelist
     29#            - process_output
     30#            - deacc_fluxes
     31#        - modified existing EIFlexpart - functions for the use in
     32#          flex_extract
     33#        - retrieve also longer term forecasts, not only analyses and
     34#          short term forecast data
     35#        - added conversion into GRIB2
     36#        - added conversion into .fp format for faster execution of FLEXPART
     37#          (see https://www.flexpart.eu/wiki/FpCtbtoWo4FpFormat)
     38#
     39#    February 2018 - Anne Philipp (University of Vienna):
     40#        - applied PEP8 style guide
     41#        - added documentation
     42#        - removed function getFlexpartTime in class ECFlexpart
     43#        - outsourced class ControlFile
     44#        - outsourced class MarsRetrieval
     45#        - changed class name from EIFlexpart to ECFlexpart
     46#        - applied minor code changes (style)
     47#        - removed "dead code" , e.g. retrieval of Q since it is not needed
     48#        - removed "times" parameter from retrieve-method since it is not used
     49#
     50# @License:
     51#    (C) Copyright 2014-2018.
     52#
     53#    This software is licensed under the terms of the Apache Licence Version 2.0
     54#    which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
     55#
     56# @Class Description:
     57#    FLEXPART needs grib files in a specifc format. All necessary data fields
     58#    for one time step are stored in a single file. The class represents an
     59#    instance with all the parameter and settings necessary for retrieving
     60#    MARS data and modifing them so they are fitting FLEXPART need. The class
     61#    is able to disaggregate the fluxes and convert grid types to the one needed
     62#    by FLEXPART, therefore using the FORTRAN program.
     63#
     64# @Class Content:
     65#    - __init__
     66#    - write_namelist
     67#    - retrieve
     68#    - process_output
     69#    - create
     70#    - deacc_fluxes
     71#
     72#*******************************************************************************
     73
    6474# ------------------------------------------------------------------------------
    6575# MODULES
    6676# ------------------------------------------------------------------------------
    6777import subprocess
    68 from argparse import ArgumentParser, ArgumentDefaultsHelpFormatter
    69 import traceback
    7078import shutil
    7179import os
    72 import errno
    7380import sys
    7481import inspect
    7582import glob
    7683import datetime
    77 from string import atoi
    7884from numpy import *
     85from argparse import ArgumentParser, ArgumentDefaultsHelpFormatter
    7986ecapi = True
    8087try:
    … …  
    8390    ecapi = False
    8491from gribapi import *
     92
     93# software specific classes and modules from flex_extract
    8594from GribTools import GribTools
    8695from Tools import init128, toparamId, silentremove, product
    87 from Control import Control
     96from ControlFile import ControlFile
    8897from MARSretrieval import MARSretrieval
    8998import Disagg
     99
    90100# ------------------------------------------------------------------------------
    91101# CLASS
    … …  
    93103class ECFlexpart:
    94104    '''
    95     Class to retrieve ECMWF data specific for running FLEXPART.
     105    Class to retrieve FLEXPART specific ECMWF data.
    96106    '''
    97107    # --------------------------------------------------------------------------
    98108    # CLASS FUNCTIONS
    99109    # --------------------------------------------------------------------------
    100     def __init__(self, c, fluxes=False): #done/ verstehen
     110    def __init__(self, c, fluxes=False):
    101111        '''
    102112        @Description:
    … …  
    108118                The current object of the class.
    109119
    110             c: instance of class Control
    111                 Contains all the parameters of control files, which are e.g.:
     120            c: instance of class ControlFile
     121                Contains all the parameters of CONTROL file, which are e.g.:
    112122                DAY1(start_date), DAY2(end_date), DTIME, MAXSTEP, TYPE, TIME,
    113123                STEP, CLASS(marsclass), STREAM, NUMBER, EXPVER, GRID, LEFT,
    … …  
    130140        '''
    131141
    132         # different mars types for retrieving reanalysis data for flexpart
     142        # different mars types for retrieving data for flexpart
    133143        self.types = dict()
    134144        try:
    … …  
    147157        self.basetime = c.basetime
    148158        self.dtime = c.dtime
    149         #self.mars = {}
    150159        i = 0
    151         #j = 0
    152160        if fluxes is True and c.maxstep < 24:
    153161            # no forecast beyond one day is needed!
    154162            # Thus, prepare flux data manually as usual
    155             # with only FC fields with start times at 00/12
     163            # with only forecast fields with start times at 00/12
    156164            # (but without 00/12 fields since these are
    157165            # the initialisation times of the flux fields
    158166            # and therefore are zero all the time)
    159             self.types['FC'] = {'times': '00/12',
    160                                 'steps': '{}/to/12/by/{}'.format(c.dtime,
    161                                                                       c.dtime)}
    162             #i = 1
    163             #for k in [0, 12]:
    164             #    for j in range(int(c.dtime), 13, int(c.dtime)):
    165             #        self.mars['{:0>3}'.format(i * int(c.dtime))] = \
    166             #               [c.type[1], '{:0>2}'.format(k), '{:0>3}'.format(j)]
    167             #        i += 1
     167            self.types[c.type[1]] = {'times': '00/12', 'steps':
     168                                     '{}/to/12/by/{}'.format(c.dtime, c.dtime)}
    168169        else:
    169170            for ty, st, ti in zip(c.type, c.step, c.time):
    … …  
    189190                            self.types[ty]['steps'] += '/'
    190191                        self.types[ty]['steps'] += st
    191 
    192                     #self.mars['{:0>3}'.format(j)] = [ty,
    193                     #                                 '{:0>2}'.format(int(ti)),
    194                     #                                 '{:0>3}'.format(int(st))]
    195                     #j += int(c.dtime)
    196 
    197192                i += 1
    198             print 'EC init: ', self.types #AP
     193
    199194        # Different grids need different retrievals
    200195        # SH = Spherical Harmonics, GG = Gaussian Grid,
    … …  
    274269                self.params['OG__ML'][0] += '/U/V/77'
    275270            elif c.gauss == '0' and c.eta == '0':
    276 #AP then remove?!?!?!?       # this is not recommended (inaccurate)
     271            # this is not recommended (inaccurate)
    277272                self.params['OG__ML'][0] += '/U/V'
    278273            elif c.gauss == '1' and c.eta == '0':
    … …  
    324319
    325320
    326     def write_namelist(self, c, filename): #done
     321    def write_namelist(self, c, filename):
    327322        '''
    328323        @Description:
    … …  
    336331                The current object of the class.
    337332
    338             c: instance of class Control
    339                 Contains all the parameters of control files, which are e.g.:
     333            c: instance of class ControlFile
     334                Contains all the parameters of CONTROL files, which are e.g.:
    340335                DAY1(start_date), DAY2(end_date), DTIME, MAXSTEP, TYPE, TIME,
    341336                STEP, CLASS(marsclass), STREAM, NUMBER, EXPVER, GRID, LEFT,
    … …  
    383378        return
    384379
    385     def retrieve(self, server, dates, times, inputdir=''):
     380    def retrieve(self, server, dates, inputdir='.'):
    386381        '''
    387382        @Description:
    388 
     383            Finalizing the retrieval information by setting final details
     384            depending on grid type.
     385            Prepares MARS retrievals per grid type and submits them.
    389386
    390387        @Input:
    391388            self: instance of ECFlexpart
    392 
    393             server: instance of ECMWFService
    394 
    395             dates:
    396 
    397             times:
     389                The current object of the class.
     390
     391            server: instance of ECMWFService or ECMWFDataServer
     392                The connection to the ECMWF server. This is different
     393                for member state users which have full access and non
     394                member state users which have only access to the public
     395                data sets. The decision is made from command line argument
     396                "public"; for public access its True (ECMWFDataServer)
     397                for member state users its False (ECMWFService)
     398
     399            dates: string
     400                Contains start and end date of the retrieval in the format
     401                "YYYYMMDD/to/YYYYMMDD"
    398402
    399403            inputdir: string, optional
    400                 Default string is empty ('').
     404                Path to the directory where the retrieved data is about
     405                to be stored. The default is the current directory ('.').
    401406
    402407        @Return:
    … …  
    405410        self.dates = dates
    406411        self.server = server
    407 
    408         if inputdir == "":
    409             self.inputdir = '.'
    410         else:
    411             self.inputdir = inputdir
    412 
    413         # Retrieve Q not for using Q but as a template for a reduced gaussian
    414         # grid one date and time is enough
    415         # Take analysis at 00
    416         qdate = self.dates
    417         idx = qdate.find("/")
    418         if (idx > 0):
    419             qdate = self.dates[:idx]
    420 
    421         #QG =  MARSretrieval(self.server, marsclass = self.marsclass, stream = self.stream, type = "an", levtype = "ML", levelist = "1",
    422                              #gaussian = "reduced",grid = '{}'.format((int(self.resol)+1)/2), resol = self.resol,accuracy = self.accuracy,target = self.inputdir+"/"+"QG.grb",
    423                              #date = qdate, time = "00",expver = self.expver, param = "133.128")
    424         #QG.displayInfo()
    425         #QG.dataRetrieve()
    426 
     412        self.inputdir = inputdir
    427413        oro = False
     414
    428415        for ftype in self.types:
    429416            for pk, pv in self.params.iteritems():
    … …  
    453440                    else:
    454441                        continue
    455 
    456442                if pk == 'GG__SL' and pv[0] == 'Q':
    457443                    area = ""
    … …  
    461447                    gaussian = self.gaussian
    462448
     449    # ------  on demand path  --------------------------------------------------
    463450                if self.basetime is None:
    464451                    MR = MARSretrieval(self.server,
    … …  
    473460                    MR.displayInfo()
    474461                    MR.dataRetrieve()
    475     # The whole else section is only necessary for operational scripts.
    476     # It could be removed
     462    # ------  operational path  ------------------------------------------------
    477463                else:
    478464                    # check if mars job requests fields beyond basetime.
    479465                    # If yes eliminate those fields since they may not
    480466                    # be accessible with user's credentials
    481                     sm1 = -1
    482467                    if 'by' in mfstep:
    483468                        sm1 = 2
    484                     tm1 = -1
     469                    else:
     470                        sm1 = -1
     471
    485472                    if 'by' in mftime:
    486473                        tm1 = 2
     474                    else:
     475                        tm1 = -1
     476
    487477                    maxtime = datetime.datetime.strptime(
    488478                                mfdate.split('/')[-1] + mftime.split('/')[tm1],
    489479                                '%Y%m%d%H') + datetime.timedelta(
    490480                                hours=int(mfstep.split('/')[sm1]))
    491 
    492481                    elimit = datetime.datetime.strptime(
    493482                                mfdate.split('/')[-1] +
    … …  
    495484
    496485                    if self.basetime == '12':
     486                        # --------------  flux data ----------------------------
    497487                        if 'acc' in pk:
    498488
    499                 # Strategy: if maxtime-elimit> = 24h reduce date by 1,
    500                 # if 12h< = maxtime-elimit<12h reduce time for last date
    501                 # if maxtime-elimit<12h reduce step for last time
    502                 # A split of the MARS job into 2 is likely necessary.
    503                             maxtime = elimit-datetime.timedelta(hours=24)
     489                        # Strategy:
     490                        # if maxtime-elimit >= 24h reduce date by 1,
     491                        # if 12h <= maxtime-elimit<12h reduce time for last date
     492                        # if maxtime-elimit<12h reduce step for last time
     493                        # A split of the MARS job into 2 is likely necessary.
     494                            maxtime = elimit - datetime.timedelta(hours=24)
    504495                            mfdate = '/'.join(('/'.join(mfdate.split('/')[:-1]),
    505496                                                datetime.datetime.strftime(
    … …  
    541532                            MR.displayInfo()
    542533                            MR.dataRetrieve()
     534                        # --------------  non flux data ------------------------
    543535                        else:
    544536                            MR = MARSretrieval(self.server,
    … …  
    555547                            MR.displayInfo()
    556548                            MR.dataRetrieve()
    557                     else:
     549                    else: # basetime == 0 ??? #AP
     550
    558551                        maxtime = elimit - datetime.timedelta(hours=24)
    559552                        mfdate = datetime.datetime.strftime(maxtime,'%Y%m%d')
    560 
    561553                        mftimesave = ''.join(mftime)
    562554
    … …  
    615607
    616608
    617     def process_output(self, c): #done
     609    def process_output(self, c):
    618610        '''
    619611        @Description:
    620             The grib files are postprocessed depending on selection in
    621             control file. The resulting files are moved to the output
     612            The grib files are postprocessed depending on the selection in
     613            CONTROL file. The resulting files are moved to the output
    622614            directory if its not equla to the input directory.
    623615            The following modifications might be done if
    624             properly switched in control file:
     616            properly switched in CONTROL file:
    625617            GRIB2 - Conversion to GRIB2
    626618            ECTRANS - Transfer of files to gateway server
    … …  
    632624                The current object of the class.
    633625
    634             c: instance of class Control
    635                 Contains all the parameters of control files, which are e.g.:
     626            c: instance of class ControlFile
     627                Contains all the parameters of CONTROL file, which are e.g.:
    636628                DAY1(start_date), DAY2(end_date), DTIME, MAXSTEP, TYPE, TIME,
    637629                STEP, CLASS(marsclass), STREAM, NUMBER, EXPVER, GRID, LEFT,
    … …  
    650642        '''
    651643
    652         print('Postprocessing:\n Format: {}\n'.format(c.format))
     644        print('\n\nPostprocessing:\n Format: {}\n'.format(c.format))
    653645
    654646        if c.ecapi is False:
    … …  
    675667            for ofile in self.outputfilelist:
    676668                p = subprocess.check_call(['ectrans', '-overwrite', '-gateway',
    677                                            c.gateway, '-remote', c.destination,
    678                                            '-source', ofile])
     669                                            c.gateway, '-remote', c.destination,
     670                                            '-source', ofile])
    679671                print('ectrans:', p)
    680672
    … …  
    682674            for ofile in self.outputfilelist:
    683675                p = subprocess.check_call(['ecp', '-o', ofile,
    684                                            os.path.expandvars(c.ecfsdir)])
     676                                            os.path.expandvars(c.ecfsdir)])
    685677
    686678        if c.outputdir != c.inputdir:
    … …  
    693685
    694686            # generate AVAILABLE file
    695             # Example of AVAILABLE file data
     687            # Example of AVAILABLE file data:
    696688            # 20131107 000000      EN13110700              ON DISC
    697689            clist = []
    … …  
    742734                i += 1
    743735
    744 #            clist.sort()
    745             # insert the date and time information of run star and end
     736            # insert the date and time information of run start and end
    746737            # into the list of lines of COMMAND file
    747738            lflist = lflist[:i+1] + \
    … …  
    753744                g.write('\n'.join(lflist) + '\n')
    754745
    755             # change to outputdir and start the
    756             # grib2flexpart run
     746            # change to outputdir and start the grib2flexpart run
    757747            # afterwards switch back to the working dir
    758748            os.chdir(c.outputdir)
    … …  
    765755        return
    766756
    767     def create(self, inputfiles, c): #done
     757    def create(self, inputfiles, c):
    768758        '''
    769759        @Description:
    … …  
    773763            An index file will be created which depends on the combination
    774764            of "date", "time" and "stepRange" values. This is used to iterate
    775             over all messages in the grib files passed through the parameter
    776             "inputfiles" to seperate specific parameters into fort.* files.
    777             Afterwards the FORTRAN program Convert2 is called to convert
     765            over all messages in each grib file which were passed through the
     766            parameter "inputfiles" to seperate specific parameters into fort.*
     767            files. Afterwards the FORTRAN program Convert2 is called to convert
    778768            the data fields all to the same grid and put them in one file
    779             per day.
     769            per unique time step (combination of "date", "time" and
     770            "stepRange").
    780771
    781772        @Input:
    … …  
    786777                Contains a list of files.
    787778
    788             c: instance of class Control
    789                 Contains all the parameters of control files, which are e.g.:
     779            c: instance of class ControlFile
     780                Contains all the parameters of CONTROL files, which are e.g.:
    790781                DAY1(start_date), DAY2(end_date), DTIME, MAXSTEP, TYPE, TIME,
    791782                STEP, CLASS(marsclass), STREAM, NUMBER, EXPVER, GRID, LEFT,
    … …  
    826817            # index_vals[2]: ('0', '12', '3', '6', '9') ; stepRange
    827818
    828         # delete old fort.* files and open them newly
    829819        fdict = {'10':None, '11':None, '12':None, '13':None, '16':None,
    830820                 '17':None, '19':None, '21':None, '22':None, '20':None}
    831         #for f in fdict.keys():
    832         #    silentremove(c.inputdir + "/fort." + f)
    833 
    834821
    835822        for prod in product(*index_vals):
     823            # flag for Fortran program CONVERT2, initially False
     824            convertFlag = False
    836825            print 'current prod: ', prod
    837826            # e.g. prod = ('20170505', '0', '12')
    … …  
    842831                grib_index_select(iid, index_keys[i], prod[i])
    843832
     833            # get first id from current product
    844834            gid = grib_new_from_index(iid)
    845             # do convert2 program if gid at this time is not None,
    846             # therefore save in hid
    847             hid = gid
     835
     836            # if there is data for this product combination
     837            # prepare some date and time parameter before reading the data
    848838            if gid is not None:
     839                # Fortran program CONVERT2 is only done if gid at this time is
     840                # not None, therefore save information in convertFlag
     841                convertFlag = True
     842                # remove old fort.* files and open new ones
    849843                for k, f in fdict.iteritems():
    850844                    silentremove(c.inputdir + "/fort." + k)
    … …  
    900894                pass
    901895
    902 
     896            # helper variable to remember which fields are already used.
    903897            savedfields = []
    904898            while 1:
    … …  
    924918                elif paramId == 155 and gridtype == 'sh':
    925919                    grib_write(gid, fdict['13'])
    926                 elif paramId in [129, 138, 155] and levtype == 'hybrid' \
    927                                                 and c.wrf == '1':
     920                elif  paramId in [129, 138, 155] and levtype == 'hybrid' \
     921                        and c.wrf == '1':
    928922                    pass
    929923                elif paramId == 246 or paramId == 247:
    … …  
    949943                try:
    950944                    if c.wrf == '1':
    951 # die if abfrage scheint ueberfluessig da eh das gleihce ausgefuehrt wird
    952                         if levtype == 'hybrid':
     945                        if levtype == 'hybrid': # model layer
    953946                            if paramId in [129, 130, 131, 132, 133, 138, 155]:
    954947                                grib_write(gid, fwrf)
    955                         else:
     948                        else: # sfc layer
    956949                            if paramId in wrfpars:
    957950                                grib_write(gid, fwrf)
    … …  
    965958                f.close()
    966959
    967             # call for CONVERT2
    968 # AUSLAGERN IN EIGENE FUNKTION
    969 
    970             if hid is not None:
     960            # call for CONVERT2 if flag is True
     961            if convertFlag:
    971962                pwd = os.getcwd()
    972963                os.chdir(c.inputdir)
    … …  
    978969                                to 1 in CONTROL file')
    979970
     971                # create the corresponding output file fort.15
     972                # (generated by CONVERT2) + fort.16 (paramId 167 and 168)
    980973                p = subprocess.check_call([os.path.expandvars(
    981974                        os.path.expanduser(c.exedir)) + '/CONVERT2'], shell=True)
    982975                os.chdir(pwd)
    983                 # create the corresponding output file fort.15
    984                 # (generated by CONVERT2)
    985                 # + fort.16 (paramId 167 and paramId 168)
     976
     977                # create final output filename, e.g. EN13040500 (ENYYMMDDHH)
    986978                fnout = c.inputdir + '/' + c.prefix
    987979                if c.maxstep > 12:
    … …  
    990982                else:
    991983                    suffix = cdateH[2:10]
    992 
    993984                fnout += suffix
    994985                print("outputfile = " + fnout)
    995986                self.outputfilelist.append(fnout) # needed for final processing
    996                 fout = open(fnout, 'wb')
    997                 shutil.copyfileobj(open(c.inputdir + '/fort.15', 'rb'), fout)
    998                 if c.cwc == '1':
    999                     shutil.copyfileobj(open(c.inputdir + '/fort.22', 'rb'), fout)
    1000                 shutil.copyfileobj(open(c.inputdir + '/flux' + cdate[0:2] +
    1001                                         suffix, 'rb'), fout)
    1002                 shutil.copyfileobj(open(c.inputdir + '/fort.16', 'rb'), fout)
    1003                 orolsm = glob.glob(c.inputdir +
    1004                                    '/OG_OROLSM__SL.*.' + c.ppid + '*')[0]
    1005                 shutil.copyfileobj(open(orolsm, 'rb'), fout)
    1006                 fout.close()
     987
     988                # create outputfile and copy all data from intermediate files
     989                # to the outputfile (final GRIB files)
     990                orolsm = os.path.basename(glob.glob(
     991                    c.inputdir + '/OG_OROLSM__SL.*.' + c.ppid + '*')[0])
     992                fluxfile = 'flux' + cdate[0:2] + suffix
     993                if c.cwc != '1':
     994                    flist = ['fort.15', fluxfile, 'fort.16', orolsm]
     995                else:
     996                    flist = ['fort.15', 'fort.22', fluxfile, 'fort.16', orolsm]
     997
     998                with open(fnout, 'wb') as fout:
     999                    for f in flist:
     1000                        shutil.copyfileobj(open(c.inputdir + '/' + f, 'rb'), fout)
     1001
    10071002                if c.omega == '1':
    1008                     fnout = c.outputdir + '/OMEGA'
    1009                     fout  =  open(fnout, 'wb')
    1010                     shutil.copyfileobj(open(c.inputdir + '/fort.25', 'rb'), fout)
     1003                    with open(c.outputdir + '/OMEGA', 'wb') as fout:
     1004                        shutil.copyfileobj(
     1005                            open(c.inputdir + '/fort.25', 'rb'), fout)
    10111006
    10121007        try:
    … …  
    10201015        return
    10211016
    1022 
    10231017    def deacc_fluxes(self, inputfiles, c):
    10241018        '''
    10251019        @Description:
    1026 
     1020            Goes through all flux fields in ordered time and de-accumulate
     1021            the fields. Afterwards the fields are disaggregated in time.
     1022            Different versions of disaggregation is provided for rainfall
     1023            data (darain, modified linear) and the surface fluxes and
     1024            stress data (dapoly, cubic polynomial).
    10271025
    10281026        @Input:
    … …  
    10331031                Contains a list of files.
    10341032
    1035             c: instance of class Control
    1036                 Contains all the parameters of control files, which are e.g.:
     1033            c: instance of class ControlFile
     1034                Contains all the parameters of CONTROL file, which are e.g.:
    10371035                DAY1(start_date), DAY2(end_date), DTIME, MAXSTEP, TYPE, TIME,
    10381036                STEP, CLASS(marsclass), STREAM, NUMBER, EXPVER, GRID, LEFT,
    … …  
    11331131                g = open(gnout, 'w')
    11341132                h = open(hnout, 'w')
     1133
    11351134            print("outputfile = " + fnout)
    11361135            f = open(fnout, 'w')

  • python/GribTools.py

    refdb01a r991df6a  
    33#************************************************************************
    44# TODO AP
    5 #AP
    65# - GribTools name möglicherweise etwas verwirrend.
    76# - change self.filename in self.filenames!!!
    8 # - add file description
    97# - bis auf --init-- und index wird keine Funktion verwendet!?
    108#************************************************************************
    11 """
    12 @Author: Anne Fouilloux (University of Oslo)
    13 
    14 @Date: July 2014
    15 
    16 @ChangeHistory:
    17    February 2018 - Anne Philipp (University of Vienna):
    18         - applied PEP8 style guide
    19         - added documentation
    20         - changed some naming
    21 
    22 @License:
    23     (C) Copyright 2014-2018.
    24 
    25     This software is licensed under the terms of the Apache Licence Version 2.0
    26     which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
    27 
    28 @Requirements:
    29     - A standard python 2.6 or 2.7 installation
    30     - dateutils
    31     - ECMWF specific packages, all available from https://software.ecmwf.int/
    32         ECMWF WebMARS, gribAPI with python enabled, emoslib and
    33         ecaccess web toolkit
    34 
    35 @Description:
    36     Further documentation may be obtained from www.flexpart.eu.
    37 
    38     ...
    39 """
     9#*******************************************************************************
     10# @Author: Anne Fouilloux (University of Oslo)
     11#
     12# @Date: July 2014
     13#
     14# @Change History:
     15#   February 2018 - Anne Philipp (University of Vienna):
     16#        - applied PEP8 style guide
     17#        - added documentation
     18#        - changed some naming
     19#
     20# @License:
     21#    (C) Copyright 2014-2018.
     22#
     23#    This software is licensed under the terms of the Apache Licence Version 2.0
     24#    which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
     25#
     26# @Class Description:
     27#    The GRIB API provides all necessary tools to work directly with the
     28#    grib files. Nevertheless, the GRIB API tools are very basic and are in
     29#    direct connection with the grib files. This class provides some higher
     30#    functions which apply a set of GRIB API tools together in the respective
     31#    context. So, the class initially contains a list of grib files (their
     32#    names) and the using program then applies the methods directly on the
     33#    class objects without having to think about how the actual GRIB API
     34#    tools have to be arranged.
     35#
     36# @Class Content:
     37#    - __init__
     38#    - getkeys
     39#    - setkeys
     40#    - copy
     41#    - index
     42#
     43#*******************************************************************************
     44
    4045# ------------------------------------------------------------------------------
    4146# MODULES
    4247# ------------------------------------------------------------------------------
     48import os
    4349from gribapi import *
    44 import traceback
    45 import sys
    46 import os
     50
    4751# ------------------------------------------------------------------------------
    4852# CLASS

  • python/MARSretrieval.py

    refdb01a r991df6a  
    33#************************************************************************
    44# TODO AP
    5 #AP
    6 # -
    75# -
    86#************************************************************************
    9 ## ------------------------------------------------------------------------------
     7#*******************************************************************************
     8# @Author: Anne Fouilloux (University of Oslo)
     9#
     10# @Date: October 2014
     11#
     12# @Change History:
     13#
     14#   November 2015 - Leopold Haimberger (University of Vienna):
     15#        - optimized displayInfo
     16#        - optimized dataRetrieve and seperate between python and shell
     17#          script call
     18#
     19#   February 2018 - Anne Philipp (University of Vienna):
     20#        - applied PEP8 style guide
     21#        - added documentation
     22#        - applied some minor modifications in programming style/structure
     23#
     24# @License:
     25#    (C) Copyright 2015-2018.
     26#
     27#    This software is licensed under the terms of the Apache Licence Version 2.0
     28#    which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
     29#
     30# @Class Description:
     31#    A MARS revtrieval has a specific syntax with a selection of keywords and
     32#    their corresponding values. This class provides the necessary functions
     33#    by displaying the selected parameters and their values and the actual
     34#    retrievement of the data through a mars request or a Python web api
     35#    interface. The initialization already expects all the keyword values.
     36#
     37# @Class Content:
     38#    - __init__
     39#    - displayInfo
     40#    - dataRetrieve
     41#
     42#*******************************************************************************
     43
     44# ------------------------------------------------------------------------------
    1045# MODULES
    1146# ------------------------------------------------------------------------------
    … …  
    1853except ImportError:
    1954    ecapi = False
    20 #from gribapi import *
     55
    2156# ------------------------------------------------------------------------------
    2257# CLASS

  • python/Tools.py

    refdb01a r991df6a  
    33#************************************************************************
    44# TODO AP
    5 #AP
    6 # -
     5# - check myerror
     6# - check normalexit
     7# - check getListAsString
     8# - seperate args and control interpretation
    79#************************************************************************
    8 """
    9 
    10 """
     10#*******************************************************************************
     11# @Author: Anne Philipp (University of Vienna)
     12#
     13# @Date: May 2018
     14#
     15# @Change History:
     16#    October 2014 - Anne Fouilloux (University of Oslo)
     17#        - created functions silentremove and product (taken from ECMWF)
     18#
     19#    November 2015 - Leopold Haimberger (University of Vienna)
     20#        - created functions: interpret_args_and_control, cleanup
     21#          myerror, normalexit, init128, toparamId
     22#
     23#    April 2018 - Anne Philipp (University of Vienna):
     24#        - applied PEP8 style guide
     25#        - added documentation
     26#        - moved all functions from file FlexpartTools to this file Tools
     27#        - added function getListAsString
     28#
     29# @License:
     30#    (C) Copyright 2014-2018.
     31#
     32#    This software is licensed under the terms of the Apache Licence Version 2.0
     33#    which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
     34#
     35# @Modul Description:
     36#    This module contains a couple of helpful functions which are
     37#    used in different places in flex_extract.
     38#
     39# @Module Content:
     40#    - interpret_args_and_control
     41#    - cleanup
     42#    - myerror
     43#    - normalexit
     44#    - product
     45#    - silentremove
     46#    - init128
     47#    - toparamId
     48#    - getListAsString
     49#
     50#*******************************************************************************
     51
    1152# ------------------------------------------------------------------------------
    1253# MODULES
    1354# ------------------------------------------------------------------------------
    14 from argparse import ArgumentParser, ArgumentDefaultsHelpFormatter
    1555import os
    1656import errno
    1757import sys
    1858import glob
     59import traceback
    1960from numpy import *
    2061from gribapi import *
    21 import Control
     62from argparse import ArgumentParser, ArgumentDefaultsHelpFormatter
     63
     64# software specific class from flex_extract
     65from ControlFile import ControlFile
    2266
    2367# ------------------------------------------------------------------------------
    … …  
    2872    '''
    2973    @Description:
    30         Assigns the command line arguments and reads control file
     74        Assigns the command line arguments and reads CONTROL file
    3175        content. Apply default values for non mentioned arguments.
    3276
    … …  
    3882            Contains the commandline arguments from script/program call.
    3983
    40         c: instance of class Control
    41             Contains all necessary information of a control file. The parameters
     84        c: instance of class ControlFile
     85            Contains all necessary information of a CONTROL file. The parameters
    4286            are: DAY1, DAY2, DTIME, MAXSTEP, TYPE, TIME, STEP, CLASS, STREAM,
    4387            NUMBER, EXPVER, GRID, LEFT, LOWER, UPPER, RIGHT, LEVEL, LEVELIST,
    … …  
    62106                        help="# of days to be retrieved at once")
    63107
    64     # some arguments that override the default in the control file
     108    # some arguments that override the default in the CONTROL file
    65109    parser.add_argument("--basetime", dest="basetime",
    66110                        help="base such as 00/12 (for half day retrievals)")
    … …  
    96140    parser.add_argument("--controlfile", dest="controlfile",
    97141                        default='CONTROL.temp',
    98                         help="file with control parameters")
     142                        help="file with CONTROL parameters")
    99143    parser.add_argument("--debug", dest="debug", default=0,
    100144                        help="Debug mode - leave temporary files intact")
    … …  
    102146    args = parser.parse_args()
    103147
    104     # create instance of Control for specified controlfile
     148    # create instance of ControlFile for specified controlfile
    105149    # and assign the parameters (and default values if necessary)
    106150    try:
    107         c = Control.Control(args.controlfile)
     151        c = ControlFile(args.controlfile)
    108152    except IOError:
    109153        try:
    110             c = Control.Control(localpythonpath + args.controlfile)
     154            c = ControlFile(localpythonpath + args.controlfile)
    111155        except:
    112             print('Could not read control file "' + args.controlfile + '"')
     156            print('Could not read CONTROL file "' + args.controlfile + '"')
    113157            print('Either it does not exist or its syntax is wrong.')
    114158            print('Try "' + sys.argv[0].split('/')[-1] +
    … …  
    119163    if  args.start_date is None and getattr(c, 'start_date') is None:
    120164        print('start_date specified neither in command line nor \
    121                in control file ' + args.controlfile)
     165               in CONTROL file ' + args.controlfile)
    122166        print('Try "' + sys.argv[0].split('/')[-1] +
    123167              ' -h" to print usage information')
    124168        exit(1)
    125169
    126     # save all existing command line parameter to the Control instance
     170    # save all existing command line parameter to the ControlFile instance
    127171    # if parameter is not specified through the command line or CONTROL file
    128172    # set default values
    … …  
    199243    @Description:
    200244        Remove all files from intermediate directory
    201         (inputdir from control file).
    202 
    203     @Input:
    204         c: instance of class Control
    205             Contains all the parameters of control files, which are e.g.:
     245        (inputdir from CONTROL file).
     246
     247    @Input:
     248        c: instance of class ControlFile
     249            Contains all the parameters of CONTROL file, which are e.g.:
    206250            DAY1(start_date), DAY2(end_date), DTIME, MAXSTEP, TYPE, TIME,
    207251            STEP, CLASS(marsclass), STREAM, NUMBER, EXPVER, GRID, LEFT,
    … …  
    240284
    241285    @Input:
    242         c: instance of class Control
    243             Contains all the parameters of control files, which are e.g.:
     286        c: instance of class ControlFile
     287            Contains all the parameters of CONTROL file, which are e.g.:
    244288            DAY1(start_date), DAY2(end_date), DTIME, MAXSTEP, TYPE, TIME,
    245289            STEP, CLASS(marsclass), STREAM, NUMBER, EXPVER, GRID, LEFT,
    … …  
    290334
    291335    @Input:
    292         c: instance of class Control
    293             Contains all the parameters of control files, which are e.g.:
     336        c: instance of class ControlFile
     337            Contains all the parameters of CONTROL file, which are e.g.:
    294338            DAY1(start_date), DAY2(end_date), DTIME, MAXSTEP, TYPE, TIME,
    295339            STEP, CLASS(marsclass), STREAM, NUMBER, EXPVER, GRID, LEFT,
    … …  
    357401            See example in description above.
    358402    '''
    359 
    360403    pools = map(tuple, args) * kwds.get('repeat', 1)
    361404    result = [[]]
    … …  
    371414    '''
    372415    @Description:
    373         Removes the file which name is passed to the function if
    374         it exists. The function does not fail if the file does not
    375         exist.
     416        If "filename" exists , it is removed.
     417        The function does not fail if the file does not exist.
    376418
    377419    @Input:
    … …  
    426468    @Input:
    427469        pars: string
    428             Addpar argument from control file in the format of
     470            Addpar argument from CONTROL file in the format of
    429471            parameter names instead of ids. The parameter short
    430472            names are sepearted with "/" and they are passed as
    … …  
    438480    @Return:
    439481        ipar: list of integer
    440             List of addpar parameters from control file transformed to
     482            List of addpar parameters from CONTROL file transformed to
    441483            parameter ids in the format of integer.
    442484    '''
    … …  
    455497    return ipar
    456498
    457 def getListAsString(listobj):
    458     '''
    459     @Description:
    460     '''
    461     return ", ".join( str(l) for l in listobj)
     499def getListAsString(listObj):
     500    '''
     501    @Description:
     502        Converts a list of arbitrary content into a single string.
     503
     504    @Input:
     505        listObj: list
     506            A list with arbitrary content.
     507
     508    @Return:
     509        strOfList: string
     510            The content of the list as a single string.
     511    '''
     512
     513    strOfList = ", ".join( str(l) for l in listObj)
     514
     515    return strOfList

  • python/UIOFiles.py

    refdb01a r991df6a  
    33#************************************************************************
    44# TODO AP
    5 #AP
    65# - checken welche regelmässigen methoden auf diese Files noch angewendet werden
    76# und dann hier implementieren
    8 # - add description of file!
     7# cleanup hier rein
    98#************************************************************************
    10 """
    11 @Author: Anne Fouilloux (University of Oslo)
     9#*******************************************************************************
     10# @Author: Anne Fouilloux (University of Oslo)
     11#
     12# @Date: October 2014
     13#
     14# @Change History:
     15#
     16#    November 2015 - Leopold Haimberger (University of Vienna):
     17#        - modified method listFiles to work with glob instead of listdir
     18#        - added pattern search in method listFiles
     19#
     20#    February 2018 - Anne Philipp (University of Vienna):
     21#        - applied PEP8 style guide
     22#        - added documentation
     23#        - optimisation of method listFiles since it didn't work correctly
     24#          for sub directories
     25#        - additional speed up of method listFiles
     26#        - modified the class so that it is initiated with a pattern instead
     27#          of suffixes. Gives more precision in selection of files.
     28#
     29# @License:
     30#    (C) Copyright 2014-2018.
     31#
     32#    This software is licensed under the terms of the Apache Licence Version 2.0
     33#    which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
     34#
     35# @Class Decription:
     36#    The class is for file manipulation. It is initiated with a regular
     37#    expression pattern for this instance and can produce a list of Files
     38#    from the given file pattern. These files can be deleted.
     39#
     40# @Class Content:
     41#    - __init__
     42#    - listFiles
     43#    - deleteFiles
     44#
     45#*******************************************************************************
    1246
    13 @Date: October 2014
    14 
    15 @ChangeHistory:
    16     November 2015 - Leopold Haimberger (University of Vienna):
    17         - modified method listFiles to work with glob instead of listdir
    18         - added pattern search in method listFiles
    19 
    20     February 2018 - Anne Philipp (University of Vienna):
    21         - applied PEP8 style guide
    22         - added documentation
    23         - optimisation of method listFiles since it didn't work correctly
    24           for sub directories
    25         - additional speed up of method listFiles
    26 
    27 @License:
    28     (C) Copyright 2014-2018.
    29 
    30     This software is licensed under the terms of the Apache Licence Version 2.0
    31     which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
    32 
    33 @Requirements:
    34     A standard python 2.6 or 2.7 installation
    35 
    36 @Description:
    37     ...
    38 """
    3947# ------------------------------------------------------------------------------
    4048# MODULES
    … …  
    4452import fnmatch
    4553import time
     54
     55# software specific module from flex_extract
    4656import profiling
     57from Tools import silentremove
     58
    4759# ------------------------------------------------------------------------------
    4860# CLASS
    4961# ------------------------------------------------------------------------------
     62
    5063class UIOFiles:
    5164    '''
    5265    Class to manipulate files. At initialisation it has the attribute
    53     suffix which stores a list of suffixes of the files associated
     66    pattern which stores a regular expression pattern for the files associated
    5467    with the instance of the class.
    5568    '''
    … …  
    5770    # CLASS FUNCTIONS
    5871    # --------------------------------------------------------------------------
    59     def __init__(self, suffix):
     72    def __init__(self, pattern):
    6073        '''
    6174        @Description:
    62             Assignes the suffixes of the files which should be
    63             associated with the instance of the class.
     75            Assignes a specific pattern for these files.
    6476
    6577        @Input:
    … …  
    6779                Description see class documentation.
    6880
    69             suffix: list of strings
    70                 The types of files which should be manipulated such as
    71                 ['grib', 'grb', 'grib1', 'grib2', 'grb1', 'grb2']
     81            pattern: string
     82                Regular expression pattern. For example: '*.grb'
    7283
    7384        @Return:
    … …  
    7586        '''
    7687
    77         self.suffix = suffix
     88        self.pattern = pattern
    7889
    7990        return
    8091
    8192    #@profiling.timefn
    82     def listFiles(self, path, pattern, callid=0):
     93    def listFiles(self, path, callid=0):
    8394        '''
    8495        @Description:
    8596            Lists all files in the directory with the matching
    86             regular expression pattern. The suffixes are already stored
    87             in a list attribute "suffix".
     97            regular expression pattern.
    8898
    8999        @Input:
    … …  
    93103            path: string
    94104                Directory where to list the files.
    95 
    96             pattern: string
    97                 Regular expression pattern. For example:
    98                 '*OG_acc_SL*.'+c.ppid+'.*'
    99105
    100106            callid: integer
    … …  
    115121
    116122        # get the file list of the path if its not a directory and
    117         # if it contains one of the suffixes
     123        # if it contains the pattern
    118124        self.files.extend([os.path.join(path, k) for k in os.listdir(path)
    119                            if fnmatch.fnmatch(k, pattern) and
    120                            os.path.splitext(k)[-1] in self.suffix])
     125                           if fnmatch.fnmatch(k, self.pattern)])
    121126
    122127        # find possible sub-directories in the path
    … …  
    127132        if subdirs:
    128133            for subdir in subdirs:
    129                 self.listFiles(os.path.join(path, subdir), pattern, callid=1)
     134                self.listFiles(os.path.join(path, subdir), callid=1)
    130135
    131136        return
     137
     138    def deleteFiles(self):
     139        '''
     140        @Description:
     141            Deletes the files.
     142
     143        @Input:
     144            self: instance of UIOFiles
     145                Description see class documentation.
     146
     147        @Return:
     148            <nothing>
     149        '''
     150
     151        for f in self.files:
     152            silentremove(f)
     153
     154        return

  • python/compilejob.ksh

    refdb01a r991df6a  
    55# start with sbatch NAME_OF_THIS_FILE directly on machine
    66
    7 #SBATCH --workdir=/scratch/ms/at/km4a
     7#SBATCH --workdir=/scratch/ms/spatlh00/lh0
    88#SBATCH --qos=normal
    99#SBATCH --job-name=flex_ecmwf
    … …  
    3333  module load grib_api/1.14.5
    3434  module load emos/437-r64
    35 export FLEXPART_ROOT_SCRIPTS=${HOME}
     35export FLEXPART_ROOT_SCRIPTS=$HOME
    3636#  export ECMWFDATA=$FLEXPART_ROOT/ECMWFDATA$VERSION
    3737#  export PYTHONPATH=$ECMWFDATA/python
    … …  
    4949  export GROUP=`echo $HOME | awk -F / '{print $4}'`
    5050  export SCRATCH=/scratch/ms/${GROUP}/${USER}
    51 export FLEXPART_ROOT_SCRIPTS=${HOME}
     51export FLEXPART_ROOT_SCRIPTS=$HOME
    5252#  export ECMWFDATA=$FLEXPART_ROOT/ECMWFDATA$VERSION
    5353#  export PYTHONPATH=$ECMWFDATA/python

  • python/getMARSdata.py

    refdb01a r991df6a  
    33#************************************************************************
    44# TODO AP
    5 # - Change History ist nicht angepasst ans File!
    6 # - add file description
     5# - add function docstrings!!!!
    76#************************************************************************
    8 """
    9 @Author: Anne Fouilloux (University of Oslo)
    10 
    11 @Date: October 2014
    12 
    13 @ChangeHistory:
    14     November 2015 - Leopold Haimberger (University of Vienna):
    15         - using the WebAPI also for general MARS retrievals
    16         - job submission on ecgate and cca
    17         - job templates suitable for twice daily operational dissemination
    18         - dividing retrievals of longer periods into digestable chunks
    19         - retrieve also longer term forecasts, not only analyses and
    20           short term forecast data
    21         - conversion into GRIB2
    22         - conversion into .fp format for faster execution of FLEXPART
    23 
    24     February 2018 - Anne Philipp (University of Vienna):
    25         - applied PEP8 style guide
    26         - added documentation
    27         - minor changes in programming style for consistence
    28 
    29 @License:
    30     (C) Copyright 2014-2018.
    31 
    32     This software is licensed under the terms of the Apache Licence Version 2.0
    33     which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
    34 
    35 @Requirements:
    36     - A standard python 2.6 or 2.7 installation
    37     - dateutils
    38     - ECMWF specific packages, all available from https://software.ecmwf.int/
    39         ECMWF WebMARS, gribAPI with python enabled, emoslib and
    40         ecaccess web toolkit
    41 
    42 @Description:
    43     Further documentation may be obtained from www.flexpart.eu.
    44 
    45 """
     7#*******************************************************************************
     8# @Author: Anne Fouilloux (University of Oslo)
     9#
     10# @Date: October 2014
     11#
     12# @Change History:
     13#
     14#    November 2015 - Leopold Haimberger (University of Vienna):
     15#        - moved the getEIdata program into a function "getMARSdata"
     16#        - moved the AgurmentParser into a seperate function
     17#        - adatpted the function for the use in flex_extract
     18#        - renamed file to getMARSdata
     19#
     20#    February 2018 - Anne Philipp (University of Vienna):
     21#        - applied PEP8 style guide
     22#        - added structured documentation
     23#        - minor changes in programming style for consistence
     24#        - added function main and moved function calls vom __main__ there
     25#          (necessary for better documentation with docstrings for later
     26#          online documentation)
     27#        - use of UIFiles class for file selection and deletion
     28
     29#
     30# @License:
     31#    (C) Copyright 2014-2018.
     32#
     33#    This software is licensed under the terms of the Apache Licence Version 2.0
     34#    which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
     35#
     36# @Program Functionality:
     37#    This program can be used as a module in the whole flex_extract process
     38#    or can be run by itself to just extract MARS data from ECMWF. To do so,
     39#    a couple of necessary parameters has to be passed with the program call.
     40#    See documentation for more details.
     41#
     42# @Program Content:
     43#    - main
     44#    - getMARSdata
     45#
     46#*******************************************************************************
     47
    4648# ------------------------------------------------------------------------------
    4749# MODULES
    4850# ------------------------------------------------------------------------------
     51import os
     52import sys
     53import datetime
     54import inspect
    4955try:
    5056    ecapi=True
    … …  
    5359    ecapi=False
    5460
    55 import calendar
    56 import shutil
    57 import datetime
    58 import time
    59 import os
    60 import glob
    61 import sys
    62 import inspect
    63 # add path to submit.py to pythonpath so that python finds its buddies
    64 localpythonpath=os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
     61# add path to pythonpath so that python finds its buddies
     62localpythonpath = os.path.dirname(os.path.abspath(
     63    inspect.getfile(inspect.currentframe())))
    6564if localpythonpath not in sys.path:
    6665    sys.path.append(localpythonpath)
    6766
    68 from Control import Control
     67# software specific classes and modules from flex_extract
     68from ControlFile import ControlFile
    6969from Tools import myerror, normalexit, \
    7070                          interpret_args_and_control
    7171from ECFlexpart import ECFlexpart
     72from UIOFiles import UIOFiles
    7273
    7374# ------------------------------------------------------------------------------
    7475# FUNCTION
    7576# ------------------------------------------------------------------------------
     77def main():
     78    '''
     79    @Description:
     80        If getMARSdata is called from command line, this function controls
     81        the program flow and calls the argumentparser function and
     82        the getMARSdata function for retrieving EC data.
     83
     84    @Input:
     85        <nothing>
     86
     87    @Return:
     88        <nothing>
     89    '''
     90    args, c = interpret_args_and_control()
     91    getMARSdata(args, c)
     92    normalexit(c)
     93
     94    return
     95
    7696def getMARSdata(args, c):
    77 
     97    '''
     98    @Description:
     99        Retrieves the EC data needed for a FLEXPART simulation.
     100        Start and end dates for retrieval period is set. Retrievals
     101        are divided into smaller periods if necessary and datechunk parameter
     102        is set.
     103
     104    @Input:
     105        args: instance of ArgumentParser
     106            Contains the commandline arguments from script/program call.
     107
     108        c: instance of class ControlFile
     109            Contains all the parameters of CONTROL file, which are e.g.:
     110            DAY1(start_date), DAY2(end_date), DTIME, MAXSTEP, TYPE, TIME,
     111            STEP, CLASS(marsclass), STREAM, NUMBER, EXPVER, GRID, LEFT,
     112            LOWER, UPPER, RIGHT, LEVEL, LEVELIST, RESOL, GAUSS, ACCURACY,
     113            OMEGA, OMEGADIFF, ETA, ETADIFF, DPDETA, SMOOTH, FORMAT,
     114            ADDPAR, WRF, CWC, PREFIX, ECSTORAGE, ECTRANS, ECFSDIR,
     115            MAILOPS, MAILFAIL, GRIB2FLEXPART, FLEXPARTDIR, BASETIME
     116            DATE_CHUNK, DEBUG, INPUTDIR, OUTPUTDIR, FLEXPART_ROOT_SCRIPTS
     117
     118            For more information about format and content of the parameter
     119            see documentation.
     120
     121    @Return:
     122        <nothing>
     123    '''
    78124
    79125    if not os.path.exists(c.inputdir):
    80126        os.makedirs(c.inputdir)
     127
     128    print("Retrieving EC data!")
    81129    print("start date %s " % (c.start_date))
    82130    print("end date %s " % (c.end_date))
    … …  
    88136
    89137    c.ecapi = ecapi
    90     print 'ecapi:', c.ecapi
    91 
    92 # Retrieve EC data for running flexpart
    93 #AP change this variant to correct format conversion with datetime
    94 #AP import datetime and timedelta explicitly
    95     syear = int(c.start_date[:4])
    96     smonth = int(c.start_date[4:6])
    97     sday = int(c.start_date[6:])
    98     start = datetime.date(year=syear, month=smonth, day=sday)
     138    print 'ecapi: ', c.ecapi
     139
     140    # set start date of retrieval period
     141    start = datetime.date(year=int(c.start_date[:4]),
     142                          month=int(c.start_date[4:6]),
     143                          day=int(c.start_date[6:]))
    99144    startm1 = start - datetime.timedelta(days=1)
    100145    if c.basetime == '00':
    101146        start = startm1
    102147
    103     eyear = int(c.end_date[:4])
    104     emonth = int(c.end_date[4:6])
    105     eday = int(c.end_date[6:])
    106     end = datetime.date(year=eyear, month=emonth, day=eday)
     148    # set end date of retrieval period
     149    end = datetime.date(year=int(c.end_date[:4]),
     150                        month=int(c.end_date[4:6]),
     151                        day=int(c.end_date[6:]))
    107152    if c.basetime == '00' or c.basetime == '12':
    108153        endp1 = end + datetime.timedelta(days=1)
    … …  
    110155        endp1 = end + datetime.timedelta(days=2)
    111156
     157    # set time period of one single retrieval
    112158    datechunk = datetime.timedelta(days=int(c.date_chunk))
    113159
    114     # retrieving of accumulated data fields (flux data), (maximum one month)
    115 
    116     # remove old files
    117     print 'removing content of ' + c.inputdir
    118     tobecleaned = glob.glob(c.inputdir + '/*_acc_*.' + \
    119                             str(os.getppid()) + '.*.grb')
    120     for f in tobecleaned:
    121         os.remove(f)
    122 
    123     times = None
     160    # --------------  flux data ------------------------------------------------
     161    print 'removing old flux content of ' + c.inputdir
     162    tobecleaned = UIOFiles('*_acc_*.' + str(os.getppid()) + '.*.grb')
     163    tobecleaned.listFiles(c.inputdir)
     164    tobecleaned.deleteFiles()
     165
    124166    # if forecast for maximum one day (upto 23h) are to be retrieved,
    125167    # collect accumulation data (flux data)
    … …  
    129171        day = startm1
    130172        while day < endp1:
    131                 # retrieve MARS data for the whole period
    132                 flexpart = ECFlexpart(c, fluxes=True)
    133                 tmpday = day + datechunk - datetime.timedelta(days=1)
    134                 if tmpday < endp1:
    135                     dates = day.strftime("%Y%m%d") + "/to/" + \
    136                             tmpday.strftime("%Y%m%d")
    137                 else:
    138                     dates = day.strftime("%Y%m%d") + "/to/" + \
    139                             end.strftime("%Y%m%d")
    140 
    141                 print "retrieve " + dates + " in dir " + c.inputdir
    142 
    143                 try:
    144                     flexpart.retrieve(server, dates, times, c.inputdir)
    145                 except IOError:
    146                     myerror(c,'MARS request failed')
    147 
    148                 day += datechunk
     173            # retrieve MARS data for the whole period
     174            flexpart = ECFlexpart(c, fluxes=True)
     175            tmpday = day + datechunk - datetime.timedelta(days=1)
     176            if tmpday < endp1:
     177                dates = day.strftime("%Y%m%d") + "/to/" + \
     178                        tmpday.strftime("%Y%m%d")
     179            else:
     180                dates = day.strftime("%Y%m%d") + "/to/" + \
     181                        end.strftime("%Y%m%d")
     182
     183            print "retrieve " + dates + " in dir " + c.inputdir
     184
     185            try:
     186                flexpart.retrieve(server, dates, c.inputdir)
     187            except IOError:
     188                myerror(c, 'MARS request failed')
     189
     190            day += datechunk
    149191
    150192    # if forecast data longer than 24h are to be retrieved,
    … …  
    156198        day = start
    157199        while day <= end:
    158                 # retrieve MARS data for the whole period
    159                 flexpart = ECFlexpart(c, fluxes=True)
    160                 tmpday = day + datechunk - datetime.timedelta(days=1)
    161                 if tmpday < end:
    162                     dates = day.strftime("%Y%m%d") + "/to/" + \
    163                             tmpday.trftime("%Y%m%d")
    164                 else:
    165                     dates = day.strftime("%Y%m%d") + "/to/" + \
    166                             end.strftime("%Y%m%d")
    167 
    168                 print "retrieve " + dates + " in dir " + c.inputdir
    169 
    170                 try:
    171                     flexpart.retrieve(server, dates, times, c.inputdir)
    172                 except IOError:
    173                     myerror(c, 'MARS request failed')
    174 
    175                 day += datechunk
    176 
    177     # retrieving of normal data fields (non flux data), (maximum one month)
    178 
    179     # remove old *__* files
    180     tobecleaned = glob.glob(c.inputdir + '/*__*.' +
    181                             str(os.getppid()) + '.*.grb')
    182     for f in tobecleaned:
    183         os.remove(f)
     200            # retrieve MARS data for the whole period
     201            flexpart = ECFlexpart(c, fluxes=True)
     202            tmpday = day + datechunk - datetime.timedelta(days=1)
     203            if tmpday < end:
     204                dates = day.strftime("%Y%m%d") + "/to/" + \
     205                        tmpday.trftime("%Y%m%d")
     206            else:
     207                dates = day.strftime("%Y%m%d") + "/to/" + \
     208                        end.strftime("%Y%m%d")
     209
     210            print "retrieve " + dates + " in dir " + c.inputdir
     211
     212            try:
     213                flexpart.retrieve(server, dates, c.inputdir)
     214            except IOError:
     215                myerror(c, 'MARS request failed')
     216
     217            day += datechunk
     218
     219    # --------------  non flux data --------------------------------------------
     220    print 'removing old non flux content of ' + c.inputdir
     221    tobecleaned = UIOFiles('*__*.' + str(os.getppid()) + '.*.grb')
     222    tobecleaned.listFiles(c.inputdir)
     223    tobecleaned.deleteFiles()
     224
    184225    day = start
    185     times = None
    186226    while day <= end:
    187             # retrieve MARS data for the whole period
     227            # retrieve all non flux MARS data for the whole period
    188228            flexpart = ECFlexpart(c, fluxes=False)
    189             tmpday = day+datechunk-datetime.timedelta(days=1)
     229            tmpday = day + datechunk - datetime.timedelta(days=1)
    190230            if tmpday < end:
    191231                dates = day.strftime("%Y%m%d") + "/to/" + \
    … …  
    198238
    199239            try:
    200                 flexpart.retrieve(server, dates, times, c.inputdir)
     240                flexpart.retrieve(server, dates, c.inputdir)
    201241            except IOError:
    202242                myerror(c, 'MARS request failed')
    … …  
    207247
    208248if __name__ == "__main__":
    209 
    210     args, c = interpret_args_and_control()
    211     getMARSdata(args, c)
    212     normalexit(c)
     249    main()
     250

  • python/install.py

    refdb01a r991df6a  
    33#************************************************************************
    44# TODO AP
    5 #AP
    6 # - Functionality Provided is not correct for this file
    75# - localpythonpath should not be set in module load section!
    86# - create a class Installation and divide installation in 3 subdefs for
    … …  
    108# - Change History ist nicht angepasst ans File! Original geben lassen
    119#************************************************************************
    12 """
    13 @Author: Anne Fouilloux (University of Oslo)
    14 
    15 @Date: October 2014
    16 
    17 @ChangeHistory:
    18     November 2015 - Leopold Haimberger (University of Vienna):
    19         - using the WebAPI also for general MARS retrievals
    20         - job submission on ecgate and cca
    21         - job templates suitable for twice daily operational dissemination
    22         - dividing retrievals of longer periods into digestable chunks
    23         - retrieve also longer term forecasts, not only analyses and
    24           short term forecast data
    25         - conversion into GRIB2
    26         - conversion into .fp format for faster execution of FLEXPART
    27 
    28     February 2018 - Anne Philipp (University of Vienna):
    29         - applied PEP8 style guide
    30         - added documentation
    31 
    32 @License:
    33     (C) Copyright 2014 UIO.
    34 
    35     This software is licensed under the terms of the Apache Licence Version 2.0
    36     which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
    37 
    38 @Requirements:
    39     - A standard python 2.6 or 2.7 installation
    40     - dateutils
    41     - matplotlib (optional, for debugging)
    42     - ECMWF specific packages, all available from https://software.ecmwf.int/
    43         ECMWF WebMARS, gribAPI with python enabled, emoslib and
    44         ecaccess web toolkit
    45 
    46 @Description:
    47     Further documentation may be obtained from www.flexpart.eu.
    48 
    49     Functionality provided:
    50         Prepare input 3D-wind fields in hybrid coordinates +
    51         surface fields for FLEXPART runs
    52 """
     10#*******************************************************************************
     11# @Author: Leopold Haimberger (University of Vienna)
     12#
     13# @Date: November 2015
     14#
     15# @Change History:
     16#
     17#    February 2018 - Anne Philipp (University of Vienna):
     18#        - applied PEP8 style guide
     19#        - added documentation
     20#
     21# @License:
     22#    (C) Copyright 2015-2018.
     23#
     24#    This software is licensed under the terms of the Apache Licence Version 2.0
     25#    which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
     26#
     27# @Program Functionality:
     28#    Depending on the selected installation environment (locally or on the
     29#    ECMWF server ecgate or cca) the program extracts the commandline
     30#    arguments and the CONTROL file parameter and prepares the corresponding
     31#    environment. The necessary files are collected in a tar-ball and placed
     32#    at the target location. There its untared, the environment variables will
     33#    be set and the Fortran code will be compiled. If the ECMWF environment is
     34#    selected a job script is prepared and submitted for the remaining
     35#    configurations after putting the tar-ball to the target ECMWF server.
     36#
     37# @Program Content:
     38#    - main
     39#    - install_args_and_control
     40#    - install_via_gateway
     41#
     42#*******************************************************************************
     43
    5344# ------------------------------------------------------------------------------
    5445# MODULES
    5546# ------------------------------------------------------------------------------
    56 import calendar
    57 import shutil
    5847import datetime
    59 import time
    60 import os,sys,glob
     48import os
     49import sys
     50import glob
    6151import subprocess
    6252import inspect
    63 # add path to submit.py to pythonpath so that python finds its buddies
    64 localpythonpath=os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
    65 sys.path.append(localpythonpath)
    66 from UIOFiles import UIOFiles
    67 from string import strip
    6853from argparse import ArgumentParser,ArgumentDefaultsHelpFormatter
    69 from GribTools import GribTools
    70 from Control import Control
    71 from getMARSdata import getMARSdata
    72 from prepareFLEXPART import prepareFLEXPART
    73 from ECFlexpart import ECFlexpart
     54
     55# add path to pythonpath so that python finds its buddies
     56localpythonpath = os.path.dirname(os.path.abspath(
     57    inspect.getfile(inspect.currentframe())))
     58if localpythonpath not in sys.path:
     59    sys.path.append(localpythonpath)
     60
     61# software specific classes and modules from flex_extract
     62from ControlFile import ControlFile
    7463
    7564# ------------------------------------------------------------------------------
    7665# FUNCTIONS
    7766# ------------------------------------------------------------------------------
     67def main():
     68    '''
     69    @Description:
     70        Controls the installation process. Calls the installation function
     71        if target is specified.
     72
     73    @Intput:
     74        <nothing>
     75
     76    @Return:
     77        <nothing>
     78    '''
     79
     80    os.chdir(localpythonpath)
     81    args, c = install_args_and_control()
     82
     83    if args.install_target is not None:
     84        install_via_gateway(c, args.install_target)
     85    else:
     86        print('Please specify installation target (local|ecgate|cca)')
     87        print('use -h or --help for help')
     88
     89    sys.exit()
     90
     91    return
     92
     93
    7894def install_args_and_control():
    7995    '''
    8096    @Description:
    8197        Assigns the command line arguments for installation and reads
    82         control file content. Apply default values for non mentioned arguments.
     98        CONTROL file content. Apply default values for non mentioned arguments.
    8399
    84100    @Input:
    … …  
    89105            Contains the commandline arguments from script/program call.
    90106
    91         c: instance of class Control
    92             Contains all necessary information of a control file. The parameters
     107        c: instance of class ControlFile
     108            Contains all necessary information of a CONTROL file. The parameters
    93109            are: DAY1, DAY2, DTIME, MAXSTEP, TYPE, TIME, STEP, CLASS, STREAM,
    94110            NUMBER, EXPVER, GRID, LEFT, LOWER, UPPER, RIGHT, LEVEL, LEVELIST,
    … …  
    130146    parser.add_argument("--controlfile", dest="controlfile",
    131147                        default='CONTROL.temp',
    132                         help="file with control parameters")
     148                        help="file with CONTROL parameters")
    133149
    134150    args = parser.parse_args()
    135151
    136152    try:
    137         c = Control(args.controlfile)
     153        c = ControlFile(args.controlfile)
    138154    except:
    139         print('Could not read control file "' + args.controlfile + '"')
     155        print('Could not read CONTROL file "' + args.controlfile + '"')
    140156        print('Either it does not exist or its syntax is wrong.')
    141157        print('Try "' + sys.argv[0].split('/')[-1] +
    … …  
    182198
    183199
    184 def main():
    185     '''
    186     '''
    187     os.chdir(localpythonpath)
    188     args, c = install_args_and_control()
    189     if args.install_target is not None:
    190         install_via_gateway(c, args.install_target)
    191     else:
    192         print('Please specify installation target (local|ecgate|cca)')
    193         print('use -h or --help for help')
    194     sys.exit()
    195 
    196200def install_via_gateway(c, target):
    197 
     201    '''
     202    @Description:
     203        Perform the actual installation on local machine or prepare data
     204        transfer to remote gate and submit a job script which will
     205        install everything on the remote gate.
     206
     207    @Input:
     208        c: instance of class ControlFile
     209            Contains all necessary information of a CONTROL file. The parameters
     210            are: DAY1, DAY2, DTIME, MAXSTEP, TYPE, TIME, STEP, CLASS, STREAM,
     211            NUMBER, EXPVER, GRID, LEFT, LOWER, UPPER, RIGHT, LEVEL, LEVELIST,
     212            RESOL, GAUSS, ACCURACY, OMEGA, OMEGADIFF, ETA, ETADIFF, DPDETA,
     213            SMOOTH, FORMAT, ADDPAR, WRF, CWC, PREFIX, ECSTORAGE, ECTRANS,
     214            ECFSDIR, MAILOPS, MAILFAIL, GRIB2FLEXPART, FLEXPARTDIR
     215            For more information about format and content of the parameter see
     216            documentation.
     217
     218        target: string
     219            The target where the installation should be processed.
     220            E.g. "local", "ecgate" or "cca"
     221
     222    @Return:
     223        <nothing>
     224    '''
    198225    ecd = c.ecmwfdatadir
    199226    template = ecd + 'python/compilejob.temp'

  • python/job.ksh

    refdb01a r991df6a  
    6464cat >$CONTROL<<EOF
    6565GATEWAY srvx8.img.univie.ac.at
    66 DESTINATION philipa8@genericSftp
     66DESTINATION annep@genericSftp
    6767accuracy 16
    6868addpar 186 187 188 235 139 39
    … …  
    7070cwc 0
    7171date_chunk 3
    72 debug 0
     72debug True
    7373dpdeta 1
    7474dtime 3

  • python/job.temp

    refdb01a r991df6a  
    6464cat >$CONTROL<<EOF
    6565GATEWAY srvx8.img.univie.ac.at
    66 DESTINATION philipa8@genericSftp
     66DESTINATION annep@genericSftp
    6767EOF
    6868cat >>$CONTROL<<EOF

  • python/joboper.ksh

    refdb01a r991df6a  
    6464cat >$CONTROL<<EOF
    6565GATEWAY srvx8.img.univie.ac.at
    66 DESTINATION philipa8@genericSftp
     66DESTINATION annep@genericSftp
    6767accuracy 16
    6868addpar 186 187 188 235 139 39
    … …  
    7070cwc 0
    7171date_chunk 3
    72 debug 0
     72debug True
    7373dpdeta 1
    7474dtime 3

  • python/plot_retrieved.py

    refdb01a r991df6a  
    11#!/usr/bin/env python
    2 # This software is licensed under the terms of the Apache Licence Version 2.0
    3 # which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
    4 #
    5 # Functionality provided: Simple tool for creating maps and time series of retrieved fields.
    6 #
    7 # Requirements:
    8 # in addition to a standard python 2.6 or 2.7 installation the following packages need to be installed
    9 # ECMWF WebMARS, gribAPI with python enabled, emoslib, ecaccess web toolkit, all available from https://software.ecmwf.int/
    10 # dateutils
    11 # matplotlib (optional, for debugging)
    12 
     2# -*- coding: utf-8 -*-
     3#************************************************************************
     4# TODO AP
     5# - documentation der Funktionen
     6# - docu der progam functionality
     7# - apply pep8
     8#************************************************************************
     9#*******************************************************************************
     10# @Author: Leopold Haimberger (University of Vienna)
     11#
     12# @Date: November 2015
     13#
     14# @Change History:
     15#
     16#    February 2018 - Anne Philipp (University of Vienna):
     17#        - applied PEP8 style guide
     18#        - added documentation
     19#        - created function main and moved the two function calls for
     20#          arguments and plotting into it
     21#
     22# @License:
     23#    (C) Copyright 2015-2018.
     24#
     25#    This software is licensed under the terms of the Apache Licence Version 2.0
     26#    which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
     27#
     28# @Program Functionality:
     29#    Simple tool for creating maps and time series of retrieved fields.
     30#
     31# @Program Content:
     32#    - plot_retrieved
     33#    - plottimeseries
     34#    - plotmap
     35#    - interpret_plotargs
     36#
     37#*******************************************************************************
     38
     39# ------------------------------------------------------------------------------
     40# MODULES
     41# ------------------------------------------------------------------------------
    1342import datetime
    1443import time
    15 import os,inspect,sys,glob
    16 import socket
    17 # add path to submit.py to pythonpath so that python finds its buddies
    18 localpythonpath=os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
    19 if localpythonpath not in sys.path:
    20     sys.path.append(localpythonpath)
     44import os
     45import inspect
     46import sys
     47import glob
     48from argparse import ArgumentParser, ArgumentDefaultsHelpFormatter
    2149
    2250from matplotlib.pylab import *
    2351import matplotlib.patches as mpatches
    24 from mpl_toolkits.basemap import Basemap,addcyclic
     52from mpl_toolkits.basemap import Basemap, addcyclic
    2553import matplotlib.colors as mcolors
    2654from matplotlib.font_manager import FontProperties
    … …  
    2856import matplotlib.cm as cmx
    2957import matplotlib.colors as colors
    30 from argparse import ArgumentParser,ArgumentDefaultsHelpFormatter
    31 
    32 from Tools import interpret_args_and_control, silentremove, product
    33 from Control import Control
     58#from rasotools.utils import stats
     59from gribapi import *
     60
     61# add path to pythonpath so that python finds its buddies
     62localpythonpath = os.path.dirname(os.path.abspath(
     63    inspect.getfile(inspect.currentframe())))
     64if localpythonpath not in sys.path:
     65    sys.path.append(localpythonpath)
     66
     67# software specific classes and modules from flex_extract
     68from Tools import silentremove, product
     69from ControlFile import ControlFile
    3470from GribTools import GribTools
    35 from gribapi import *
    36 from rasotools.utils import stats
    37 
    38 def plot_retrieved(args,c):
    39 
    40     start = datetime.datetime.strptime(c.start_date,'%Y%m%d%H')
    41     end = datetime.datetime.strptime(c.end_date,'%Y%m%d%H')
    42 
    43     c.paramIds=asarray(c.paramIds,dtype='int')
    44     c.levels=asarray(c.levels,dtype='int')
    45     c.area=asarray(c.area)
    46 
    47     index_keys=["date","time","step"]
    48     indexfile=c.inputdir+"/date_time_stepRange.idx"
     71
     72# ------------------------------------------------------------------------------
     73# FUNCTION
     74# ------------------------------------------------------------------------------
     75def main():
     76    '''
     77    @Description:
     78        If plot_retrieved is called from command line, this function controls
     79        the program flow and calls the argumentparser function and
     80        the plot_retrieved function for plotting the retrieved GRIB data.
     81
     82    @Input:
     83        <nothing>
     84
     85    @Return:
     86        <nothing>
     87    '''
     88    args, c = interpret_plotargs()
     89    plot_retrieved(args, c)
     90
     91    return
     92
     93def plot_retrieved(args, c):
     94    '''
     95    @Description:
     96        Reads GRIB data from a specified time period, a list of levels
     97        and a specified list of parameter.
     98
     99    @Input:
     100        args: instance of ArgumentParser
     101            Contains the commandline arguments from script/program call.
     102
     103        c: instance of class ControlFile
     104            Contains all necessary information of a CONTROL file. The parameters
     105            are: DAY1, DAY2, DTIME, MAXSTEP, TYPE, TIME, STEP, CLASS, STREAM,
     106            NUMBER, EXPVER, GRID, LEFT, LOWER, UPPER, RIGHT, LEVEL, LEVELIST,
     107            RESOL, GAUSS, ACCURACY, OMEGA, OMEGADIFF, ETA, ETADIFF, DPDETA,
     108            SMOOTH, FORMAT, ADDPAR, WRF, CWC, PREFIX, ECSTORAGE, ECTRANS,
     109            ECFSDIR, MAILOPS, MAILFAIL, GRIB2FLEXPART, DEBUG, INPUTDIR,
     110            OUTPUTDIR, FLEXPART_ROOT_SCRIPTS
     111            For more information about format and content of the parameter see
     112            documentation.
     113
     114    @Return:
     115        <nothing>
     116    '''
     117    start = datetime.datetime.strptime(c.start_date, '%Y%m%d%H')
     118    end = datetime.datetime.strptime(c.end_date, '%Y%m%d%H')
     119
     120    c.paramIds = asarray(c.paramIds, dtype='int')
     121    c.levels = asarray(c.levels, dtype='int')
     122    c.area = asarray(c.area)
     123
     124    index_keys = ["date", "time", "step"]
     125    indexfile = c.inputdir + "/date_time_stepRange.idx"
    49126    silentremove(indexfile)
    50     files=glob.glob(c.inputdir+'/'+c.prefix+'*')
    51     grib=GribTools(files)
    52     iid=grib.index(index_keys=index_keys, index_file = indexfile)
    53 
    54     gdict=dict(Ni = 360, Nj = 181,  iScansNegatively = 0,  jScansPositively = 0,
    55                jPointsAreConsecutive = 0,  alternativeRowScanning = 0,
    56                latitudeOfFirstGridPointInDegrees = 90,
    57                longitudeOfFirstGridPointInDegrees = 181,
    58                latitudeOfLastGridPointInDegrees = -90,
    59                longitudeOfLastGridPointInDegrees = 180,
    60                iDirectionIncrementInDegrees = 1,
    61                jDirectionIncrementInDegrees = 1
    62                )
     127    files = glob.glob(c.inputdir + '/' + c.prefix + '*')
     128    grib = GribTools(files)
     129    iid = grib.index(index_keys=index_keys, index_file = indexfile)
     130
     131    gdict = dict(Ni = 360, Nj = 181,
     132                iScansNegatively = 0,  jScansPositively = 0,
     133                jPointsAreConsecutive = 0,  alternativeRowScanning = 0,
     134                latitudeOfFirstGridPointInDegrees = 90,
     135                longitudeOfFirstGridPointInDegrees = 181,
     136                latitudeOfLastGridPointInDegrees = -90,
     137                longitudeOfLastGridPointInDegrees = 180,
     138                iDirectionIncrementInDegrees = 1,
     139                jDirectionIncrementInDegrees = 1
     140                )
    63141
    64142    index_vals = []
    65143    for key in index_keys:
    66         key_vals = grib_index_get(iid,key)
     144        key_vals = grib_index_get(iid, key)
    67145        print key_vals
    68146
    69147        index_vals.append(key_vals)
    70148
    71     fdict=dict()
    72     fmeta=dict()
    73     fstamp=dict()
     149    fdict = dict()
     150    fmeta = dict()
     151    fstamp = dict()
    74152    for p in c.paramIds:
    75153        for l in c.levels:
    76             key='{:0>3}_{:0>3}'.format(p,l)
    77             fdict[key]=[]
    78             fmeta[key]=[]
    79             fstamp[key]=[]
     154            key = '{:0>3}_{:0>3}'.format(p, l)
     155            fdict[key] = []
     156            fmeta[key] = []
     157            fstamp[key] = []
    80158    for prod in product(*index_vals):
    81159        for i in range(len(index_keys)):
    82             grib_index_select(iid,index_keys[i],prod[i])
     160            grib_index_select(iid, index_keys[i], prod[i])
    83161
    84162        gid = grib_new_from_index(iid)
    85 #        if gid is not None:
    86163
    87164        while(gid is not None):
    88165            date = grib_get(gid, 'date')
    89             fdate= datetime.datetime(date/10000,mod(date,10000)/100,mod(date,100))
     166            fdate = datetime.datetime(date/10000, mod(date,10000)/100,
     167                                      mod(date,100))
    90168            gtime = grib_get(gid, 'time')
    91169            step = grib_get(gid, 'step')
    92             fdatetime=fdate+datetime.timedelta(hours=gtime/100)
    93 #            if fdatetime<start or fdatetime>end:
    94 #                break
     170            fdatetime = fdate + datetime.timedelta(hours=gtime/100)
    95171            gtype = grib_get(gid, 'type')
    96172            paramId = grib_get(gid, 'paramId')
    97173            parameterName = grib_get(gid, 'parameterName')
    98             level=grib_get(gid,'level')
    99             if step>=c.start_step and step <=c.end_step and fdatetime>=start and fdatetime<=end and paramId in c.paramIds and level in c.levels:
    100                 key='{:0>3}_{:0>3}'.format(paramId,level)
     174            level = grib_get(gid, 'level')
     175            if step >= c.start_step and step <= c.end_step and \
     176               fdatetime >= start and fdatetime <= end and \
     177               paramId in c.paramIds and level in c.levels:
     178                key = '{:0>3}_{:0>3}'.format(paramId, level)
    101179                print key
    102                 fdatetimestep=fdatetime+datetime.timedelta(hours=step)
    103                 if len(fstamp)==0:
     180                fdatetimestep = fdatetime + datetime.timedelta(hours=step)
     181                if len(fstamp) == 0:
    104182                    fstamp[key].append(fdatetimestamp)
    105                     fmeta[key].append((paramId,parameterName,gtype,fdatetime,gtime,step,level))
    106                     fdict[key].append(flipud(reshape(grib_get_values(gid),[gdict['Nj'],gdict['Ni']])))
     183                    fmeta[key].append((paramId, parameterName, gtype,
     184                                       fdatetime, gtime, step, level))
     185                    fdict[key].append(flipud(reshape(
     186                            grib_get_values(gid), [gdict['Nj'], gdict['Ni']])))
    107187                else:
    108                     i=0
    109                     inserted=False
     188                    i = 0
     189                    inserted = False
    110190                    for i in range(len(fstamp[key])):
    111                         if fdatetimestep<fstamp[key][i]:
    112                             fstamp[key][i:i]=[fdatetimestep]
    113                             fmeta[key][i:i]=[(paramId,parameterName,gtype,fdatetime,gtime,step,level)]
    114                             fdict[key][i:i]=[flipud(reshape(grib_get_values(gid),[gdict['Nj'],gdict['Ni']]))]
    115                             inserted=True
     191                        if fdatetimestep < fstamp[key][i]:
     192                            fstamp[key][i:i] = [fdatetimestep]
     193                            fmeta[key][i:i] = [(paramId, parameterName, gtype,
     194                                                fdatetime, gtime, step, level)]
     195                            fdict[key][i:i] = [flipud(reshape(
     196                                                grib_get_values(gid),
     197                                                [gdict['Nj'], gdict['Ni']]))]
     198                            inserted = True
    116199                            break
    117200                    if not inserted:
    118201                        fstamp[key].append(fdatetimestep)
    119                         fmeta[key].append((paramId,parameterName,gtype,fdatetime,gtime,step,level))
    120                         fdict[key].append(flipud(reshape(grib_get_values(gid),[gdict['Nj'],gdict['Ni']])))
    121 
     202                        fmeta[key].append((paramId, parameterName, gtype,
     203                                           fdatetime, gtime, step, level))
     204                        fdict[key].append(flipud(reshape(
     205                            grib_get_values(gid), [gdict['Nj'], gdict['Ni']])))
    122206
    123207            grib_release(gid)
    … …  
    125209
    126210    for k in fdict.keys():
    127         fml=fmeta[k]
    128         fdl=fdict[k]
    129 
    130         for fd,fm in zip(fdl,fml):
    131             ftitle=fm[1]+' {} '.format(fm[-1])+datetime.datetime.strftime(fm[3],'%Y%m%d%H')+' '+stats(fd)
    132             pname='_'.join(fm[1].split())+'_{}_'.format(fm[-1])+datetime.datetime.strftime(fm[3],'%Y%m%d%H')+'.{:0>3}'.format(fm[5])
    133             plotmap(fd, fm,gdict,ftitle,pname+'.eps')
     211        fml = fmeta[k]
     212        fdl = fdict[k]
     213
     214        for fd, fm in zip(fdl, fml):
     215            ftitle = fm[1] + ' {} '.format(fm[-1]) + \
     216                datetime.datetime.strftime(fm[3], '%Y%m%d%H') #+ ' ' + stats(fd)
     217            pname = '_'.join(fm[1].split()) + '_{}_'.format(fm[-1]) + \
     218                datetime.datetime.strftime(fm[3], '%Y%m%d%H') + \
     219                '.{:0>3}'.format(fm[5])
     220            plotmap(fd, fm, gdict, ftitle, pname + '.eps')
    134221
    135222    for k in fdict.keys():
    136         fml=fmeta[k]
    137         fdl=fdict[k]
    138         fsl=fstamp[k]
     223        fml = fmeta[k]
     224        fdl = fdict[k]
     225        fsl = fstamp[k]
    139226        if fdl:
    140             fm=fml[0]
    141             fd=fdl[0]
    142             ftitle=fm[1]+' {} '.format(fm[-1])+datetime.datetime.strftime(fm[3],'%Y%m%d%H')+' '+stats(fd)
    143             pname='_'.join(fm[1].split())+'_{}_'.format(fm[-1])+datetime.datetime.strftime(fm[3],'%Y%m%d%H')+'.{:0>3}'.format(fm[5])
    144             lat=-20
    145             lon=20
    146             plottimeseries(fdl,fml,fsl,lat,lon, gdict, ftitle, pname+'.eps')
    147 
    148 def plottimeseries(flist,fmetalist,ftimestamps,lat,lon,gdict,ftitle,filename):
    149     t1=time.time()
    150     latindex=(lat+90)*180/(gdict['Nj']-1)
    151     lonindex=(lon+179)*360/gdict['Ni']
    152     farr=asarray(flist)
    153     ts=farr[:,latindex,lonindex]
    154     f=plt.figure(figsize=(12,6.7))
    155     plt.plot(ftimestamps,ts)
     227            fm = fml[0]
     228            fd = fdl[0]
     229            ftitle = fm[1] + ' {} '.format(fm[-1]) + \
     230                datetime.datetime.strftime(fm[3], '%Y%m%d%H') #+ ' ' + stats(fd)
     231            pname = '_'.join(fm[1].split()) + '_{}_'.format(fm[-1]) + \
     232                datetime.datetime.strftime(fm[3], '%Y%m%d%H') + \
     233                '.{:0>3}'.format(fm[5])
     234            lat = -20
     235            lon = 20
     236            plottimeseries(fdl, fml, fsl, lat, lon,
     237                           gdict, ftitle, pname + '.eps')
     238
     239    return
     240
     241def plottimeseries(flist, fmetalist, ftimestamps, lat, lon,
     242                   gdict, ftitle, filename):
     243    '''
     244    @Description:
     245
     246    @Input:
     247        flist:
     248            The actual data values to be plotted from the grib messages.
     249
     250        fmetalist: list of strings
     251            Contains some meta date for the data field to be plotted:
     252            parameter id, parameter Name, grid type, date and time,
     253            time, forecast step, level
     254
     255        ftimestamps: list of datetime
     256            Contains the time stamps in a datetime format, e.g.
     257
     258        lat:
     259
     260        lon:
     261
     262        gdict:
     263
     264        ftitle: string
     265            The title of the timeseries.
     266
     267        filename: string
     268            The time series is stored in a file with this name.
     269
     270    @Return:
     271        <nothing>
     272    '''
     273    t1 = time.time()
     274    latindex = (lat + 90) * 180 / (gdict['Nj'] - 1)
     275    lonindex = (lon + 179) * 360 / gdict['Ni']
     276    farr = asarray(flist)
     277    ts = farr[:, latindex, lonindex]
     278    f = plt.figure(figsize=(12,6.7))
     279    plt.plot(ftimestamps, ts)
    156280    plt.title(ftitle)
    157     savefig(c.outputdir+'/'+filename)
    158     print 'created ',c.outputdir+'/'+filename
     281    savefig(c.outputdir + '/' + filename)
     282    print 'created ', c.outputdir + '/' + filename
    159283    plt.close(f)
    160     print time.time()-t1,'s'
    161 
    162 def plotmap(flist,fmetalist,gdict,ftitle,filename):
    163     t1=time.time()
    164     f=plt.figure(figsize=(12,6.7))
     284    print time.time() - t1, 's'
     285
     286    return
     287
     288def plotmap(flist, fmetalist, gdict, ftitle, filename):
     289    '''
     290    @Description:
     291
     292    @Input:
     293        flist
     294        fmetalist
     295        gdict
     296        ftitle
     297        filename
     298
     299    @Return:
     300        <nothing>
     301    '''
     302    t1 = time.time()
     303    f = plt.figure(figsize=(12, 6.7))
    165304    mbaxes = f.add_axes([0.05, 0.15, 0.8, 0.7])
    166     m =Basemap(llcrnrlon=-180.,llcrnrlat=-90.,urcrnrlon=180,urcrnrlat=90.)
     305    m = Basemap(llcrnrlon=-180., llcrnrlat=-90., urcrnrlon=180, urcrnrlat=90.)
    167306    #if bw==0 :
    168307        #fill_color=rgb(0.6,0.8,1)
    … …  
    170309        #fill_color=rgb(0.85,0.85,0.85)
    171310
    172     lw=0.3
     311    lw = 0.3
    173312    m.drawmapboundary()
    174     parallels = arange(-90.,91,90.)
    175     # labels = [left,right,top,bottom]
    176     m.drawparallels(parallels,labels=[True,True,True,True],linewidth=lw)
    177     meridians = arange(-180.,181.,60.)
    178     m.drawmeridians(meridians,labels=[True,True,True,True],linewidth=lw)
     313    parallels = arange(-90., 91, 90.)
     314    # labels = [left, right, top, bottom]
     315    m.drawparallels(parallels, labels=[True, True, True, True], linewidth=lw)
     316    meridians = arange(-180., 181., 60.)
     317    m.drawmeridians(meridians, labels=[True, True, True, True], linewidth=lw)
    179318    m.drawcoastlines(linewidth=lw)
    180     xleft=gdict['longitudeOfFirstGridPointInDegrees']
    181     if xleft>180.0:
    182         xleft-=360.
    183     x=linspace(xleft,gdict['longitudeOfLastGridPointInDegrees'],gdict['Ni'])
    184     y=linspace(gdict['latitudeOfLastGridPointInDegrees'],gdict['latitudeOfFirstGridPointInDegrees'],gdict['Nj'])
    185     xx, yy = m(*meshgrid(x,y))
    186 
    187     s=m.contourf(xx,yy, flist)
    188     title(ftitle,y=1.1)
     319    xleft = gdict['longitudeOfFirstGridPointInDegrees']
     320    if xleft > 180.0:
     321        xleft -= 360.
     322    x = linspace(xleft, gdict['longitudeOfLastGridPointInDegrees'], gdict['Ni'])
     323    y = linspace(gdict['latitudeOfLastGridPointInDegrees'],
     324                 gdict['latitudeOfFirstGridPointInDegrees'], gdict['Nj'])
     325    xx, yy = m(*meshgrid(x, y))
     326
     327    s = m.contourf(xx, yy, flist)
     328    title(ftitle, y=1.1)
    189329    cbaxes = f.add_axes([0.9, 0.2, 0.04, 0.6])
    190     cb=colorbar(cax=cbaxes)
    191 
    192     savefig(c.outputdir+'/'+filename)
    193     print 'created ',c.outputdir+'/'+filename
     330    cb = colorbar(cax=cbaxes)
     331
     332    savefig(c.outputdir + '/' + filename)
     333    print 'created ', c.outputdir + '/' + filename
    194334    plt.close(f)
    195     print time.time()-t1,'s'
    196 
    197 
    198 def interpret_plotargs(*args,**kwargs):
    199 
    200     parser = ArgumentParser(description='Retrieve FLEXPART input from ECMWF MARS archive',
     335    print time.time() - t1, 's'
     336
     337    return
     338
     339def interpret_plotargs():
     340    '''
     341    @Description:
     342        Assigns the command line arguments and reads CONTROL file
     343        content. Apply default values for non mentioned arguments.
     344
     345    @Input:
     346        <nothing>
     347
     348    @Return:
     349        args: instance of ArgumentParser
     350            Contains the commandline arguments from script/program call.
     351
     352        c: instance of class ControlFile
     353            Contains all necessary information of a CONTROL file. The parameters
     354            are: DAY1, DAY2, DTIME, MAXSTEP, TYPE, TIME, STEP, CLASS, STREAM,
     355            NUMBER, EXPVER, GRID, LEFT, LOWER, UPPER, RIGHT, LEVEL, LEVELIST,
     356            RESOL, GAUSS, ACCURACY, OMEGA, OMEGADIFF, ETA, ETADIFF, DPDETA,
     357            SMOOTH, FORMAT, ADDPAR, WRF, CWC, PREFIX, ECSTORAGE, ECTRANS,
     358            ECFSDIR, MAILOPS, MAILFAIL, GRIB2FLEXPART, DEBUG, INPUTDIR,
     359            OUTPUTDIR, FLEXPART_ROOT_SCRIPTS
     360            For more information about format and content of the parameter see
     361            documentation.
     362    '''
     363    parser = ArgumentParser(description='Retrieve FLEXPART input from ' + \
     364                            'ECMWF MARS archive',
    201365                            formatter_class=ArgumentDefaultsHelpFormatter)
    202366
    … …  
    212376                         help="end_step in hours")
    213377
    214 # some arguments that override the default in the control file
    215     parser.add_argument("--levelist", dest="levelist",help="Vertical levels to be retrieved, e.g. 30/to/60")
    216     parser.add_argument("--area", dest="area", help="area defined as north/west/south/east")
    217     parser.add_argument("--paramIds", dest="paramIds", help="parameter IDs")
    218     parser.add_argument("--prefix", dest="prefix",default='EN', help="output file name prefix")
     378# some arguments that override the default in the CONTROL file
     379    parser.add_argument("--levelist", dest="levelist",
     380                        help="Vertical levels to be retrieved, e.g. 30/to/60")
     381    parser.add_argument("--area", dest="area",
     382                        help="area defined as north/west/south/east")
     383    parser.add_argument("--paramIds", dest="paramIds",
     384                        help="parameter IDs")
     385    parser.add_argument("--prefix", dest="prefix", default='EN',
     386                        help="output file name prefix")
    219387
    220388# set the working directories
    221     parser.add_argument("--inputdir", dest="inputdir",default=None,
     389    parser.add_argument("--inputdir", dest="inputdir", default=None,
    222390                        help="root directory for storing intermediate files")
    223     parser.add_argument("--outputdir", dest="outputdir",default=None,
     391    parser.add_argument("--outputdir", dest="outputdir", default=None,
    224392                        help="root directory for storing output files")
    225393    parser.add_argument("--flexpart_root_scripts", dest="flexpart_root_scripts",
    226                         help="FLEXPART root directory (to find grib2flexpart and COMMAND file)\n\
    227                               Normally ECMWFDATA resides in the scripts directory of the FLEXPART distribution")
    228 
    229     parser.add_argument("--controlfile", dest="controlfile",default='CONTROL.temp',
    230                         help="file with control parameters")
     394                        help="FLEXPART root directory (to find \
     395                        'grib2flexpart and COMMAND file)\n \
     396                        Normally ECMWFDATA resides in the scripts directory \
     397                        of the FLEXPART distribution")
     398
     399    parser.add_argument("--controlfile", dest="controlfile",
     400                        default='CONTROL.temp', help="file with CONTROL parameters")
    231401    args = parser.parse_args()
    232402
    233403    try:
    234         c=Control(args.controlfile)
     404        c = ControlFile(args.controlfile)
    235405    except IOError:
    236406        try:
    237             c=Control(localpythonpath+args.controlfile)
     407            c = ControlFile(localpythonpath + args.controlfile)
    238408
    239409        except:
    240             print 'Could not read control file "'+args.controlfile+'"'
     410            print 'Could not read CONTROL file "' + args.controlfile + '"'
    241411            print 'Either it does not exist or its syntax is wrong.'
    242             print 'Try "'+sys.argv[0].split('/')[-1]+' -h" to print usage information'
     412            print 'Try "' + sys.argv[0].split('/')[-1] + \
     413                  ' -h" to print usage information'
    243414            exit(1)
    244415
    245416    if args.levelist:
    246         c.levels=args.levelist.split('/')
     417        c.levels = args.levelist.split('/')
    247418    else:
    248         c.levels=[0]
     419        c.levels = [0]
    249420    if args.area:
    250         c.area=args.area.split('/')
     421        c.area = args.area.split('/')
    251422    else:
    252         c.area='[0,0]'
    253 
    254     c.paramIds=args.paramIds.split('/')
     423        c.area = '[0,0]'
     424
     425    c.paramIds = args.paramIds.split('/')
    255426    if args.start_step:
    256         c.start_step=int(args.start_step)
     427        c.start_step = int(args.start_step)
    257428    else:
    258         c.start_step=0
     429        c.start_step = 0
    259430    if args.end_step:
    260         c.end_step=int(args.end_step)
     431        c.end_step = int(args.end_step)
    261432    else:
    262         c.end_step=0
    263 
    264     c.start_date=args.start_date
    265     c.end_date=args.end_date
    266     c.prefix=args.prefix
    267     c.inputdir=args.inputdir
     433        c.end_step = 0
     434
     435    c.start_date = args.start_date
     436    c.end_date = args.end_date
     437    c.prefix = args.prefix
     438    c.inputdir = args.inputdir
    268439    if args.outputdir:
    269         c.outputdir=args.outputdir
     440        c.outputdir = args.outputdir
    270441    else:
    271         c.outputdir=c.inputdir
    272 
    273         return args,c
     442        c.outputdir = c.inputdir
     443
     444    return args, c
    274445
    275446if __name__ == "__main__":
    276 
    277     args,c=interpret_plotargs()
    278     plot_retrieved(args,c)
     447    main()
     448

  • python/prepareFLEXPART.py

    refdb01a r991df6a  
    33#************************************************************************
    44# TODO AP
    5 #AP
    6 # - Change History ist nicht angepasst ans File! Original geben lassen
    75# - wieso cleanup in main wenn es in prepareflexpart bereits abgefragt wurde?
    86#   doppelt gemoppelt?
    … …  
    108#   relevant sind? verstehe ich nicht
    119#************************************************************************
    12 """
    13 @Author: Anne Fouilloux (University of Oslo)
    14 
    15 @Date: October 2014
    16 
    17 @ChangeHistory:
    18     November 2015 - Leopold Haimberger (University of Vienna):
    19         - using the WebAPI also for general MARS retrievals
    20         - job submission on ecgate and cca
    21         - job templates suitable for twice daily operational dissemination
    22         - dividing retrievals of longer periods into digestable chunks
    23         - retrieve also longer term forecasts, not only analyses and
    24           short term forecast data
    25         - conversion into GRIB2
    26         - conversion into .fp format for faster execution of FLEXPART
    27 
    28     February 2018 - Anne Philipp (University of Vienna):
    29         - applied PEP8 style guide
    30         - added documentation
    31         - minor changes in programming style for consistence
    32         - BUG: removed call of cleanup-Function after call of prepareFlexpart
    33                 since it is already called in prepareFlexpart at the end!
    34 
    35 @License:
    36     (C) Copyright 2014-2018.
    37 
    38     This software is licensed under the terms of the Apache Licence Version 2.0
    39     which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
    40 
    41 @Requirements:
    42     - A standard python 2.6 or 2.7 installation
    43     - dateutils
    44     - ECMWF specific packages, all available from https://software.ecmwf.int/
    45         ECMWF WebMARS, gribAPI with python enabled, emoslib and
    46         ecaccess web toolkit
    47 
    48 @Description:
    49     Further documentation may be obtained from www.flexpart.eu.
    50 
    51     Functionality provided:
    52         Prepare input 3D-wind fields in hybrid coordinates +
    53         surface fields for FLEXPART runs
    54 """
     10#*******************************************************************************
     11# @Author: Anne Fouilloux (University of Oslo)
     12#
     13# @Date: October 2014
     14#
     15# @Change History:
     16#
     17#    November 2015 - Leopold Haimberger (University of Vienna):
     18#        - using the WebAPI also for general MARS retrievals
     19#        - job submission on ecgate and cca
     20#        - job templates suitable for twice daily operational dissemination
     21#        - dividing retrievals of longer periods into digestable chunks
     22#        - retrieve also longer term forecasts, not only analyses and
     23#          short term forecast data
     24#        - conversion into GRIB2
     25#        - conversion into .fp format for faster execution of FLEXPART
     26#
     27#    February 2018 - Anne Philipp (University of Vienna):
     28#        - applied PEP8 style guide
     29#        - added documentation
     30#        - minor changes in programming style for consistence
     31#        - BUG: removed call of cleanup-Function after call of
     32#               prepareFlexpart in main since it is already called in
     33#               prepareFlexpart at the end!
     34#        - created function main and moved the two function calls for
     35#          arguments and prepareFLEXPART into it
     36#
     37# @License:
     38#    (C) Copyright 2014-2018.
     39#
     40#    This software is licensed under the terms of the Apache Licence Version 2.0
     41#    which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
     42#
     43# @Program Functionality:
     44#    This program prepares the final version of the grib files which are
     45#    then used by FLEXPART. It converts the bunch of grib files extracted
     46#    via getMARSdata by doing for example the necessary conversion to get
     47#    consistent grids or the disaggregation of flux data. Finally, the
     48#    program combines the data fields in files per available hour with the
     49#    naming convention xxYYMMDDHH, where xx should be 2 arbitrary letters
     50#    (mostly xx is chosen to be "EN").
     51#
     52# @Program Content:
     53#    - main
     54#    - prepareFLEXPART
     55#
     56#*******************************************************************************
     57
    5558# ------------------------------------------------------------------------------
    5659# MODULES
    5760# ------------------------------------------------------------------------------
    58 import calendar
    5961import shutil
    6062import datetime
    61 import time
     63#import time
    6264import os
    6365import inspect
    … …  
    6567import socket
    6668from argparse import ArgumentParser, ArgumentDefaultsHelpFormatter
    67 import UIOFiles
    68 import Control
    69 import Tools
    70 import ECFlexpart
    7169
    7270hostname = socket.gethostname()
    … …  
    7876    ecapi = False
    7977
    80 # add path to submit.py to pythonpath so that python finds its buddies
    81 localpythonpath=os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
     78# add path to pythonpath so that python finds its buddies
     79localpythonpath = os.path.dirname(os.path.abspath(
     80    inspect.getfile(inspect.currentframe())))
    8281if localpythonpath not in sys.path:
    8382    sys.path.append(localpythonpath)
     83
     84# software specific classes and modules from flex_extract
     85from UIOFiles import UIOFiles
     86from Tools import interpret_args_and_control, cleanup
     87from ECFlexpart import ECFlexpart
    8488# ------------------------------------------------------------------------------
    8589# FUNCTION
    8690# ------------------------------------------------------------------------------
     91def main():
     92    '''
     93    @Description:
     94        If prepareFLEXPART is called from command line, this function controls
     95        the program flow and calls the argumentparser function and
     96        the prepareFLEXPART function for preparation of GRIB data for FLEXPART.
     97
     98    @Input:
     99        <nothing>
     100
     101    @Return:
     102        <nothing>
     103    '''
     104    args, c = interpret_args_and_control()
     105    prepareFLEXPART(args, c)
     106
     107    return
     108
    87109def prepareFLEXPART(args, c):
    88110    '''
    89111    @Description:
    90 
     112        Lists all grib files retrieved from MARS with getMARSdata and
     113        uses prepares data for the use in FLEXPART. Specific data fields
     114        are converted to a different grid and the flux data are going to be
     115        disaggregated. The data fields are collected by hour and stored in
     116        a file with a specific FLEXPART relevant naming convention.
    91117
    92118    @Input:
    … …  
    94120            Contains the commandline arguments from script/program call.
    95121
    96         c: instance of class Control
    97             Contains all the parameters of control files, which are e.g.:
     122        c: instance of class ControlFile
     123            Contains all the parameters of CONTROL file, which are e.g.:
    98124            DAY1(start_date), DAY2(end_date), DTIME, MAXSTEP, TYPE, TIME,
    99125            STEP, CLASS(marsclass), STREAM, NUMBER, EXPVER, GRID, LEFT,
    … …  
    134160
    135161    # get all files with flux data to be deaccumulated
    136     inputfiles = UIOFiles.UIOFiles(['.grib', '.grb', '.grib1',
    137                            '.grib2', '.grb1', '.grb2'])
    138 
    139     inputfiles.listFiles(c.inputdir, '*OG_acc_SL*.' + c.ppid + '.*')
     162    inputfiles = UIOFiles('*OG_acc_SL*.' + c.ppid + '.*')
     163    inputfiles.listFiles(c.inputdir)
    140164
    141165    # create output dir if necessary
    … …  
    144168
    145169    # deaccumulate the flux data
    146     flexpart = ECFlexpart.ECFlexpart(c, fluxes=True)
     170    flexpart = ECFlexpart(c, fluxes=True)
    147171    flexpart.write_namelist(c, 'fort.4')
    148172    flexpart.deacc_fluxes(inputfiles, c)
    … …  
    152176
    153177    # get a list of all files from the root inputdir
    154     inputfiles = UIOFiles.UIOFiles(['.grib', '.grb', '.grib1',
    155                            '.grib2', '.grb1', '.grb2'])
    156 
    157     inputfiles.listFiles(c.inputdir, '????__??.*' + c.ppid + '.*')
     178    inputfiles = UIOFiles('????__??.*' + c.ppid + '.*')
     179    inputfiles.listFiles(c.inputdir)
    158180
    159181    # produce FLEXPART-ready GRIB files and
    … …  
    163185        start = startm1
    164186
    165     flexpart = ECFlexpart.ECFlexpart(c, fluxes=False)
     187    flexpart = ECFlexpart(c, fluxes=False)
    166188    flexpart.create(inputfiles, c)
    167189    flexpart.process_output(c)
    … …  
    172194        print('Temporary files left intact')
    173195    else:
    174         Tools.cleanup(c)
     196        cleanup(c)
    175197
    176198    return
    177199
    178200if __name__ == "__main__":
    179     args, c = Tools.interpret_args_and_control()
    180     prepareFLEXPART(args, c)
     201    main()

  • python/profiling.py

    refdb01a r991df6a  
    33#************************************************************************
    44# TODO AP
    5 # - add description of file
    65# - check of license of book content
    76#************************************************************************
    8 """
    9 @Author: Anne Philipp (University of Vienna)
     7#*******************************************************************************
     8#
     9# @Author: Anne Philipp (University of Vienna)
     10#
     11# @Date: March 2018
     12#
     13# @License:
     14#    (C) Copyright 2018.
     15#
     16#    This software is licensed under the terms of the Apache Licence Version 2.0
     17#    which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
     18#
     19# @Program functionality:
     20#    This module is not part of flex_extract. It is just used for testing and
     21#    performance analysis of some functions.
     22#
     23# @Program Content:
     24#    - timefn
     25#
     26#*******************************************************************************
    1027
    11 @Date: March 2018
    12 
    13 @License:
    14     (C) Copyright 2018.
    15 
    16     This software is licensed under the terms of the Apache Licence Version 2.0
    17     which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
    18 
    19 @Requirements:
    20     A standard python 2.6 or 2.7 installation
    21 
    22 @Description:
    23     ...
    24 """
    2528# ------------------------------------------------------------------------------
    2629# MODULES

  • python/submit.py

    refdb01a r991df6a  
    77# - dead code ? what to do?
    88# - seperate operational and reanlysis for clarification
    9 # - add correct file description
    109# - divide in two submits , ondemand und operational
    1110# -
    1211#************************************************************************
    13 """
    14 @Author: Anne Fouilloux (University of Oslo)
    1512
    16 @Date: October 2014
     13#*******************************************************************************
     14# @Author: Anne Fouilloux (University of Oslo)
     15#
     16# @Date: October 2014
     17#
     18# @Change History:
     19#
     20#    November 2015 - Leopold Haimberger (University of Vienna):
     21#        - job submission on ecgate and cca
     22#        - job templates suitable for twice daily operational dissemination
     23#
     24#    February 2018 - Anne Philipp (University of Vienna):
     25#        - applied PEP8 style guide
     26#        - added documentation
     27#        - minor changes in programming style (for consistence)
     28#
     29# @License:
     30#    (C) Copyright 2014-2018.
     31#
     32#    This software is licensed under the terms of the Apache Licence Version 2.0
     33#    which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
     34#
     35# @Program Functionality:
     36#    This program is the main program of flex_extract and controls the
     37#    program flow.
     38#    If it is supposed to work locally then it works through the necessary
     39#    functions getMARSdata and prepareFlexpart. Otherwise it prepares
     40#    a shell job script which will do the necessary work on the
     41#    ECMWF server and is submitted via ecaccess-job-submit.
     42#
     43# @Program Content:
     44#    - main
     45#    - submit
     46#
     47#*******************************************************************************
    1748
    18 @ChangeHistory:
    19     November 2015 - Leopold Haimberger (University of Vienna):
    20         - job submission on ecgate and cca
    21         - job templates suitable for twice daily operational dissemination
    22 
    23     February 2018 - Anne Philipp (University of Vienna):
    24         - applied PEP8 style guide
    25         - added documentation
    26         - minor changes in programming style (for consistence)
    27 
    28 @License:
    29     (C) Copyright 2014-2018.
    30 
    31     This software is licensed under the terms of the Apache Licence Version 2.0
    32     which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
    33 
    34 @Requirements:
    35     - A standard python 2.6 or 2.7 installation
    36 
    37 @Description:
    38     Further documentation may be obtained from www.flexpart.eu.
    39 
    40 
    41 """
    4249# ------------------------------------------------------------------------------
    4350# MODULES
    … …  
    4855import subprocess
    4956import inspect
    50 # add the pythondir path so that python finds its buddies (from flex_extract)
    51 localpythonpath = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
    52 sys.path.append(localpythonpath)
     57# add path to pythonpath so that python finds its buddies
     58localpythonpath = os.path.dirname(os.path.abspath(
     59    inspect.getfile(inspect.currentframe())))
     60if localpythonpath not in sys.path:
     61    sys.path.append(localpythonpath)
    5362
    5463# software specific classes and modules from flex_extract
    … …  
    5665from getMARSdata import getMARSdata
    5766from prepareFLEXPART import prepareFLEXPART
     67
    5868# ------------------------------------------------------------------------------
    5969# FUNCTIONS
    6070# ------------------------------------------------------------------------------
     71
    6172def main():
    6273    '''
    6374    @Description:
    64         Get the arguments from script call and from Control file.
     75        Get the arguments from script call and from CONTROL file.
    6576        Decides from the argument "queue" if the local version
    6677        is done "queue=None" or the gateway version with "queue=ecgate"
    … …  
    102113            Default is "job.temp".
    103114
    104         c: instance of class Control
    105             Contains all the parameters of control files, which are e.g.:
     115        c: instance of class ControlFile
     116            Contains all the parameters of CONTROL file, which are e.g.:
    106117            DAY1(start_date), DAY2(end_date), DTIME, MAXSTEP, TYPE, TIME,
    107118            STEP, CLASS(marsclass), STREAM, NUMBER, EXPVER, GRID, LEFT,
    … …  
    127138        insert_point = lftext.index('EOF')
    128139
    129     # put all parameters of control instance into a list
     140    # put all parameters of ControlFile instance into a list
    130141    clist = c.tolist()
    131142    colist = []  # operational

  • python/testsuite.py

    refdb01a r991df6a  
    11#!/usr/bin/env python
     2# -*- coding: utf-8 -*-
     3#************************************************************************
     4# TODO AP
     5#
     6# - provide more tests
     7# - provide more documentation
     8# -
     9#************************************************************************
    210
    3 # This software is licensed under the terms of the Apache Licence Version 2.0
    4 # which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
     11#*******************************************************************************
     12# @Author: Leopold Haimberger (University of Vienna)
    513#
    6 # Leopold Haimberger, Dec 2015
     14# @Date: December 2015
    715#
    8 # Functionality provided: This script triggers the ECMWFDATA test suite. Call with
    9 # testsuite.py [test group]
     16# @Change History:
    1017#
     18#    February 2018 - Anne Philipp (University of Vienna):
     19#        - applied PEP8 style guide
     20#        - added documentation
    1121#
    12 # Further documentation may be obtained from www.flexpart.eu
     22# @License:
     23#    (C) Copyright 2015-2018.
    1324#
    14 # Test groups are specified in testsuite.json
    15 # in addition to a standard python 2.6 or 2.7 installation the following packages need to be installed
    16 # ECMWF WebMARS, gribAPI with python enabled, emoslib, ecaccess web toolkit, all available from https://software.ecmwf.int/
    17 # dateutils
    18 # matplotlib (optional, for debugging)
     25#    This software is licensed under the terms of the Apache Licence Version 2.0
     26#    which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
     27#
     28# @Program Functionality:
     29#    This script triggers the ECMWFDATA test suite. Call with
     30#    testsuite.py [test group]
     31#
     32# @Program Content:
     33#
     34#*******************************************************************************
    1935
    20 import os,sys
     36# ------------------------------------------------------------------------------
     37# MODULES
     38# ------------------------------------------------------------------------------
     39import os
     40import sys
    2141import json
    2242import subprocess
    2343
     44# ------------------------------------------------------------------------------
     45# PROGRAM
     46# ------------------------------------------------------------------------------
    2447try:
    25     taskfile=open('testsuite.json')
     48    taskfile = open('testsuite.json')
    2649except:
    2750    print 'could not open suite definition file testsuite.json'
    … …  
    3356    exit()
    3457
    35 fprs=os.getenv('FLEXPART_ROOT_SCRIPTS')
     58fprs = os.getenv('FLEXPART_ROOT_SCRIPTS')
    3659if fprs is None:
    3760    print 'FLEXPART_ROOT_SCRIPTS not set .. some test jobs may fail'
    3861
    39 tasks=json.load(taskfile,encoding='latin-1')
     62tasks = json.load(taskfile, encoding='latin-1')
    4063taskfile.close()
    4164if not os.path.exists('../test'):
    4265    os.makedirs('../test')
    43 if len(sys.argv)>1:
    44     groups=sys.argv[1:]
     66if len(sys.argv) > 1:
     67    groups = sys.argv[1:]
    4568else:
    46     groups=['xinstall','default','ops','work','cv','fc']#,'hires']
    47 jobcounter=0
    48 jobfailed=0
     69    groups = ['xinstall', 'default', 'ops', 'work', 'cv', 'fc']#,'hires']
     70jobcounter = 0
     71jobfailed = 0
    4972for g in groups:
    5073    try:
    51         tk,tv=g,tasks[g]
     74        tk, tv = g, tasks[g]
    5275    except:
    5376        continue
    54     garglist=[]
    55     for ttk,ttv in tv.iteritems():
    56         if isinstance(ttv,basestring):
    57             if ttk!='script':
    58                 garglist.append('--'+ttk)
    59                 if '$'==ttv[0]:
     77    garglist = []
     78    for ttk, ttv in tv.iteritems():
     79        if isinstance(ttv, basestring):
     80            if ttk != 'script':
     81                garglist.append('--' + ttk)
     82                if '$' == ttv[0]:
    6083                    garglist.append(os.path.expandvars(ttv))
    6184                else:
    6285                    garglist.append(ttv)
    63     for ttk,ttv in tv.iteritems():
    64         if isinstance(ttv,dict):
    65             arglist=[]
    66             for tttk,tttv in ttv.iteritems():
    67                 if isinstance(tttv,basestring):
    68                         arglist.append('--'+tttk)
     86    for ttk, ttv in tv.iteritems():
     87        if isinstance(ttv, dict):
     88            arglist = []
     89            for tttk, tttv in ttv.iteritems():
     90                if isinstance(tttv, basestring):
     91                        arglist.append('--' + tttk)
    6992                        if '$' in tttv[0]:
    7093                            arglist.append(os.path.expandvars(tttv))
    7194                        else:
    7295                            arglist.append(tttv)
    73             print 'Command: ',' '.join([tv['script']]+garglist+arglist)
    74             o='../test/'+tk+'_'+ttk+'_'+'_'.join(ttv.keys())
    75             print 'Output will be sent to ',o
    76             f=open(o,'w')
     96            print 'Command: ', ' '.join([tv['script']] + garglist + arglist)
     97            o = '../test/' + tk + '_' + ttk + '_' + '_'.join(ttv.keys())
     98            print 'Output will be sent to ', o
     99            f = open(o, 'w')
    77100            try:
    78                 p=subprocess.check_call([tv['script']]+garglist+arglist,stdout=f,stderr=f)
     101                p = subprocess.check_call([tv['script']] + garglist + arglist,
     102                                          stdout=f, stderr=f)
    79103            except:
    80104                f.write('\nFAILED\n')
    81105                print 'FAILED'
    82                 jobfailed+=1
    83             jobcounter+=1
     106                jobfailed += 1
     107            jobcounter += 1
    84108            f.close()
    85109
    86110print 'Test suite tasks completed'
    87 print str(jobcounter-jobfailed)+' successful, '+str(jobfailed)+' failed'
     111print str(jobcounter-jobfailed) + ' successful, ' + str(jobfailed) + ' failed'
    88112print 'If tasks have been submitted via ECACCESS please check emails'
    89113

  • src/Makefile.local.gfortran

    rd69b677 r991df6a  
    1212
    1313
    14 GRIB_API_INCLUDE_DIR=/usr/local/gcc-4.9.3/grib1.12.3//include
    15 GRIB_API_LIB= -L/usr/local/gcc-4.9.3/grib1.12.3/lib -Bstatic  -lgrib_api_f77 -lgrib_api_f90 -lgrib_api -Bdynamic  -lm  -ljasper
     14GRIB_API_INCLUDE_DIR=/usr/local/gcc-4.9.3/grib_api-1.14.3//include
     15GRIB_API_LIB= -L/usr/local/gcc-4.9.3/grib_api-1.14.3/lib -Bstatic  -lgrib_api_f77 -lgrib_api_f90 -lgrib_api -Bdynamic  -lm  -ljasper
    1616EMOSLIB=-lemosR64
    1717

  • src/Makefile.local.ifort

    refdb01a r991df6a  
    1212
    1313
    14 GRIB_API_INCLUDE_DIR=/usr/local/ifort/grib1.12.3//include
    15 GRIB_API_LIB=-openmp -L/usr/local/ifort/grib1.12.3//lib -Bstatic  -lgrib_api_f77 -lgrib_api_f90 -lgrib_api -Bdynamic  -lm  -ljasper #-lopenjpeg
     14GRIB_API_INCLUDE_DIR=/home/srvx1/tmc/TestEnv/Libraries/eccodes-2.6.0_ifort/include
     15GRIB_API_LIB= -L/home/srvx1/tmc/TestEnv/Libraries/eccodes-2.6.0_ifort/lib -Bstatic  -leccodes_f90  -leccodes -Bdynamic  -lm  -ljasper
    1616
    17 OPT     = -g
     17OPT     = -g -O3 -mcmodel=medium -unroll -inline -heap-arrays 32
    1818DEBUG   = -g
    1919LIB     =  $(GRIB_API_LIB) -lemosR64 -lgfortran
    2020
    21 FC=ifort -132  -traceback -r8
    22 F90C=ifort -132  -traceback -r8
     21FC=/opt/intel/bin/ifort -132  -traceback -r8
     22F90C=/opt/intel/bin/ifort -132  -traceback -r8
    2323
    2424FFLAGS =  $(OPT) -I. -I$(GRIB_API_INCLUDE_DIR)
    … …  
    4343
    4444phgrreal.o: phgrreal.f
    45         $(F90C) -c -g -O3 -fopenmp phgrreal.f
     45        $(F90C) -c -g -O3 phgrreal.f
    4646
    4747grphreal.o: grphreal.f
    48         $(F90C) -c -g -O3 -fopenmp grphreal.f
     48        $(F90C) -c -g -O3 grphreal.f
    4949
    5050ftrafo.o: ftrafo.f
    51         $(F90C) -c -g -O3 -fopenmp ftrafo.f
     51        $(F90C) -c -g -O3 ftrafo.f
    5252
    5353$(BINDIR)/CONVERT2:     phgrreal.o grphreal.o ftrafo.o rwGRIB2.o  posnam.o preconvert.o
Note: See TracChangeset for help on using the changeset viewer.
hosted by ZAMG