Changeset efdb01a in flex_extract.git for python


Ignore:
Timestamp:
May 9, 2018, 12:15:00 PM (6 years ago)
Author:
Anne Philipp <anne.philipp@…>
Branches:
master, ctbto, dev
Children:
991df6a
Parents:
02c8c50
Message:

whole bunch of modifications due to new structure of ECMWFDATA, added basics of documentation, minor programming corrections

Location:
python
Files:
6 added
1 deleted
15 edited
1 moved

Legend:

Unmodified
Added
Removed

  • python/CONTROL.temp

    rd69b677 refdb01a  
    1010EXPVER 1
    1111GRID 5000 
    12 LEFT -175000
    13 LOWER -90000
    14 UPPER 90000
    15 RIGHT 180000
     12LEFT -15000
     13LOWER 30000
     14UPPER 75000
     15RIGHT 45000
    1616LEVEL 60
    17 LEVELIST 1/to/60
     17LEVELIST 55/to/60
    1818RESOL 63
    1919GAUSS 1
    … …  
    2727FORMAT GRIB1
    2828ADDPAR 186/187/188/235/139/39
    29 PREFIX EN
    30 ECSTORAGE 1
     29PREFIX EI
     30ECSTORAGE 0
    3131ECTRANS 0
    3232ECFSDIR ectmp:/${USER}/econdemand/

  • python/ECMWF_ENV

    rd69b677 refdb01a  
    1 ECUID lh0
    2 ECGID spatlh00
    3 GATEWAY srvx7.img.univie.ac.at
    4 DESTINATION leo@genericSftp
     1ECUID km4a
     2ECGID at
     3GATEWAY srvx8.img.univie.ac.at
     4DESTINATION philipa8@genericSftp

  • python/GribTools.py

    r02c8c50 refdb01a  
    66# - GribTools name möglicherweise etwas verwirrend.
    77# - change self.filename in self.filenames!!!
    8 # -
     8# - add file description
     9# - bis auf --init-- und index wird keine Funktion verwendet!?
    910#************************************************************************
    1011"""
    … …  
    2021
    2122@License:
    22     (C) Copyright 2014 UIO.
     23    (C) Copyright 2014-2018.
    2324
    2425    This software is licensed under the terms of the Apache Licence Version 2.0
    … …  
    2829    - A standard python 2.6 or 2.7 installation
    2930    - dateutils
    30     - matplotlib (optional, for debugging)
    3131    - ECMWF specific packages, all available from https://software.ecmwf.int/
    3232        ECMWF WebMARS, gribAPI with python enabled, emoslib and
    … …  
    3636    Further documentation may be obtained from www.flexpart.eu.
    3737
    38     Functionality provided:
    39         Prepare input 3D-wind fields in hybrid coordinates +
    40         surface fields for FLEXPART runs
     38    ...
    4139"""
    4240# ------------------------------------------------------------------------------
    … …  
    4543from gribapi import *
    4644import traceback
    47 import sys, os
    48 
     45import sys
     46import os
    4947# ------------------------------------------------------------------------------
    5048# CLASS
    … …  
    5553    '''
    5654    # --------------------------------------------------------------------------
    57     # FUNCTIONS
     55    # CLASS FUNCTIONS
    5856    # --------------------------------------------------------------------------
    5957    def __init__(self, filenames):

  • python/UIOFiles.py

    r02c8c50 refdb01a  
    44# TODO AP
    55#AP
    6 # - File name und Klassenname gleichsetzen?
    76# - checken welche regelmässigen methoden auf diese Files noch angewendet werden
    87# und dann hier implementieren
    9 # - löschen?
     8# - add description of file!
    109#************************************************************************
    1110"""
    … …  
    1514
    1615@ChangeHistory:
    17    February 2018 - Anne Philipp (University of Vienna):
     16    November 2015 - Leopold Haimberger (University of Vienna):
     17        - modified method listFiles to work with glob instead of listdir
     18        - added pattern search in method listFiles
     19
     20    February 2018 - Anne Philipp (University of Vienna):
    1821        - applied PEP8 style guide
    1922        - added documentation
     23        - optimisation of method listFiles since it didn't work correctly
     24          for sub directories
     25        - additional speed up of method listFiles
    2026
    2127@License:
    22     (C) Copyright 2014 UIO.
     28    (C) Copyright 2014-2018.
    2329
    2430    This software is licensed under the terms of the Apache Licence Version 2.0
    … …  
    2632
    2733@Requirements:
    28     - A standard python 2.6 or 2.7 installation
    29     - dateutils
    30     - matplotlib (optional, for debugging)
    31     - ECMWF specific packages, all available from https://software.ecmwf.int/
    32         ECMWF WebMARS, gribAPI with python enabled, emoslib and
    33         ecaccess web toolkit
     34    A standard python 2.6 or 2.7 installation
    3435
    3536@Description:
    36     Further documentation may be obtained from www.flexpart.eu.
    37 
    38     Functionality provided:
    39         Prepare input 3D-wind fields in hybrid coordinates +
    40         surface fields for FLEXPART runs
     37    ...
    4138"""
    4239# ------------------------------------------------------------------------------
    … …  
    4542import os
    4643import glob
    47 
     44import fnmatch
     45import time
     46import profiling
    4847# ------------------------------------------------------------------------------
    49 # Class
     48# CLASS
    5049# ------------------------------------------------------------------------------
    5150class UIOFiles:
    … …  
    5655    '''
    5756    # --------------------------------------------------------------------------
    58     # FUNCTIONS
     57    # CLASS FUNCTIONS
    5958    # --------------------------------------------------------------------------
    6059    def __init__(self, suffix):
    … …  
    8079        return
    8180
    82     def listFiles(self, path, pattern):
     81    #@profiling.timefn
     82    def listFiles(self, path, pattern, callid=0):
    8383        '''
    8484        @Description:
    … …  
    9898                '*OG_acc_SL*.'+c.ppid+'.*'
    9999
     100            callid: integer
     101                Id which tells the function if its the first call
     102                or a recursive call. Default and first call is 0.
     103                Everything different from 0 is ment to be a recursive case.
     104
    100105        @Return:
    101106            <nothing>
    102107        '''
    103108
     109        # initialize variable in first function call
     110        if callid == 0:
     111            self.files = []
     112
    104113        # Get the absolute path
    105114        path = os.path.abspath(path)
    106115
    107         # Get a list of files in pathname
    108         filesInCurDir0 = glob.glob(path + '/' + pattern)
    109         filesInCurDir = []
    110         for f in filesInCurDir0:
    111             filesInCurDir.append(f.split('/')[-1])
     116        # get the file list of the path if its not a directory and
     117        # if it contains one of the suffixes
     118        self.files.extend([os.path.join(path, k) for k in os.listdir(path)
     119                           if fnmatch.fnmatch(k, pattern) and
     120                           os.path.splitext(k)[-1] in self.suffix])
    112121
    113         self.counter = 0
    114         self.files = []
    115         # Traverse through all files
    116         for file in filesInCurDir:
    117             curFile = os.path.join(path, file)
     122        # find possible sub-directories in the path
     123        subdirs = [s for s in os.listdir(path)
     124                   if os.path.isdir(os.path.join(path, s))]
    118125
    119             # Check if it's a normal file or directory
    120             if os.path.isfile(curFile):
    121                 # Get the file extension
    122                 fileNoExt, curFileExtension = os.path.splitext(curFile)
    123                 # Check if the file has an extension of typical video files
    124                 if curFileExtension in self.suffix:
    125                     # We have got a file file! Increment the counter
    126                     self.counter += 1
    127                     # add this filename in the list
    128                     self.files.append(curFile)
    129             else:
    130                 # We got a directory, enter into it for further processing
    131                 self.listFiles(curFile)
     126        # do recursive calls for sub-direcorties
     127        if subdirs:
     128            for subdir in subdirs:
     129                self.listFiles(os.path.join(path, subdir), pattern, callid=1)
    132130
    133131        return

  • python/compilejob.ksh

    rd69b677 refdb01a  
    55# start with sbatch NAME_OF_THIS_FILE directly on machine
    66
    7 #SBATCH --workdir=/scratch/ms/spatlh00/lh0
     7#SBATCH --workdir=/scratch/ms/at/km4a
    88#SBATCH --qos=normal
    99#SBATCH --job-name=flex_ecmwf
    … …  
    2525
    2626set -x
    27 export VERSION=7.0
     27export VERSION=7.1
    2828case $HOST in
    2929  *ecg*)
    … …  
    3333  module load grib_api/1.14.5
    3434  module load emos/437-r64
    35 export FLEXPART_ROOT_SCRIPTS=$HOME
    36 #  export ECMWFDATA=$FLEXPART_ROOT/ECMWFDATA7.0
     35export FLEXPART_ROOT_SCRIPTS=${HOME}
     36#  export ECMWFDATA=$FLEXPART_ROOT/ECMWFDATA$VERSION
    3737#  export PYTHONPATH=$ECMWFDATA/python
    3838#  export PATH=${PATH}:$ECMWFDATA/python
    … …  
    4949  export GROUP=`echo $HOME | awk -F / '{print $4}'`
    5050  export SCRATCH=/scratch/ms/${GROUP}/${USER}
    51 export FLEXPART_ROOT_SCRIPTS=$HOME
    52 #  export ECMWFDATA=$FLEXPART_ROOT/ECMWFDATA7.0
     51export FLEXPART_ROOT_SCRIPTS=${HOME}
     52#  export ECMWFDATA=$FLEXPART_ROOT/ECMWFDATA$VERSION
    5353#  export PYTHONPATH=$ECMWFDATA/python
    5454#  export PATH=${PATH}:$ECMWFDATA/python
    … …  
    5959mkdir -p $FLEXPART_ROOT_SCRIPTS/ECMWFDATA$VERSION
    6060cd $FLEXPART_ROOT_SCRIPTS/ECMWFDATA$VERSION   # if FLEXPART_ROOT is not set this means cd to the home directory
    61 tar -xvf $SCRATCH/ECMWFDATA$VERSION.tar
     61tar -xvf $HOME/ECMWFDATA$VERSION.tar
    6262cd src
    6363\rm *.o *.mod CONVERT2

  • python/compilejob.temp

    rd69b677 refdb01a  
    2525
    2626set -x
    27 export VERSION=7.0
     27export VERSION=7.1
    2828case $HOST in
    2929  *ecg*)
    … …  
    3434  module load emos/437-r64
    3535  export FLEXPART_ROOT_SCRIPTS=
    36 #  export ECMWFDATA=$FLEXPART_ROOT/ECMWFDATA7.0
     36#  export ECMWFDATA=$FLEXPART_ROOT/ECMWFDATA$VERSION
    3737#  export PYTHONPATH=$ECMWFDATA/python
    3838#  export PATH=${PATH}:$ECMWFDATA/python
    … …  
    5050  export SCRATCH=/scratch/ms/${GROUP}/${USER}
    5151  export FLEXPART_ROOT_SCRIPTS=
    52 #  export ECMWFDATA=$FLEXPART_ROOT/ECMWFDATA7.0
     52#  export ECMWFDATA=$FLEXPART_ROOT/ECMWFDATA$VERSION
    5353#  export PYTHONPATH=$ECMWFDATA/python
    5454#  export PATH=${PATH}:$ECMWFDATA/python
    … …  
    5959mkdir -p $FLEXPART_ROOT_SCRIPTS/ECMWFDATA$VERSION
    6060cd $FLEXPART_ROOT_SCRIPTS/ECMWFDATA$VERSION   # if FLEXPART_ROOT is not set this means cd to the home directory
    61 tar -xvf $SCRATCH/ECMWFDATA$VERSION.tar
     61tar -xvf $HOME/ECMWFDATA$VERSION.tar
    6262cd src
    6363\rm *.o *.mod CONVERT2

  • python/getMARSdata.py

    • Property mode changed from 100644 to 100755
    r02c8c50 refdb01a  
    11#!/usr/bin/env python
    2 #
    3 # This software is licensed under the terms of the Apache Licence Version 2.0
    4 # which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
    5 #
    6 # Functionality provided: Prepare input 3D-wind fields in hybrid coordinates + surface fields for FLEXPART runs
    7 #
    8 # Creation: October  2014 - Anne Fouilloux - University of Oslo
    9 # Extension November 2015 - Leopold Haimberger - University of Vienna for:
    10 # - using the WebAPI also for general MARS retrievals
    11 # - job submission on ecgate and cca
    12 # - job templates suitable for twice daily operational dissemination
    13 # - dividing retrievals of longer periods into digestable chunks
    14 # - retrieve also longer term forecasts, not only analyses and short term forecast data
    15 # - conversion into GRIB2
    16 # - conversion into .fp format for faster execution of FLEXPART
    17 #
    18 #
    19 # Further documentation may be obtained from www.flexpart.eu
    20 #
    21 # Requirements:
    22 # in addition to a standard python 2.6 or 2.7 installation the following packages need to be installed
    23 # ECMWF WebMARS, gribAPI with python enabled, emoslib, ecaccess web toolkit, all available from https://software.ecmwf.int/
    24 # dateutils
    25 # matplotlib (optional, for debugging)
    26 #
    27 # Get MARS GRIB fields from ECMWF for FLEXPART
    28 #
    29 
    30 #import socket
    31 
    32 #hostname=socket.gethostname()
    33 #ecapi= 'ecmwf' not in hostname
     2# -*- coding: utf-8 -*-
     3#************************************************************************
     4# TODO AP
     5# - Change History ist nicht angepasst ans File!
     6# - add file description
     7#************************************************************************
     8"""
     9@Author: Anne Fouilloux (University of Oslo)
     10
     11@Date: October 2014
     12
     13@ChangeHistory:
     14    November 2015 - Leopold Haimberger (University of Vienna):
     15        - using the WebAPI also for general MARS retrievals
     16        - job submission on ecgate and cca
     17        - job templates suitable for twice daily operational dissemination
     18        - dividing retrievals of longer periods into digestable chunks
     19        - retrieve also longer term forecasts, not only analyses and
     20          short term forecast data
     21        - conversion into GRIB2
     22        - conversion into .fp format for faster execution of FLEXPART
     23
     24    February 2018 - Anne Philipp (University of Vienna):
     25        - applied PEP8 style guide
     26        - added documentation
     27        - minor changes in programming style for consistence
     28
     29@License:
     30    (C) Copyright 2014-2018.
     31
     32    This software is licensed under the terms of the Apache Licence Version 2.0
     33    which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
     34
     35@Requirements:
     36    - A standard python 2.6 or 2.7 installation
     37    - dateutils
     38    - ECMWF specific packages, all available from https://software.ecmwf.int/
     39        ECMWF WebMARS, gribAPI with python enabled, emoslib and
     40        ecaccess web toolkit
     41
     42@Description:
     43    Further documentation may be obtained from www.flexpart.eu.
     44
     45"""
     46# ------------------------------------------------------------------------------
     47# MODULES
     48# ------------------------------------------------------------------------------
    3449try:
    3550    ecapi=True
    … …  
    4257import datetime
    4358import time
    44 import os,glob,sys,inspect
    45 #from string import strip
    46 from argparse import ArgumentParser,ArgumentDefaultsHelpFormatter
     59import os
     60import glob
     61import sys
     62import inspect
    4763# add path to submit.py to pythonpath so that python finds its buddies
    4864localpythonpath=os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
    … …  
    5066    sys.path.append(localpythonpath)
    5167
    52 from FlexpartTools import ECFlexpart,  \
    53                           Control, myerror, normalexit, \
     68from Control import Control
     69from Tools import myerror, normalexit, \
    5470                          interpret_args_and_control
    55 
    56 
     71from ECFlexpart import ECFlexpart
     72
     73# ------------------------------------------------------------------------------
     74# FUNCTION
     75# ------------------------------------------------------------------------------
    5776def getMARSdata(args, c):
    5877
    … …  
    7190    print 'ecapi:', c.ecapi
    7291
    73 # Retrieve ERA interim data for running flexpart
     92# Retrieve EC data for running flexpart
    7493#AP change this variant to correct format conversion with datetime
    7594#AP import datetime and timedelta explicitly
    … …  
    81100    if c.basetime == '00':
    82101        start = startm1
     102
    83103    eyear = int(c.end_date[:4])
    84104    emonth = int(c.end_date[4:6])
    … …  
    91111
    92112    datechunk = datetime.timedelta(days=int(c.date_chunk))
     113
     114    # retrieving of accumulated data fields (flux data), (maximum one month)
     115
     116    # remove old files
    93117    print 'removing content of ' + c.inputdir
    94     tobecleaned = glob.glob(c.inputdir + '/*_acc_*.' + str(os.getppid()) + '.*.grb')
     118    tobecleaned = glob.glob(c.inputdir + '/*_acc_*.' + \
     119                            str(os.getppid()) + '.*.grb')
    95120    for f in tobecleaned:
    96121        os.remove(f)
    97122
    98     times=None
    99     if c.maxstep<24:
    100         day=startm1
    101         while day<endp1:
    102                 # we need to retrieve MARS data for this period (maximum one month)
    103                 flexpart = ECFlexpart(c,fluxes=True)
    104                 if day+datechunk-datetime.timedelta(days=1)<endp1:
    105                     dates= day.strftime("%Y%m%d") + "/to/" + (day+datechunk-datetime.timedelta(days=1)).strftime("%Y%m%d")
     123    times = None
     124    # if forecast for maximum one day (upto 23h) are to be retrieved,
     125    # collect accumulation data (flux data)
     126    # with additional days in the beginning and at the end
     127    # (used for complete disaggregation of original period)
     128    if c.maxstep < 24:
     129        day = startm1
     130        while day < endp1:
     131                # retrieve MARS data for the whole period
     132                flexpart = ECFlexpart(c, fluxes=True)
     133                tmpday = day + datechunk - datetime.timedelta(days=1)
     134                if tmpday < endp1:
     135                    dates = day.strftime("%Y%m%d") + "/to/" + \
     136                            tmpday.strftime("%Y%m%d")
    106137                else:
    107                     dates= day.strftime("%Y%m%d") + "/to/" + end.strftime("%Y%m%d")
     138                    dates = day.strftime("%Y%m%d") + "/to/" + \
     139                            end.strftime("%Y%m%d")
    108140
    109141                print "retrieve " + dates + " in dir " + c.inputdir
     142
    110143                try:
    111144                    flexpart.retrieve(server, dates, times, c.inputdir)
    … …  
    113146                    myerror(c,'MARS request failed')
    114147
    115                 day+=datechunk
     148                day += datechunk
     149
     150    # if forecast data longer than 24h are to be retrieved,
     151    # collect accumulation data (flux data)
     152    # with the exact start and end date
     153    # (disaggregation will be done for the
     154    # exact time period with boundary conditions)
    116155    else:
    117         day=start
    118         while day<=end:
    119                 # we need to retrieve MARS data for this period (maximum one month)
    120                 flexpart = ECFlexpart(c,fluxes=True)
    121                 if day+datechunk-datetime.timedelta(days=1)<end:
    122                     dates= day.strftime("%Y%m%d") + "/to/" + (day+datechunk-datetime.timedelta(days=1)).strftime("%Y%m%d")
     156        day = start
     157        while day <= end:
     158                # retrieve MARS data for the whole period
     159                flexpart = ECFlexpart(c, fluxes=True)
     160                tmpday = day + datechunk - datetime.timedelta(days=1)
     161                if tmpday < end:
     162                    dates = day.strftime("%Y%m%d") + "/to/" + \
     163                            tmpday.trftime("%Y%m%d")
    123164                else:
    124                     dates= day.strftime("%Y%m%d") + "/to/" + end.strftime("%Y%m%d")
     165                    dates = day.strftime("%Y%m%d") + "/to/" + \
     166                            end.strftime("%Y%m%d")
    125167
    126168                print "retrieve " + dates + " in dir " + c.inputdir
    127                 flexpart.retrieve(server, dates, times, c.inputdir)
    128                 day+=datechunk
    129 
    130 
    131 
    132     tobecleaned=glob.glob(c.inputdir+'/*__*.'+str(os.getppid())+'.*.grb')
     169
     170                try:
     171                    flexpart.retrieve(server, dates, times, c.inputdir)
     172                except IOError:
     173                    myerror(c, 'MARS request failed')
     174
     175                day += datechunk
     176
     177    # retrieving of normal data fields (non flux data), (maximum one month)
     178
     179    # remove old *__* files
     180    tobecleaned = glob.glob(c.inputdir + '/*__*.' +
     181                            str(os.getppid()) + '.*.grb')
    133182    for f in tobecleaned:
    134183        os.remove(f)
    135     day=start
    136     times=None
    137     while day<=end:
    138 
    139                # we need to retrieve MARS data for this period (maximum one month)
    140             flexpart = ECFlexpart(c)
    141             if day+datechunk-datetime.timedelta(days=1)<end:
    142                 dates= day.strftime("%Y%m%d") + "/to/" + (day+datechunk-datetime.timedelta(days=1)).strftime("%Y%m%d")
     184    day = start
     185    times = None
     186    while day <= end:
     187            # retrieve MARS data for the whole period
     188            flexpart = ECFlexpart(c, fluxes=False)
     189            tmpday = day+datechunk-datetime.timedelta(days=1)
     190            if tmpday < end:
     191                dates = day.strftime("%Y%m%d") + "/to/" + \
     192                        tmpday.strftime("%Y%m%d")
    143193            else:
    144                 dates= day.strftime("%Y%m%d") + "/to/" + end.strftime("%Y%m%d")
     194                dates = day.strftime("%Y%m%d") + "/to/" + \
     195                        end.strftime("%Y%m%d")
     196
    145197            print "retrieve " + dates + " in dir " + c.inputdir
    146198
    147             flexpart.retrieve(server, dates, times, c.inputdir)
    148             day+=datechunk
    149 
     199            try:
     200                flexpart.retrieve(server, dates, times, c.inputdir)
     201            except IOError:
     202                myerror(c, 'MARS request failed')
     203
     204            day += datechunk
     205
     206    return
    150207
    151208if __name__ == "__main__":
    152209
    153     args,c=interpret_args_and_control()
    154     getMARSdata(args,c)
     210    args, c = interpret_args_and_control()
     211    getMARSdata(args, c)
    155212    normalexit(c)

  • python/install.py

    • Property mode changed from 100644 to 100755
    r02c8c50 refdb01a  
    6464localpythonpath=os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
    6565sys.path.append(localpythonpath)
    66 from UIOTools import UIOFiles
     66from UIOFiles import UIOFiles
    6767from string import strip
    6868from argparse import ArgumentParser,ArgumentDefaultsHelpFormatter
    6969from GribTools import GribTools
    70 from FlexpartTools import ECFlexpart, Control, install_args_and_control
     70from Control import Control
    7171from getMARSdata import getMARSdata
    7272from prepareFLEXPART import prepareFLEXPART
     73from ECFlexpart import ECFlexpart
    7374
    7475# ------------------------------------------------------------------------------
    7576# FUNCTIONS
    7677# ------------------------------------------------------------------------------
     78def install_args_and_control():
     79    '''
     80    @Description:
     81        Assigns the command line arguments for installation and reads
     82        control file content. Apply default values for non mentioned arguments.
     83
     84    @Input:
     85        <nothing>
     86
     87    @Return:
     88        args: instance of ArgumentParser
     89            Contains the commandline arguments from script/program call.
     90
     91        c: instance of class Control
     92            Contains all necessary information of a control file. The parameters
     93            are: DAY1, DAY2, DTIME, MAXSTEP, TYPE, TIME, STEP, CLASS, STREAM,
     94            NUMBER, EXPVER, GRID, LEFT, LOWER, UPPER, RIGHT, LEVEL, LEVELIST,
     95            RESOL, GAUSS, ACCURACY, OMEGA, OMEGADIFF, ETA, ETADIFF, DPDETA,
     96            SMOOTH, FORMAT, ADDPAR, WRF, CWC, PREFIX, ECSTORAGE, ECTRANS,
     97            ECFSDIR, MAILOPS, MAILFAIL, GRIB2FLEXPART, FLEXPARTDIR
     98            For more information about format and content of the parameter see
     99            documentation.
     100    '''
     101    parser = ArgumentParser(description='Install ECMWFDATA software locally or \
     102                            on ECMWF machines',
     103                            formatter_class=ArgumentDefaultsHelpFormatter)
     104
     105    parser.add_argument('--target', dest='install_target',
     106                        help="Valid targets: local | ecgate | cca , \
     107                        the latter two are at ECMWF")
     108    parser.add_argument("--makefile", dest="makefile",
     109                        help='Name of Makefile to use for compiling CONVERT2')
     110    parser.add_argument("--ecuid", dest="ecuid",
     111                        help='user id at ECMWF')
     112    parser.add_argument("--ecgid", dest="ecgid",
     113                        help='group id at ECMWF')
     114    parser.add_argument("--gateway", dest="gateway",
     115                        help='name of local gateway server')
     116    parser.add_argument("--destination", dest="destination",
     117                        help='ecaccess destination, e.g. leo@genericSftp')
     118
     119    parser.add_argument("--flexpart_root_scripts", dest="flexpart_root_scripts",
     120                        help="FLEXPART root directory on ECMWF servers \
     121                        (to find grib2flexpart and COMMAND file)\n\
     122                        Normally ECMWFDATA resides in the scripts directory \
     123                        of the FLEXPART distribution, thus the:")
     124
     125# arguments for job submission to ECMWF, only needed by submit.py
     126    parser.add_argument("--job_template", dest='job_template',
     127                        default="job.temp.o",
     128                        help="job template file for submission to ECMWF")
     129
     130    parser.add_argument("--controlfile", dest="controlfile",
     131                        default='CONTROL.temp',
     132                        help="file with control parameters")
     133
     134    args = parser.parse_args()
     135
     136    try:
     137        c = Control(args.controlfile)
     138    except:
     139        print('Could not read control file "' + args.controlfile + '"')
     140        print('Either it does not exist or its syntax is wrong.')
     141        print('Try "' + sys.argv[0].split('/')[-1] +
     142              ' -h" to print usage information')
     143        exit(1)
     144
     145    if args.install_target != 'local':
     146        if (args.ecgid is None or args.ecuid is None or args.gateway is None
     147            or args.destination is None):
     148            print('Please enter your ECMWF user id and group id as well as \
     149                   the \nname of the local gateway and the ectrans \
     150                   destination ')
     151            print('with command line options --ecuid --ecgid \
     152                   --gateway --destination')
     153            print('Try "' + sys.argv[0].split('/')[-1] +
     154                  ' -h" to print usage information')
     155            print('Please consult ecaccess documentation or ECMWF user support \
     156                   for further details')
     157            sys.exit(1)
     158        else:
     159            c.ecuid = args.ecuid
     160            c.ecgid = args.ecgid
     161            c.gateway = args.gateway
     162            c.destination = args.destination
     163
     164    try:
     165        c.makefile = args.makefile
     166    except:
     167        pass
     168
     169    if args.install_target == 'local':
     170        if args.flexpart_root_scripts is None:
     171            c.flexpart_root_scripts = '../'
     172        else:
     173            c.flexpart_root_scripts = args.flexpart_root_scripts
     174
     175    if args.install_target != 'local':
     176        if args.flexpart_root_scripts is None:
     177            c.ec_flexpart_root_scripts = '${HOME}'
     178        else:
     179            c.ec_flexpart_root_scripts = args.flexpart_root_scripts
     180
     181    return args, c
     182
     183
    77184def main():
    78185    '''
    … …  
    139246                        c.ecuid + 'flex_ecmwf.$Jobname.$Job_ID.out'
    140247            if  'export PATH=${PATH}:' in data:
    141                 data += c.ec_flexpart_root_scripts + '/ECMWFDATA7.0/python'
     248                data += c.ec_flexpart_root_scripts + '/ECMWFDATA7.1/python'
    142249            if 'cat>>' in data or 'cat >>' in data:
    143250                i = data.index('>')
    … …  
    171278                os.chdir('/')
    172279                p = subprocess.check_call(['tar', '-cvf',
    173                                            ecd + '../ECMWFDATA7.0.tar',
     280                                           ecd + '../ECMWFDATA7.1.tar',
    174281                                           ecd + 'python',
    175282                                           ecd + 'grib_templates',
    176283                                           ecd + 'src'])
    177284                try:
    178                     os.makedirs(c.flexpart_root_scripts + '/ECMWFDATA7.0')
     285                    os.makedirs(c.flexpart_root_scripts + '/ECMWFDATA7.1')
    179286                except:
    180287                    pass
    181                 os.chdir(c.flexpart_root_scripts + '/ECMWFDATA7.0')
     288                os.chdir(c.flexpart_root_scripts + '/ECMWFDATA7.1')
    182289                p = subprocess.check_call(['tar', '-xvf',
    183                                            ecd + '../ECMWFDATA7.0.tar'])
    184                 os.chdir(c.flexpart_root_scripts + '/ECMWFDATA7.0/src')
     290                                           ecd + '../ECMWFDATA7.1.tar'])
     291                os.chdir(c.flexpart_root_scripts + '/ECMWFDATA7.1/src')
    185292
    186293        os.chdir('../src')
    187         print(('install ECMWFDATA7.0 software on ' + target + ' in directory '
     294        print(('install ECMWFDATA7.1 software on ' + target + ' in directory '
    188295               + os.getcwd()))
    189296        if c.makefile is None:
    … …  
    197304            print(('Using makefile: ' + makefile))
    198305            p = subprocess.check_call(['make', '-f', makefile])
    199             p = subprocess.check_call(['ls', '-l',' CONVERT2'])
     306            p = subprocess.check_call(['ls', '-l','CONVERT2'])
    200307        except:
    201308            print('compile failed - please edit ' + makefile +
    202309                  ' or try another Makefile in the src directory.')
    203             print('most likely GRIB_API_INCLUDE_DIR, GRIB_API_LIB \
    204                     and EMOSLIB must be adapted.')
     310            print('most likely GRIB_API_INCLUDE_DIR, GRIB_API_LIB '
     311                    'and EMOSLIB must be adapted.')
    205312            print('Available Makefiles:')
    206313            print(glob.glob('Makefile*'))
    … …  
    209316        os.chdir('/')
    210317        p = subprocess.check_call(['tar', '-cvf',
    211                                    ecd + '../ECMWFDATA7.0.tar',
     318                                   ecd + '../ECMWFDATA7.1.tar',
    212319                                   ecd + 'python',
    213320                                   ecd + 'grib_templates',
    … …  
    215322        try:
    216323            p = subprocess.check_call(['ecaccess-file-put',
    217                                        ecd + '../ECMWFDATA7.0.tar',
    218                                        'ecgate:/scratch/ms/' + c.ecgid + '/' +
    219                                        c.ecuid + '/ECMWFDATA7.0.tar'])
     324                                       ecd + '../ECMWFDATA7.1.tar',
     325                                       'ecgate:/home/ms/' + c.ecgid + '/' +
     326                                       c.ecuid + '/ECMWFDATA7.1.tar'])
    220327        except:
    221328            print('ecaccess-file-put failed! Probably the eccert key has expired.')
    … …  
    225332                                   target,
    226333                                   ecd + 'python/compilejob.ksh'])
    227         print('compilejob.ksh has been submitted to ecgate for \
    228                 installation in ' + c.ec_flexpart_root_scripts +
    229                 '/ECMWFDATA7.0')
    230         print('You should get an email with subject flexcompile within \
    231                 the next few minutes')
     334        print('compilejob.ksh has been submitted to ecgate for '
     335                'installation in ' + c.ec_flexpart_root_scripts +
     336                '/ECMWFDATA7.1')
     337        print('You should get an email with subject flexcompile within '
     338                'the next few minutes')
    232339
    233340    elif target.lower() == 'cca':
    234341        os.chdir('/')
    235342        p = subprocess.check_call(['tar', '-cvf',
    236                                    ecd + '../ECMWFDATA7.0.tar',
     343                                   ecd + '../ECMWFDATA7.1.tar',
    237344                                   ecd + 'python',
    238345                                   ecd + 'grib_templates',
    … …  
    240347        try:
    241348            p = subprocess.check_call(['ecaccess-file-put',
    242                                        ecd + '../ECMWFDATA7.0.tar',
    243                                        'cca:/scratch/ms/' + c.ecgid + '/' +
    244                                        c.ecuid + '/ECMWFDATA7.0.tar'])
     349                                       ecd + '../ECMWFDATA7.1.tar',
     350                                       'cca:/home/ms/' + c.ecgid + '/' +
     351                                       c.ecuid + '/ECMWFDATA7.1.tar'])
    245352        except:
    246             print('ecaccess-file-put failed! \
    247                     Probably the eccert key has expired.')
     353            print('ecaccess-file-put failed! '
     354                    'Probably the eccert key has expired.')
    248355            exit(1)
    249356
    … …  
    251358                                '-queueName',
    252359                                target,
    253                                 ecd + 'python/compilejob.ksh']))
     360                                ecd + 'python/compilejob.ksh'])
    254361        print('compilejob.ksh has been submitted to cca for installation in ' +
    255               c.ec_flexpart_root_scripts + '/ECMWFDATA7.0')
    256         print('You should get an email with subject flexcompile \
    257                 within the next few minutes')
     362              c.ec_flexpart_root_scripts + '/ECMWFDATA7.1')
     363        print('You should get an email with subject flexcompile '
     364                'within the next few minutes')
    258365
    259366    else:

  • python/job.ksh

    rd69b677 refdb01a  
    55# start with sbatch NAME_OF_THIS_FILE directly on machine
    66
    7 #SBATCH --workdir=/scratch/ms/spatlh00/lh0
     7#SBATCH --workdir=/scratch/ms/at/km4a
    88#SBATCH --qos=normal
    99#SBATCH --job-name=flex_ecmwf
    … …  
    2525
    2626set -x
    27 
     27export VERSION=7.1
    2828case $HOST in
    2929  *ecg*)
    … …  
    3333  module load grib_api/1.14.5
    3434  module load emos/437-r64
    35 #  export ECMWFDATA=$HOME/ECMWFDATA7.0
     35#  export ECMWFDATA=$HOME/ECMWFDATA$VERSION
    3636#  export PYTHONPATH=$ECMWFDATA/python
    37   export PATH=${PATH}:${HOME}/ECMWFDATA7.0/python
     37  export PATH=${PATH}:${HOME}/ECMWFDATA7.1/python
    3838  ;;
    3939  *cca*)
    … …  
    4343  module load python
    4444  export SCRATCH=$TMPDIR
    45 #  export ECMWFDATA=$HOME/ECMWFDATA7.0
     45#  export ECMWFDATA=$HOME/ECMWFDATA$VERSION
    4646#  export PYTHONPATH=$ECMWFDATA/python
    47   export PATH=${PATH}:${HOME}/ECMWFDATA7.0/python
     47  export PATH=${PATH}:${HOME}/ECMWFDATA7.1/python
    4848  ;;
    4949#  *)
    50 #  export ECMWFDATA=$HOME/ECMWFDATA7.0
     50#  export ECMWFDATA=$HOME/ECMWFDATA$VERSION
    5151#  export PATH=/opt/anaconda/bin:$ECMWFDATA/python:${PATH}
    5252#  export PYTHONPATH=/opt/anaconda/lib/python2.7/site-packages/grib_api:$ECMWFDATA/python
    … …  
    6363
    6464cat >$CONTROL<<EOF
    65 GATEWAY srvx7.img.univie.ac.at
    66 DESTINATION leo@genericSftp
     65GATEWAY srvx8.img.univie.ac.at
     66DESTINATION philipa8@genericSftp
    6767accuracy 16
    6868addpar 186 187 188 235 139 39
    … …  
    7474dtime 3
    7575ecfsdir ectmp:/${USER}/econdemand/
    76 ecstorage 1
     76ecstorage 0
    7777ectrans 0
    78 end_date 20131107
     78end_date 20160809
    7979eta 0
    8080etadiff 0
    … …  
    8585grid 5000
    8686inputdir ../work
    87 left -175000
     87left -15000
    8888level 60
    89 levelist 1/to/60
    90 lower -90000
     89levelist 55/to/60
     90lower 30000
    9191mailfail ${USER}
    9292mailops ${USER}
    … …  
    9898omegadiff 0
    9999outputdir ../work
    100 prefix EN
     100prefix EI
    101101resol 63
    102 right 180000
     102right 45000
    103103smooth 0
    104 start_date 20131107
     104start_date 20160809
    105105step 00 01 02 03 04 05 00 07 08 09 10 11 00 01 02 03 04 05 00 07 08 09 10 11
    106106stream OPER
    107107time 00 00 00 00 00 00 06 00 00 00 00 00 12 12 12 12 12 12 18 12 12 12 12 12
    108108type AN FC FC FC FC FC AN FC FC FC FC FC AN FC FC FC FC FC AN FC FC FC FC FC
    109 upper 90000
     109upper 75000
    110110EOF
    111111
    112112
    113 submit.py --controlfile=$CONTROL --inputdir=./work --outputdir=./work >prot
     113submit.py --controlfile=$CONTROL --inputdir=./work --outputdir=./work 1> prot 2>&1
    114114
    115115if [ $? -eq 0 ] ; then

  • python/job.temp

    rd69b677 refdb01a  
    55# start with sbatch NAME_OF_THIS_FILE directly on machine
    66
    7 #SBATCH --workdir=/scratch/ms/spatlh00/lh0
     7#SBATCH --workdir=/scratch/ms/at/km4a
    88#SBATCH --qos=normal
    99#SBATCH --job-name=flex_ecmwf
    … …  
    2525
    2626set -x
    27 
     27export VERSION=7.1
    2828case $HOST in
    2929  *ecg*)
    … …  
    3333  module load grib_api/1.14.5
    3434  module load emos/437-r64
    35 #  export ECMWFDATA=$HOME/ECMWFDATA7.0
     35#  export ECMWFDATA=$HOME/ECMWFDATA$VERSION
    3636#  export PYTHONPATH=$ECMWFDATA/python
    37   export PATH=${PATH}:${HOME}/ECMWFDATA7.0/python
     37  export PATH=${PATH}:${HOME}/ECMWFDATA7.1/python
    3838  ;;
    3939  *cca*)
    … …  
    4343  module load python
    4444  export SCRATCH=$TMPDIR
    45 #  export ECMWFDATA=$HOME/ECMWFDATA7.0
     45#  export ECMWFDATA=$HOME/ECMWFDATA$VERSION
    4646#  export PYTHONPATH=$ECMWFDATA/python
    47   export PATH=${PATH}:${HOME}/ECMWFDATA7.0/python
     47  export PATH=${PATH}:${HOME}/ECMWFDATA7.1/python
    4848  ;;
    4949#  *)
    50 #  export ECMWFDATA=$HOME/ECMWFDATA7.0
     50#  export ECMWFDATA=$HOME/ECMWFDATA$VERSION
    5151#  export PATH=/opt/anaconda/bin:$ECMWFDATA/python:${PATH}
    5252#  export PYTHONPATH=/opt/anaconda/lib/python2.7/site-packages/grib_api:$ECMWFDATA/python
    … …  
    6363
    6464cat >$CONTROL<<EOF
    65 GATEWAY srvx7.img.univie.ac.at
    66 DESTINATION leo@genericSftp
     65GATEWAY srvx8.img.univie.ac.at
     66DESTINATION philipa8@genericSftp
    6767EOF
    6868cat >>$CONTROL<<EOF
    … …  
    7070
    7171
    72 submit.py --controlfile=$CONTROL --inputdir=./work --outputdir=./work >prot
     72submit.py --controlfile=$CONTROL --inputdir=./work --outputdir=./work 1> prot 2>&1
    7373
    7474if [ $? -eq 0 ] ; then

  • python/job.temp.o

    rd69b677 refdb01a  
    2525
    2626set -x
    27 
     27export VERSION=7.1
    2828case $HOST in
    2929  *ecg*)
    … …  
    3333  module load grib_api/1.14.5
    3434  module load emos/437-r64
    35 #  export ECMWFDATA=$HOME/ECMWFDATA7.0
     35#  export ECMWFDATA=$HOME/ECMWFDATA$VERSION
    3636#  export PYTHONPATH=$ECMWFDATA/python
    3737  export PATH=${PATH}:
    … …  
    4343  module load python
    4444  export SCRATCH=$TMPDIR
    45 #  export ECMWFDATA=$HOME/ECMWFDATA7.0
     45#  export ECMWFDATA=$HOME/ECMWFDATA$VERSION
    4646#  export PYTHONPATH=$ECMWFDATA/python
    4747  export PATH=${PATH}:
    4848  ;;
    4949#  *)
    50 #  export ECMWFDATA=$HOME/ECMWFDATA7.0
     50#  export ECMWFDATA=$HOME/ECMWFDATA$VERSION
    5151#  export PATH=/opt/anaconda/bin:$ECMWFDATA/python:${PATH}
    5252#  export PYTHONPATH=/opt/anaconda/lib/python2.7/site-packages/grib_api:$ECMWFDATA/python
    … …  
    6666
    6767
    68 submit.py --controlfile=$CONTROL --inputdir=./work --outputdir=./work >prot
     68submit.py --controlfile=$CONTROL --inputdir=./work --outputdir=./work 1> prot 2>&1
    6969
    7070if [ $? -eq 0 ] ; then

  • python/joboper.ksh

    rd69b677 refdb01a  
    55# start with sbatch NAME_OF_THIS_FILE directly on machine
    66
    7 #SBATCH --workdir=/scratch/ms/spatlh00/lh0
     7#SBATCH --workdir=/scratch/ms/at/km4a
    88#SBATCH --qos=normal
    99#SBATCH --job-name=flex_ecmwf
    … …  
    2525
    2626set -x
    27 
     27export VERSION=7.1
    2828case $HOST in
    2929  *ecg*)
    … …  
    3333  module load grib_api/1.14.5
    3434  module load emos/437-r64
    35 #  export ECMWFDATA=$HOME/ECMWFDATA7.0
     35#  export ECMWFDATA=$HOME/ECMWFDATA$VERSION
    3636#  export PYTHONPATH=$ECMWFDATA/python
    37   export PATH=${PATH}:${HOME}/ECMWFDATA7.0/python
     37  export PATH=${PATH}:${HOME}/ECMWFDATA7.1/python
    3838  ;;
    3939  *cca*)
    … …  
    4343  module load python
    4444  export SCRATCH=$TMPDIR
    45 #  export ECMWFDATA=$HOME/ECMWFDATA7.0
     45#  export ECMWFDATA=$HOME/ECMWFDATA$VERSION
    4646#  export PYTHONPATH=$ECMWFDATA/python
    47   export PATH=${PATH}:${HOME}/ECMWFDATA7.0/python
     47  export PATH=${PATH}:${HOME}/ECMWFDATA7.1/python
    4848  ;;
    4949#  *)
    50 #  export ECMWFDATA=$HOME/ECMWFDATA7.0
     50#  export ECMWFDATA=$HOME/ECMWFDATA$VERSION
    5151#  export PATH=/opt/anaconda/bin:$ECMWFDATA/python:${PATH}
    5252#  export PYTHONPATH=/opt/anaconda/lib/python2.7/site-packages/grib_api:$ECMWFDATA/python
    … …  
    6363
    6464cat >$CONTROL<<EOF
    65 GATEWAY srvx7.img.univie.ac.at
    66 DESTINATION leo@genericSftp
     65GATEWAY srvx8.img.univie.ac.at
     66DESTINATION philipa8@genericSftp
    6767accuracy 16
    6868addpar 186 187 188 235 139 39
    … …  
    7474dtime 3
    7575ecfsdir ectmp:/${USER}/econdemand/
    76 ecstorage 1
     76ecstorage 0
    7777ectrans 0
    7878start_date ${MSJ_YEAR}${MSJ_MONTH}${MSJ_DAY}
    … …  
    8585grid 5000
    8686inputdir ../work
    87 left -175000
     87left -15000
    8888level 60
    89 levelist 1/to/60
    90 lower -90000
     89levelist 55/to/60
     90lower 30000
    9191mailfail ${USER}
    9292mailops ${USER}
    … …  
    9898omegadiff 0
    9999outputdir ../work
    100 prefix EN
     100prefix EI
    101101resol 63
    102 right 180000
     102right 45000
    103103smooth 0
    104104start_date ${MSJ_YEAR}${MSJ_MONTH}${MSJ_DAY}
    … …  
    107107time 00 00 00 00 00 00 06 00 00 00 00 00 12 12 12 12 12 12 18 12 12 12 12 12
    108108type AN FC FC FC FC FC AN FC FC FC FC FC AN FC FC FC FC FC AN FC FC FC FC FC
    109 upper 90000
     109upper 75000
    110110EOF
    111111
    112112
    113 submit.py --controlfile=$CONTROL --inputdir=./work --outputdir=./work >prot
     113submit.py --controlfile=$CONTROL --inputdir=./work --outputdir=./work 1> prot 2>&1
    114114
    115115if [ $? -eq 0 ] ; then

  • python/plot_retrieved.py

    • Property mode changed from 100644 to 100755
    r64cf353 refdb01a  
    3030from argparse import ArgumentParser,ArgumentDefaultsHelpFormatter
    3131
    32 from FlexpartTools import interpret_args_and_control,silentremove,product,Control
     32from Tools import interpret_args_and_control, silentremove, product
     33from Control import Control
    3334from GribTools import GribTools
    3435from gribapi import *

  • python/prepareFLEXPART.py

    • Property mode changed from 100644 to 100755
    r02c8c50 refdb01a  
    3030        - added documentation
    3131        - minor changes in programming style for consistence
     32        - BUG: removed call of cleanup-Function after call of prepareFlexpart
     33                since it is already called in prepareFlexpart at the end!
    3234
    3335@License:
    34     (C) Copyright 2014.
     36    (C) Copyright 2014-2018.
    3537
    3638    This software is licensed under the terms of the Apache Licence Version 2.0
    … …  
    4042    - A standard python 2.6 or 2.7 installation
    4143    - dateutils
    42     - matplotlib (optional, for debugging)
    4344    - ECMWF specific packages, all available from https://software.ecmwf.int/
    4445        ECMWF WebMARS, gribAPI with python enabled, emoslib and
    … …  
    5253        surface fields for FLEXPART runs
    5354"""
    54 
     55# ------------------------------------------------------------------------------
     56# MODULES
     57# ------------------------------------------------------------------------------
    5558import calendar
    5659import shutil
    5760import datetime
    5861import time
    59 import os,inspect,sys
     62import os
     63import inspect
     64import sys
    6065import socket
    6166from argparse import ArgumentParser, ArgumentDefaultsHelpFormatter
    62 # add path to submit.py to pythonpath so that python finds its buddies
    63 localpythonpath=os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
    64 if localpythonpath not in sys.path:
    65     sys.path.append(localpythonpath)
    66 from UIOTools import UIOFiles
    67 #from string import strip
    68 from GribTools import GribTools
    69 from FlexpartTools import ECFlexpart, Control, interpret_args_and_control, cleanup
     67import UIOFiles
     68import Control
     69import Tools
     70import ECFlexpart
    7071
    71 hostname=socket.gethostname()
    72 ecapi= 'ecmwf' not in hostname
    73 
     72hostname = socket.gethostname()
     73ecapi = 'ecmwf' not in hostname
    7474try:
    7575    if ecapi:
    … …  
    7878    ecapi = False
    7979
    80 
     80# add path to submit.py to pythonpath so that python finds its buddies
     81localpythonpath=os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
     82if localpythonpath not in sys.path:
     83    sys.path.append(localpythonpath)
     84# ------------------------------------------------------------------------------
     85# FUNCTION
     86# ------------------------------------------------------------------------------
    8187def prepareFLEXPART(args, c):
    8288    '''
    … …  
    121127                        day=int(c.end_date[6:]))
    122128
    123     # to deaccumulated the fluxes correctly
     129    # to deaccumulate the fluxes correctly
    124130    # one day ahead of the start date and
    125131    # one day after the end date is needed
    … …  
    128134
    129135    # get all files with flux data to be deaccumulated
    130     inputfiles = UIOFiles(['.grib', '.grb', '.grib1',
     136    inputfiles = UIOFiles.UIOFiles(['.grib', '.grb', '.grib1',
    131137                           '.grib2', '.grb1', '.grb2'])
    132138
    … …  
    138144
    139145    # deaccumulate the flux data
    140     flexpart = ECFlexpart(c, fluxes=True)
     146    flexpart = ECFlexpart.ECFlexpart(c, fluxes=True)
    141147    flexpart.write_namelist(c, 'fort.4')
    142148    flexpart.deacc_fluxes(inputfiles, c)
    … …  
    146152
    147153    # get a list of all files from the root inputdir
    148     inputfiles = UIOFiles(['.grib', '.grb', '.grib1',
     154    inputfiles = UIOFiles.UIOFiles(['.grib', '.grb', '.grib1',
    149155                           '.grib2', '.grb1', '.grb2'])
    150156
    … …  
    157163        start = startm1
    158164
    159     flexpart = ECFlexpart(c, fluxes=False)
     165    flexpart = ECFlexpart.ECFlexpart(c, fluxes=False)
    160166    flexpart.create(inputfiles, c)
    161167    flexpart.process_output(c)
    162168
    163169    # check if in debugging mode, then store all files
     170    # otherwise delete temporary files
    164171    if int(c.debug) != 0:
    165172        print('Temporary files left intact')
    166173    else:
    167         cleanup(c)
     174        Tools.cleanup(c)
    168175
    169176    return
    170177
    171178if __name__ == "__main__":
    172     args, c = interpret_args_and_control()
     179    args, c = Tools.interpret_args_and_control()
    173180    prepareFLEXPART(args, c)
    174     cleanup(c)
    175 

  • python/submit.py

    • Property mode changed from 100644 to 100755
    r02c8c50 refdb01a  
    33#************************************************************************
    44# TODO AP
    5 #AP
     5#
    66# - Change History ist nicht angepasst ans File! Original geben lassen
    77# - dead code ? what to do?
    88# - seperate operational and reanlysis for clarification
     9# - add correct file description
     10# - divide in two submits , ondemand und operational
     11# -
    912#************************************************************************
    1013"""
    … …  
    1518@ChangeHistory:
    1619    November 2015 - Leopold Haimberger (University of Vienna):
    17         - using the WebAPI also for general MARS retrievals
    1820        - job submission on ecgate and cca
    1921        - job templates suitable for twice daily operational dissemination
    20         - dividing retrievals of longer periods into digestable chunks
    21         - retrieve also longer term forecasts, not only analyses and
    22           short term forecast data
    23         - conversion into GRIB2
    24         - conversion into .fp format for faster execution of FLEXPART
    2522
    2623    February 2018 - Anne Philipp (University of Vienna):
    2724        - applied PEP8 style guide
    2825        - added documentation
    29         - minor changes in programming style for consistence
     26        - minor changes in programming style (for consistence)
    3027
    3128@License:
    32     (C) Copyright 2014 UIO.
     29    (C) Copyright 2014-2018.
    3330
    3431    This software is licensed under the terms of the Apache Licence Version 2.0
    … …  
    3734@Requirements:
    3835    - A standard python 2.6 or 2.7 installation
    39     - dateutils
    40     - matplotlib (optional, for debugging)
    41     - ECMWF specific packages, all available from https://software.ecmwf.int/
    42         ECMWF WebMARS, gribAPI with python enabled, emoslib and
    43         ecaccess web toolkit
    4436
    4537@Description:
    4638    Further documentation may be obtained from www.flexpart.eu.
    4739
    48     Functionality provided:
    49         Prepare input 3D-wind fields in hybrid coordinates +
    50         surface fields for FLEXPART runs
     40
    5141"""
    5242# ------------------------------------------------------------------------------
    5343# MODULES
    5444# ------------------------------------------------------------------------------
    55 import calendar
    56 import shutil
    57 import datetime
    58 import time
    59 import os, sys, glob
     45import os
     46import sys
     47import glob
    6048import subprocess
    6149import inspect
    62 # add path to submit.py to pythonpath so that python finds its buddies
     50# add the pythondir path so that python finds its buddies (from flex_extract)
    6351localpythonpath = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
    6452sys.path.append(localpythonpath)
    65 from UIOTools import UIOFiles
    66 from string import strip
    67 from argparse import ArgumentParser, ArgumentDefaultsHelpFormatter
    68 from GribTools import GribTools
    69 from FlexpartTools import ECFlexpart, Control, interpret_args_and_control, normalexit, myerror
     53
     54# software specific classes and modules from flex_extract
     55from Tools import interpret_args_and_control, normalexit
    7056from getMARSdata import getMARSdata
    7157from prepareFLEXPART import prepareFLEXPART
    … …  
    7662    '''
    7763    @Description:
    78         Get the arguments from script call and initialize an object from
    79         Control class. Decides from the argument "queue" if the local version
    80         is done "queue=None" or the gateway version "queue=ecgate".
     64        Get the arguments from script call and from Control file.
     65        Decides from the argument "queue" if the local version
     66        is done "queue=None" or the gateway version with "queue=ecgate"
     67        or "queue=cca".
    8168
    8269    @Input:
    … …  
    8875    calledfromdir = os.getcwd()
    8976    args, c = interpret_args_and_control()
     77    # on local side
    9078    if args.queue is None:
    9179        if c.inputdir[0] != '/':
    … …  
    9684        prepareFLEXPART(args, c)
    9785        normalexit(c)
     86    # on ECMWF server
    9887    else:
    9988        submit(args.job_template, c, args.queue)
    … …  
    10291
    10392def submit(jtemplate, c, queue):
    104     #AP divide in two submits , ondemand und operational
    10593    '''
    10694    @Description:
    … …  
    138126        lftext = f.read().split('\n')
    139127        insert_point = lftext.index('EOF')
    140 #AP es gibt mehrere EOFs überprüfen!
    141128
    142129    # put all parameters of control instance into a list
    143     clist = c.tolist()  # reanalysis (EI)
     130    clist = c.tolist()
    144131    colist = []  # operational
    145132    mt = 0
    146133
    147134#AP wieso 2 for loops?
    148 #AP dieser part ist für die CTBTO Operational retrieves bis zum aktuellen Tag.
    149135    for elem in clist:
    150136        if 'maxstep' in elem:
    … …  
    162148            elem = 'time ' + '${MSJ_BASETIME} {MSJ_BASETIME}'
    163149        colist.append(elem)
    164 #AP end
    165 
    166 #AP whats the difference between clist and colist ?! What is MSJ?
    167150
    168151    lftextondemand = lftext[:insert_point] + clist + lftext[insert_point + 2:]
    … …  
    172155        h.write('\n'.join(lftextondemand))
    173156
    174 #AP this is not used ?! what is it for?
    175 #maybe a differentiation is needed
    176     h = open('joboper.ksh', 'w')
    177     h.write('\n'.join(lftextoper))
    178     h.close()
    179 #AP end
     157    with open('joboper.ksh', 'w') as h:
     158        h.write('\n'.join(lftextoper))
    180159
    181160    # submit job script to queue
    182161    try:
    183162        p = subprocess.check_call(['ecaccess-job-submit', '-queueName',
    184                                    queue,' job.ksh'])
     163                                   queue, 'job.ksh'])
    185164    except:
    186165        print('ecaccess-job-submit failed, probably eccert has expired')

  • python/testsuite.py

    • Property mode changed from 100644 to 100755
Note: See TracChangeset for help on using the changeset viewer.
hosted by ZAMG