Changeset 8209738 in flex_extract.git for Source/Python
- Timestamp:
- May 7, 2020, 3:32:03 PM (4 years ago)
- Branches:
- master, ctbto, dev
- Children:
- b936fd3
- Parents:
- 05e9362
- Location:
- Source/Python
- Files:
-
- 5 edited
Legend:
- Unmodified
- Added
- Removed
-
Source/Python/Mods/checks.py
r50f9ca6 r8209738 119 119 '''Defines the correct area string. 120 120 121 Checks on the format of the four area components. Wether it is of 122 the order of 1000 or 1. Also checks wether area was already set by command 123 line, then the four components are overwritten. 121 Checks the format of the four area components wether it is on 122 the order of 1000 or 1. 123 Also checks wether area was already set on command line, 124 then the four components are overwritten. 124 125 Convert to correct format of the order of magnitude "1" and sets the 125 126 area parameter (North/West/South/East). … … 135 136 136 137 upper : str 137 The northern 138 The northernmost latitude. 138 139 139 140 lower : str 140 The souther 141 The southernmost latitude. 141 142 142 143 left : str 143 The western 144 The westernmost longitude. 144 145 145 146 right : str 146 The eastern most longiude.147 The easternmost longitude. 147 148 148 149 Return 149 150 ------ 150 151 grid : str 151 Contains grid in format Lat/lon. E.g. 0.1/0.1152 Contains grid in format lat/lon. E.g. 0.1/0.1 152 153 ''' 153 154 if 'N' in grid: # Gaussian output grid … … 180 181 float(right)) 181 182 else: 182 raise ValueError(' The area components have different'183 raise ValueError('Area components have inconsisten or unrecognised ' 183 184 'formats (upper, lower, left, right): ' 184 185 '{}/{}/{}/{}'.format(str(upper), str(lower), … … 209 210 level : str 210 211 Specifies the maximum level. It has to be one of the 211 available maximum level number as contained in the variable212 MAX_LEVEL_LIST in "_config" . E.g.[16, 19, 31, 40, 50, 60, 62, 91, 137]212 available maximum level numbers as contained in the variable 213 MAX_LEVEL_LIST in "_config": [16, 19, 31, 40, 50, 60, 62, 91, 137] 213 214 214 215 ''' … … 346 347 for i, val in enumerate(ftype): 347 348 if ftype[i] == 'AN' and int(steps[i]) != 0: 348 print(' Analysis retrievals must have STEP = 0 (now set to 0)')349 print('For analysis data, STEP = 0 is needed. Setting to 0 now.)') 349 350 ftype[i] = 0 350 351 … … 386 387 maxstep : int 387 388 The maximum forecast time step in hours from the forecast base time. 388 This is the maximum step for non flux (accumulated) forecast data.389 This is the maximum step for non-flux (not accumulated) forecast data. 389 390 390 391 purefc : int … … 405 406 if not len(ftype) == len(ftime) == len(steps): 406 407 raise ValueError('ERROR: The number of field types, times and steps ' 407 'are not the same! Please check the setting in the '408 'are not the same! Please check the settings in the ' 408 409 'CONTROL file!') 409 410 … … 423 424 424 425 def check_mail(mail): 425 '''Check the string of mail addresses, sep erate them and convert to alist.426 '''Check the string of mail addresses, separate them and convert to list. 426 427 427 428 Parameters … … 529 530 530 531 def check_dates(start, end): 531 '''Checks if there is at least a start date for a one 532 '''Checks if there is at least a start date for a one-day retrieval. 532 533 533 534 Checks if end date lies after start date and end date is set. … … 628 629 629 630 def check_request(request, marsfile): 630 '''Check if there is an old mars request file andremove it.631 '''Check if there is an old MARS request file; if so, remove it. 631 632 632 633 Parameters … … 651 652 652 653 def check_public(public, dataset): 653 '''Check wether the dataset parameter is set fora654 public data set retrieval.654 '''Check wether the dataset parameter is set to a 655 public data set. 655 656 656 657 Parameters … … 667 668 ''' 668 669 if public and not dataset: 669 raise ValueError('ERROR: If public mars data wantsto be retrieved, '670 'the "dataset"-parameter has to be set too!')670 raise ValueError('ERROR: If public MARS data are to be retrieved, ' 671 'the "dataset"-parameter has to be set, too!') 671 672 return 672 673 … … 674 675 '''Guarantees that the accumulation field type is set. 675 676 676 If not set, it is deriv ated as inthe old method (TYPE[1]).677 If not set, it is derived with the old method (TYPE[1]). 677 678 678 679 Parameters … … 710 711 '''Guarantees that the accumulation forecast times were set. 711 712 712 If it is not set, it tries to set the value for some of the713 most commonly used data sets. Otherwise it raises an error.713 If not set, setting the value to some of the most commonly used data sets 714 is attempted. Otherwise, an eror is raised. 714 715 715 716 Parameters 716 717 ---------- 717 718 acctime : str 718 The starting time f romthe accumulated forecasts.719 The starting time for the accumulated forecasts. 719 720 720 721 marsclass : str … … 727 728 ------ 728 729 acctime : str 729 The starting time f romthe accumulated forecasts.730 The starting time for the accumulated forecasts. 730 731 ''' 731 732 732 733 if not acctime: 733 print('... Control parameter ACCTIME was not defined.')734 print('... Control parameter ACCTIME was not set.') 734 735 print('... Value will be set depending on field type:\n ' 735 736 '\t\t EA=06/18\n\t\t EI/OD=00/12\n\t\t EP=18') … … 745 746 acctime = time[0] 746 747 else: 747 raise ValueError('ERROR: Accumulation forecast time can not '748 ' automatically be derived!')748 raise ValueError('ERROR: Accumulation forecast time can not be' 749 'derived automatically!') 749 750 return acctime 750 751 751 752 def check_accmaxstep(accmaxstep, marsclass, purefc, maxstep): 752 '''Guarantees that the accumulation forecast step w ereset.753 '''Guarantees that the accumulation forecast step was set. 753 754 754 755 Parameters … … 765 766 maxstep : str 766 767 The maximum forecast time step in hours from the forecast base time. 767 This is the maximum step for non 768 This is the maximum step for non-flux (accumulated) forecast data. 768 769 769 770 Return … … 773 774 ''' 774 775 if not accmaxstep: 775 print('... Control parameter ACCMAXSTEP was not defined.')776 print('... Control parameter ACCMAXSTEP was not set.') 776 777 print('... Value will be set depending on field type/time: ' 777 778 '\n\t\t EA/EI/OD=12\n\t\t EP=24') … … 785 786 print('... For pure forecast mode, the accumulated forecast must ' 786 787 'have the same maxstep as the normal forecast fields!\n' 787 '\t\t A ccmaxstep was set to maxstep!')788 '\t\t ACCMAXSTEP was set to MAXSTEP!') 788 789 else: 789 raise ValueError('ERROR: Accumulation forecast step can not '790 ' automatically be derived!')790 raise ValueError('ERROR: Accumulation forecast step can not be' 791 'derived automatically!') 791 792 else: 792 793 if purefc and int(accmaxstep) != int(maxstep): … … 794 795 print('... For pure forecast mode, the accumulated forecast must ' 795 796 'have the same maxstep as the normal forecast fields!\n' 796 '\t\t A ccmaxstep was set to maxstep!')797 '\t\t ACCMAXSTEP was set to MAXSTEP!') 797 798 return accmaxstep 798 799 799 800 def check_addpar(addpar): 800 801 '''Check that addpar has correct format of additional parameters in 801 a single string, so that it can be easily appended to the hard 802 parameters that areretrieved in any case.802 a single string, so that it can be easily appended to the hard-coded 803 parameters retrieved in any case. 803 804 804 805 Parameters … … 826 827 827 828 def check_job_chunk(job_chunk): 828 '''Checks that if job chunk is set, the number is positive and non 829 '''Checks that if job chunk is set, the number is positive and nonzero. 829 830 830 831 Parameters … … 844 845 845 846 if job_chunk < 0: 846 raise ValueError('ERROR: The number of job chunk is negative!\n'847 raise ValueError('ERROR: The number of job chunks is negative!\n' 847 848 'It has to be a positive number!') 848 849 elif job_chunk == 0: -
Source/Python/Mods/get_mars_data.py
r0f89116 r8209738 10 10 # November 2015 - Leopold Haimberger (University of Vienna): 11 11 # - moved the getEIdata program into a function "get_mars_data" 12 # - moved the AgurmentParser into a sep erate function13 # - ada tpted the function for theuse in flex_extract14 # - renamed file to get_mars_data12 # - moved the AgurmentParser into a separate function 13 # - adapted the function for use in flex_extract 14 # - renamed source file to get_mars_data 15 15 # 16 16 # February 2018 - Anne Philipp (University of Vienna): … … 22 22 # online documentation) 23 23 # - use of UIFiles class for file selection and deletion 24 # - sep erated get_mars_data function into several smaller pieces:24 # - separated get_mars_data function into several smaller pieces: 25 25 # write_reqheader, mk_server, mk_dates, remove_old, do_retrievment 26 26 # … … 36 36 # Creative Commons, PO Box 1866, Mountain View, CA 94042, USA. 37 37 #******************************************************************************* 38 '''This script extracts MARS data from ECMWF servers.38 '''This script extracts MARS data from ECMWF. 39 39 40 40 At first, the necessary parameters from command line and CONTROL files are … … 44 44 functions: 45 45 46 * main - the main function of the script47 * get_mars_data - overall control of ECMWF data retrievment46 * main - the main function of the script 47 * get_mars_data - overall control of ECMWF data retrievment 48 48 * write_reqheader - writes the header into the mars_request file 49 * mk_server - creates the server connection to ECMWF servers50 * mk_dates - defines the start and end date51 * remove_old - deletes old retrieved grib files52 * do_retriev ement- creates individual retrievals53 54 Type :get_mars_data.py --help49 * mk_server - creates the server connection to ECMWF servers 50 * mk_dates - defines the start and end date 51 * remove_old - deletes old retrieved grib files 52 * do_retrieval - creates individual retrievals 53 54 Type get_mars_data.py --help 55 55 to get information about command line parameters. 56 56 Read the documentation for usage instructions. … … 66 66 from datetime import datetime, timedelta 67 67 68 # software 69 # add path to local main python path for flex_extract to get full access68 # software-specific classes and modules from flex_extract 69 # add path to local main Python path for flex_extract to get full access 70 70 sys.path.append(os.path.dirname(os.path.abspath( 71 71 inspect.getfile(inspect.currentframe()))) + '/../') … … 94 94 # ------------------------------------------------------------------------------ 95 95 def main(): 96 '''Controls the program to get data out of mars.97 98 This is done if it iscalled directly from command line.99 Then it also takes program call arguments and control fileinput.96 '''Controls the program to retrieve data from MARS. 97 98 This is done if called directly from command line. 99 Then, arguments and control file are taken as input. 100 100 101 101 Parameters … … 114 114 115 115 def get_mars_data(c): 116 '''Retrieves the EC data needed for a FLEXPART simulation.117 118 Start and end dates for retrieval period isset. Retrievals119 are divided into s maller periods if necessary anddatechunk parameter116 '''Retrieves the ECMWF data required for a FLEXPART simulation. 117 118 Start and end dates for retrieval period are set. Retrievals 119 are divided into shorter periods if necessary and if datechunk parameter 120 120 is set. 121 121 … … 137 137 138 138 if c.request == 0: 139 print("Retrieving EC data!")139 print("Retrieving ECMWF data!") 140 140 else: 141 141 if c.request == 1: 142 print("Printing marsrequests!")142 print("Printing MARS requests!") 143 143 elif c.request == 2: 144 print("Retrieving EC data and printing marsrequest!")144 print("Retrieving ECMWF data and printing MARS request!") 145 145 write_reqheader(os.path.join(c.inputdir, _config.FILE_MARS_REQUESTS)) 146 146 … … 165 165 166 166 def write_reqheader(marsfile): 167 '''Writes header with column names into marsrequest file.167 '''Writes header with column names into MARS request file. 168 168 169 169 Parameters 170 170 ---------- 171 171 marsfile : str 172 Path to the marsrequest file.172 Path to the MARS request file. 173 173 174 174 Return … … 188 188 189 189 def mk_server(c): 190 '''Creates a server connection with available python API. 191 192 Which API is used depends on availability and the dataset to be retrieved. 193 The CDS API is used for ERA5 dataset no matter if the user is a member or 194 a public user. ECMWF WebAPI is used for all other available datasets. 190 '''Creates a server connection with available Python API. 191 192 The API selected depends on availability and the data set to be retrieved. 193 The CDS API is used for ERA5 data, no matter whether the user is a 194 member-state or a public user. 195 ECMWF WebAPI is used for all other available datasets. 195 196 196 197 Parameters … … 247 248 248 249 def mk_dates(c, fluxes): 249 '''Prepares start and end date depending on flux or non fluxdata.250 251 If forecast for maximum one day (upto 24h) are to be retrieved, then250 '''Prepares start and end date depending on flux or non-flux type of data. 251 252 If forecasts for a maximum of one day (24 h) are to be retrieved, then 252 253 collect accumulation data (flux data) with additional days in the 253 beginning and at the end ( used for complete disaggregation of254 beginning and at the end (needed for complete disaggregation of 254 255 original period) 255 256 256 If forecast data longer than 24h are to be retrieved, then257 If forecast data for more than +24 h are to be retrieved, then 257 258 collect accumulation data (flux data) with the exact start and end date 258 259 (disaggregation will be done for the exact time period with … … 314 315 ---------- 315 316 pattern : str 316 The sub 317 The substring pattern which identifies the files to be deleted. 317 318 318 319 inputdir : str, optional 319 Path to the directory where the retrieved data isstored.320 321 Return 322 ------ 323 324 ''' 325 print('... removing old content of' + inputdir)320 Path to the directory where the retrieved data are stored. 321 322 Return 323 ------ 324 325 ''' 326 print('... removing old files in ' + inputdir) 326 327 327 328 tobecleaned = UioFiles(inputdir, pattern) … … 332 333 333 334 def do_retrievement(c, server, start, end, delta_t, fluxes=False): 334 '''Divides the complete retrieval period insmaller chunks and335 '''Divides the total retrieval period into smaller chunks and 335 336 retrieves the data from MARS. 336 337 … … 351 352 352 353 delta_t : datetime 353 Delta_t + 1 is the maximal time period of a single 354 retrieval. 354 Delta_t + 1 is the maximum time period of a single retrieval. 355 355 356 356 fluxes : boolean, optional … … 365 365 366 366 # since actual day also counts as one day, 367 # we only need to add datechunk - 1 days to retrieval 368 # for a period 367 # we only need to add datechunk - 1 days to retrieval for a period 369 368 delta_t_m1 = delta_t - timedelta(days=1) 370 369 -
Source/Python/Mods/prepare_flexpart.py
r0f89116 r8209738 13 13 # - job templates suitable for twice daily operational dissemination 14 14 # - dividing retrievals of longer periods into digestable chunks 15 # - retrieve also longer 16 # short 15 # - retrieve also longer-term forecasts, not only analyses and 16 # short-term forecast data 17 17 # - conversion into GRIB2 18 # - conversion into .fp format for faster execution of FLEXPART18 # 19 19 # 20 20 # February 2018 - Anne Philipp (University of Vienna): … … 45 45 46 46 It converts the bunch of grib files extracted via get_mars_data before, 47 by doing the necessary conversion to getconsistent grids or the47 by doing the necessary conversion to obtain consistent grids or the 48 48 disaggregation of flux data. Finally, the data fields are combined 49 in files per available hourwith the naming convention xxYYMMDDHH,49 in files per hour available with the naming convention xxYYMMDDHH, 50 50 where xx should be 2 arbitrary letters (mostly xx is chosen to be "EN"). 51 51 … … 88 88 # ------------------------------------------------------------------------------ 89 89 def main(): 90 '''Controls the program to prepare flexpart input files from marsdata.90 '''Controls the program to prepare FLEXPART input files from MARS data. 91 91 92 This is done if it is called directly fromcommand line.93 Then it also takes program call arguments and control fileinput.92 This is done if called directly from the command line. 93 Then, arguments and control file are taken as input. 94 94 95 95 Parameters … … 108 108 109 109 def prepare_flexpart(ppid, c): 110 '''Converts the mars data into flexpart ready input files.110 '''Converts the MARS data into files ready as input for FLEXPART. 111 111 112 Specific data fields are converted to a different grid and the flux 113 data are going to be disaggregated. The data fields are collected by 114 hour and stored in a file with a specific FLEXPART relevant naming 115 convention. 112 Certain fields are converted to a different grid and the flux 113 data are disaggregated. Fields are collected by hour and stored in a file 114 with a specific naming convention. 116 115 117 116 Parameters -
Source/Python/Mods/tools.py
rd1bfa24 r8209738 18 18 # - added documentation 19 19 # - moved all non class methods from former file Flexparttools in here 20 # - sep erated args and control interpretation20 # - separated args and control interpretation 21 21 # - added functions get_list_as_string, read_ecenv, send_mail, make_dir, 22 22 # put_file_to_ecserver, submit_job_to_ecserver, get_informations, … … 118 118 119 119 def none_or_str(value): 120 '''Converts the input string into pythons None-type if the string121 contains string "None".120 '''Converts the input string into Pythons None type if it 121 contains the string "None". 122 122 123 123 Parameters … … 130 130 None or value: 131 131 Return depends on the content of the input value. If it was "None", 132 then the python type None is returned. Otherwise the string itself.132 then the Python type None is returned, otherwise the string itself. 133 133 ''' 134 134 if value == 'None': … … 137 137 138 138 def none_or_int(value): 139 '''Converts the input string into pythons None-type if the string140 contains string "None" . Otherwise it is converted to an integer value.139 '''Converts the input string into Pythons None-type if it 140 contains string "None"; otherwise it is converted to an integer value. 141 141 142 142 Parameters … … 158 158 def get_cmdline_args(): 159 159 '''Decomposes the command line arguments and assigns them to variables. 160 Apply default values for non mentioned arguments.160 Apply default values for arguments not present. 161 161 162 162 Parameters … … 166 166 ------ 167 167 args : Namespace 168 Contains the command line arguments from script/program call.168 Contains the command line arguments from the script / program call. 169 169 ''' 170 170 … … 191 191 parser.add_argument("--basetime", dest="basetime", 192 192 type=none_or_int, default=None, 193 help="base such as 0 or 12 (for half day retrievals)")193 help="base time such as 0 or 12 (for half day retrievals)") 194 194 parser.add_argument("--step", dest="step", 195 195 type=none_or_str, default=None, … … 200 200 parser.add_argument("--area", dest="area", 201 201 type=none_or_str, default=None, 202 help="area defined asnorth/west/south/east")202 help="area, defined by north/west/south/east") 203 203 204 204 # some switches 205 205 parser.add_argument("--debug", dest="debug", 206 206 type=none_or_int, default=None, 207 help="debug mode - leave temporary files intact")207 help="debug mode - temporary files will be conserved") 208 208 parser.add_argument("--oper", dest="oper", 209 209 type=none_or_int, default=None, 210 help='operational mode - prepares dates with'210 help='operational mode - prepares dates from ' 211 211 'environment variables') 212 212 parser.add_argument("--request", dest="request", 213 213 type=none_or_int, default=None, 214 help="list all marsrequests in file mars_requests.dat")214 help="list all MARS requests in file mars_requests.dat") 215 215 parser.add_argument("--public", dest="public", 216 216 type=none_or_int, default=None, 217 help="public mode - retrieves thepublic datasets")217 help="public mode - retrieves public datasets") 218 218 parser.add_argument("--rrint", dest="rrint", 219 219 type=none_or_int, default=None, 220 help='Selection of old or new precipitation'221 'interpolation :\n'220 help='Selection of old or new ' 221 'interpolation method for precipitation:\n' 222 222 ' 0 - old method\n' 223 223 ' 1 - new method (additional subgrid points)') … … 226 226 parser.add_argument("--inputdir", dest="inputdir", 227 227 type=none_or_str, default=None, 228 help='Path to t he temporary directory for the'229 'retriev algrib files and other processing files.')228 help='Path to temporary directory for ' 229 'retrieved grib files and other processing files.') 230 230 parser.add_argument("--outputdir", dest="outputdir", 231 231 type=none_or_str, default=None, 232 help='Path to the final directory where the final'233 'FLEXPART ready input files are stored.')232 help='Path to final directory where ' 233 'FLEXPART input files will be stored.') 234 234 235 235 # this is only used by prepare_flexpart.py to rerun a postprocessing step 236 236 parser.add_argument("--ppid", dest="ppid", 237 237 type=none_or_str, default=None, 238 help='This is the specify parent process id of a '238 help='This is the specify the parent process id of a ' 239 239 'single flex_extract run to identify the files. ' 240 240 'It is the second number in the GRIB files.') … … 243 243 parser.add_argument("--job_template", dest='job_template', 244 244 type=none_or_str, default="job.temp", 245 help='The job template file which are adapted to be ' 246 'submitted to the batch system on ECMWF server.') 245 help='Job template file. Will be used for submission ' 246 'to the batch system on the ECMWF server after ' 247 'modification.') 247 248 parser.add_argument("--queue", dest="queue", 248 249 type=none_or_str, default=None, 249 help='The ECMWF server name for submission of the'250 'job script to the batch system '250 help='The name of the ECMWF server name where the' 251 'job script is to be submitted ' 251 252 '(e.g. ecgate | cca | ccb)') 252 253 … … 289 290 290 291 It keeps the final FLEXPART input files if program runs without 291 ECMWF A piand keywords "ectrans" or "ecstorage" are set to "1".292 ECMWF API and keywords "ectrans" or "ecstorage" are set to "1". 292 293 293 294 Parameters … … 410 411 411 412 This method combines the single characters of the passed arguments 412 with each other . Sothat each character of each argument value413 with each other in a way that each character of each argument value 413 414 will be combined with each character of the other arguments as a tuple. 414 415 … … 519 520 pars : str 520 521 Addpar argument from CONTROL file in the format of 521 parameter names instead of ids. The parameter short 522 names are sepearted with "/" and they are passed as 523 one single string. 522 parameter names instead of IDs. The parameter short 523 names are separated by "/" and passed as one single string. 524 524 525 525 table : dict … … 555 555 556 556 def to_param_id_with_tablenumber(pars, table): 557 '''Transform parameter names to parameter ids and add table id.557 '''Transform parameter names to parameter IDs and add table ID. 558 558 559 559 Conversion with ECMWF grib table 128. … … 563 563 pars : str 564 564 Addpar argument from CONTROL file in the format of 565 parameter names instead of ids. The parameter short 566 names are sepearted with "/" and they are passed as 567 one single string. 565 parameter names instead of ID. The parameter short 566 names are separated by "/" and passed as one single string. 568 567 569 568 table : dict … … 576 575 spar : str 577 576 List of addpar parameters from CONTROL file transformed to 578 parameter ids in the format of integer.577 parameter IDs in the format of integer. 579 578 ''' 580 579 if not pars: … … 598 597 599 598 def get_list_as_string(list_obj, concatenate_sign=', '): 600 '''Converts a list of arbitrary content into a single string. 599 '''Converts a list of arbitrary content into a single string using a given 600 concatenation character. 601 601 602 602 Parameters … … 624 624 '''Creates a directory. 625 625 626 I t gives a warning if the directory already exists and skips process.627 The program stops only if there is another problem.626 If the directory already exists, an information is printed and the creation 627 skipped. The program stops only if there is another problem. 628 628 629 629 Parameters … … 641 641 # errno.EEXIST = directory already exists 642 642 if e.errno == errno.EEXIST: 643 print(' WARNING: Directory {0} already exists!'.format(directory))643 print('INFORMATION: Directory {0} already exists!'.format(directory)) 644 644 else: 645 645 raise # re-raise exception if a different error occured … … 720 720 ------ 721 721 job_id : int 722 The id number of the job as a reference at the ecmwfserver.722 The id number of the job as a reference at the ECMWF server. 723 723 ''' 724 724 … … 732 732 733 733 print('\n... Do you have a valid ecaccess certification key?') 734 sys.exit('... ECACCESS-JOB-SUBMITFAILED!')734 sys.exit('... ecaccess-job-submit FAILED!') 735 735 except OSError as e: 736 736 print('... ERROR CODE: ' + str(e.errno)) … … 738 738 739 739 print('\n... Most likely the ECACCESS library is not available!') 740 sys.exit('... ECACCESS-JOB-SUBMITFAILED!')740 sys.exit('... ecaccess-job-submit FAILED!') 741 741 742 742 return job_id.decode() … … 744 744 745 745 def get_informations(filename): 746 ''' Gets basic information from an example grib file.747 748 Th ese information are importantfor later use and the749 initialization of numpy arrays for data storing.746 '''Extracts basic information from a sample grib file. 747 748 This information is needed for later use and the 749 initialization of numpy arrays where data are stored. 750 750 751 751 Parameters … … 768 768 769 769 # --- open file --- 770 print("Opening file for getting information data--- %s" % filename)770 print("Opening grib file for extraction of information --- %s" % filename) 771 771 with open(filename, 'rb') as f: 772 772 # load first message from file … … 785 785 ] 786 786 787 print('\nInformation s are: ')787 print('\nInformation extracted: ') 788 788 for key in keys: 789 789 # Get the value of the key in a grib message. … … 798 798 799 799 def get_dimensions(info, purefc, dtime, index_vals, start_date, end_date): 800 '''This function specifies the correct dimensions for x, y and t.800 '''This function specifies the correct dimensions for x, y, and t. 801 801 802 802 Parameters … … 817 817 index_vals : list of list of str 818 818 Contains the values from the keys used for a distinct selection 819 of grib messages in processingthe grib files.819 of GRIB messages in processing the grib files. 820 820 Content looks like e.g.: 821 821 index_vals[0]: ('20171106', '20171107', '20171108') ; date … … 849 849 850 850 def execute_subprocess(cmd_list, error_msg='SUBPROCESS FAILED!'): 851 '''Executes a command line instructionvia a subprocess.851 '''Executes a command via a subprocess. 852 852 853 853 Error handling is done if an error occures. … … 856 856 ---------- 857 857 cmd_list : list of str 858 A list of the components for the command line execution. Each 859 list entry is a single part of the command which is seperated from 860 the rest by a blank space. 861 E.g. ['mv', file1, file2] 858 A list of the components for the command line execution. 859 They will be concatenated with blank space for the command 860 to be submitted, like ['mv', file1, file2] for mv file1 file2. 862 861 863 862 Return 864 863 ------ 865 864 error_msg : str, optional 866 The possible error message if the subprocess failed.867 By default it will just tell"SUBPROCESS FAILED!".865 Error message if the subprocess fails. 866 By default it will just say "SUBPROCESS FAILED!". 868 867 ''' 869 868 -
Source/Python/install.py
r50f9ca6 r8209738 46 46 47 47 Depending on the selected installation environment (locally or on the 48 ECMWF server ecgate or cca) the program extracts the command line48 ECMWF server ecgate or cca) the program extracts the command line 49 49 arguments and the CONTROL file parameter and prepares the corresponding 50 50 environment. 51 The necessary files are collected in a tar -ball and placed52 at the target location. There its untared, the environment variables will53 be set and the Fortran code will becompiled.54 If the ECMWF environment is selected a job script is prepared and submitted55 for the remaining configurations after putting the tar -ball tothe51 The necessary files are collected in a tar ball and placed 52 at the target location. There, is is untared, the environment variables are 53 set, and the Fortran code is compiled. 54 If the ECMWF environment is selected, a job script is prepared and submitted 55 for the remaining configurations after putting the tar ball on the 56 56 target ECMWF server. 57 57 … … 107 107 def get_install_cmdline_args(): 108 108 '''Decomposes the command line arguments and assigns them to variables. 109 Apply default values for non mentioned arguments.109 Apply default values for arguments not present. 110 110 111 111 Parameters … … 127 127 parser.add_argument("--makefile", dest="makefile", 128 128 type=none_or_str, default=None, 129 help='Name of Makefile to use for compiling the '129 help='Name of makefile for compiling the ' 130 130 'Fortran program') 131 131 parser.add_argument("--ecuid", dest="ecuid", 132 132 type=none_or_str, default=None, 133 help=' The user id at ECMWF.')133 help='User id at ECMWF') 134 134 parser.add_argument("--ecgid", dest="ecgid", 135 135 type=none_or_str, default=None, 136 help=' The group id at ECMWF.')136 help='Group id at ECMWF') 137 137 parser.add_argument("--gateway", dest="gateway", 138 138 type=none_or_str, default=None, 139 help=' The name of the local gateway server.')139 help='Name of the local gateway server') 140 140 parser.add_argument("--destination", dest="destination", 141 141 type=none_or_str, default=None, 142 help=' Theecaccess association, e.g. '142 help='ecaccess association, e.g. ' 143 143 'myUser@genericSftp') 144 144 145 145 parser.add_argument("--installdir", dest="installdir", 146 146 type=none_or_str, default=None, 147 help='Root directory where '148 'flex_extract will be installed to.')147 help='Root directory of the ' 148 'flex_extract installation') 149 149 150 150 # arguments for job submission to ECMWF, only needed by submit.py 151 151 parser.add_argument("--job_template", dest='job_template', 152 152 type=none_or_str, default="job.template", 153 help=' The rudimentary template file to create a batch '154 'job template for submission to ECMWF servers .')153 help='Rudimentary template file to create a batch ' 154 'job template for submission to ECMWF servers') 155 155 156 156 parser.add_argument("--controlfile", dest="controlfile", 157 157 type=none_or_str, default='CONTROL_EA5', 158 help=" The file withall CONTROL parameters.")158 help="A file that contains all CONTROL parameters.") 159 159 160 160 args = parser.parse_args() … … 164 164 165 165 def install_via_gateway(c): 166 '''Prepare data transfer to remote gate and submit a job script which will167 install everything on the remote gate .166 '''Prepare data transfer to remote gateway and submit a job script which will 167 install everything on the remote gateway. 168 168 169 169 Parameters … … 259 259 260 260 def check_install_conditions(c): 261 '''Checks a couple ofnecessary attributes and conditions262 for the installation such as ifthey exist and contain values.261 '''Checks necessary attributes and conditions 262 for the installation, e.g. whether they exist and contain values. 263 263 Otherwise set default values. 264 264 … … 310 310 '''Creates a tarball with all necessary files which need to be sent to the 311 311 installation directory. 312 It does not matter ifthis is local or remote.313 Collects all python files, the Fortran source and makefiles,312 It does not matter whether this is local or remote. 313 Collects all Python files, the Fortran source and makefiles, 314 314 the ECMWF_ENV file, the CONTROL files as well as the 315 315 template files. … … 497 497 ---------- 498 498 makefile : str 499 Name of the makefile which should be used to compile FORTRAN500 CONVERT2program.499 Name of the makefile which should be used to compile the Fortran 500 program. 501 501 502 502 ecuid : str … … 686 686 print(perr.decode()) 687 687 print('Please edit ' + makefile + 688 ' or try another Makefile in the src directory.')688 ' or try another makefile in the src directory.') 689 689 print('Most likely GRIB_API_INCLUDE_DIR, GRIB_API_LIB ' 690 690 'and EMOSLIB must be adapted.') 691 print('Available Makefiles:')692 print(UioFiles(src_path, ' Makefile*'))691 print('Available makefiles:') 692 print(UioFiles(src_path, 'makefile*')) 693 693 sys.exit('Compilation failed!') 694 694 except ValueError as e: 695 print('ERROR: Makefile callfailed:')695 print('ERROR: make of Fortran code failed:') 696 696 print(e) 697 697 else: 698 execute_subprocess(['ls', '-l', 698 execute_subprocess(['ls', '-l', 699 699 os.path.join(src_path, _config.FORTRAN_EXECUTABLE)], 700 700 error_msg='FORTRAN EXECUTABLE COULD NOT BE FOUND!')
Note: See TracChangeset
for help on using the changeset viewer.