Changeset 6f951ca in flex_extract.git
- Timestamp:
- Jan 15, 2019, 1:05:10 AM (5 years ago)
- Branches:
- master, ctbto, dev
- Children:
- d4696e0
- Parents:
- 2625ca8
- Location:
- source/python
- Files:
-
- 12 edited
Legend:
- Unmodified
- Added
- Removed
-
source/python/_config.py
r1eca806 r6f951ca 1 1 #!/usr/bin/env python 2 2 # -*- coding: utf-8 -*- 3 #******************************************************************************* 4 # @Author: Anne Philipp (University of Vienna) 5 # 6 # @Date: August 2018 7 # 8 # @Change History: 9 # 10 # @License: 11 # (C) Copyright 2014-2019. 12 # Anne Philipp, Leopold Haimberger 13 # 14 # This work is licensed under the Creative Commons Attribution 4.0 15 # International License. To view a copy of this license, visit 16 # http://creativecommons.org/licenses/by/4.0/ or send a letter to 17 # Creative Commons, PO Box 1866, Mountain View, CA 94042, USA. 18 #******************************************************************************* 19 '''Configurations for flex_extract. 20 21 Defines a couple of global constant parameters as well as filenames 22 and pathes. 3 23 ''' 4 *******************************************************************************5 @Author: Anne Philipp (University of Vienna)6 7 @Date: August 20188 9 @Change History:10 11 @License:12 (C) Copyright 2014-2018.13 14 This software is licensed under the terms of the Apache Licence Version 2.015 which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.16 17 @Description:18 Contains constant value parameter for flex_extract.19 20 *******************************************************************************21 '''22 23 24 # ------------------------------------------------------------------------------ 24 25 # MODULES … … 95 96 # 96 97 # ------------------------------------------------------------------------------ 97 # for making the installation tar ball, the relative pathes to the 98 # flex_extract root directory are needed 98 # RELATIVE PATHES FOR INSTALLATION TAR-BALL 99 99 # ------------------------------------------------------------------------------ 100 100 -
source/python/classes/ControlFile.py
rf2616a3 r6f951ca 21 21 # 22 22 # @License: 23 # (C) Copyright 2015-2018. 23 # (C) Copyright 2014-2019. 24 # Anne Philipp, Leopold Haimberger 24 25 # 25 # This software is licensed under the terms of the Apache Licence Version 2.0 26 # which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. 26 # This work is licensed under the Creative Commons Attribution 4.0 27 # International License. To view a copy of this license, visit 28 # http://creativecommons.org/licenses/by/4.0/ or send a letter to 29 # Creative Commons, PO Box 1866, Mountain View, CA 94042, USA. 27 30 # 28 # @Class Description: 29 # The CONTROL file is the steering part of the FLEXPART extraction 30 # software. All necessary parameters needed to retrieve the data fields 31 # from the MARS archive for driving FLEXPART are set in a CONTROL file. 32 # Some specific parameters like the start and end dates can be overwritten 33 # by the command line parameters, but in generel all parameters needed 34 # for a complete set of fields for FLEXPART can be set in the CONTROL file. 35 # 36 # @Class Content: 37 # - __init__ 38 # - __read_controlfile__ 39 # - __str__ 40 # - assign_args_to_control 41 # - assign_envs_to_control 42 # - check_conditions 43 # - check_install_conditions 44 # - to_list 45 # 46 # @Class Attributes: 47 # 48 # 31 # @Class Methods: 32 # __init__ 33 # _read_controlfile 34 # __str__ 35 # assign_args_to_control 36 # assign_envs_to_control 37 # check_conditions 38 # check_install_conditions 39 # to_list 49 40 #******************************************************************************* 50 41 … … 76 67 ''' 77 68 Contains the information which are stored in the CONTROL files. 69 70 The CONTROL file is the steering part of the FLEXPART extraction 71 software. All necessary parameters needed to retrieve the data fields 72 from the MARS archive for driving FLEXPART are set in a CONTROL file. 73 Some specific parameters like the start and end dates can be overwritten 74 by the command line parameters, but in generel all parameters needed 75 for a complete set of fields for FLEXPART can be set in the CONTROL file. 76 77 Attributes 78 ---------- 79 controlfile : str 80 The name of the control file to be processed. Default value is the 81 filename passed to the init function when initialised. 82 83 start_date : str 84 The first day of the retrieval period. Default value is None. 85 86 end_date :str 87 The last day of the retrieval period. Default value is None. 88 89 date_chunk : int 90 Length of period for a single mars retrieval. Default value is 3. 91 92 dtime :str 93 The time step in hours. Default value is None. 94 95 basetime : str 96 The time for a half day retrieval. The 12 hours upfront are to be 97 retrieved. Default value is None. 98 99 maxstep : int 100 The maximum forecast step for non flux data. Default value is None. 101 102 type : list of str 103 List of field type per retrieving hour. Default value is None. 104 105 time : list of str 106 List of retrieving times in hours. Default valuer is None. 107 108 step : list of str or str 109 List of forecast time steps in hours for non flux data. 110 Default value is None. 111 112 acctype : str 113 The field type for the accumulated forecast fields. 114 Default value is None. 115 116 acctime : str 117 The starting time of the accumulated forecasts. Default value is None. 118 119 accmaxstep : int 120 The maximum forecast step for the accumulated forecast fields 121 (flux data). Default value is None. 122 123 marsclass : str 124 Characterisation of dataset. Default value is None. 125 126 dataset : str 127 For public datasets there is the specific naming and parameter 128 dataset which has to be used to characterize the type of 129 data. Default value is None. 130 131 stream : str 132 Identifies the forecasting system used to generate the data. 133 Default value is None. 134 135 number : str 136 Selects the member in ensemble forecast run. Default value is 'OFF'. 137 138 expver : str 139 The version number of the dataset. Default value is '1'. 140 141 gaussian : str 142 This parameter is deprecated and should no longer be used. 143 Specifies the desired type of Gaussian grid for the output. 144 Default value is an empty string ''. 145 146 grid : str 147 Specifies the output grid which can be either a Gaussian grid 148 or a Latitude/Longitude grid. Default value is None. 149 150 area : str 151 Specifies the desired sub-area of data to be extracted. 152 Default value is None. 153 154 left : str 155 The western most longitude of the area to be extracted. 156 Default value is None. 157 158 lower : str 159 The southern most latitude of the area to be extracted. 160 Default value is None. 161 162 upper : str 163 The northern most latitued of the area to be extracted. 164 Default value is None. 165 166 right : str 167 The eastern most longitude of the area to be extracted. 168 Default value is None. 169 170 level : str 171 Specifies the maximum level. Default value is None. 172 173 levelist : str 174 Specifies the required level list. Default value is None. 175 176 resol : str 177 Specifies the desired triangular truncation of retrieved data, 178 before carrying out any other selected post-processing. 179 Default value is None. 180 181 gauss : int 182 Switch to select gaussian fields (1) or regular lat/lon (0). 183 Default value is 0. 184 185 accuracy : int 186 Specifies the number of bits per value to be used in the 187 generated GRIB coded fields. Default value is 24. 188 189 omega : int 190 Switch to select omega retrieval (1) or not (0). Default value is 0. 191 192 omegadiff : int 193 Switch to decide to calculate Omega and Dps/Dt from continuity 194 equation for diagnostic purposes (1) or not (0). Default value is 0. 195 196 eta : int 197 Switch to select direct retrieval of etadot from MARS (1) or 198 wether it has to be calculated (0). Then Default value is 0. 199 200 etadiff : int 201 Switch to select calculation of etadot and Dps/Dt from continuity 202 equation for diagnostic purposes (1) or not (0). Default value is 0. 203 204 etapar : int 205 GRIB parameter Id for etadot fields. Default value is 77. 206 207 dpdeta : int 208 Switch to select multiplication of etadot with dpdeta. 209 Default value is 1. 210 211 smooth : int 212 Spectral truncation of ETADOT after calculation on Gaussian grid. 213 Default value is 0. 214 215 format : str 216 The format of the GRIB data. Default value is 'GRIB1'. 217 218 addpar : str 219 List of additional surface level ECMWF parameter to be retrieved. 220 Default value is None. 221 222 prefix : str 223 Prefix string for the final FLEXPART/FLEXTRA ready input files. 224 Default value is 'EN'. 225 226 cwc : int 227 Switch to select wether the sum of cloud liquid water content and 228 cloud ice water content should be retrieved. Default value is 0. 229 230 wrf : int 231 Switch to select further parameters for retrievment to support 232 WRF simulations. Default value is 0. 233 234 ecfsdir : str 235 Path to the ECMWF storage 'ectmp:/${USER}/econdemand/' 236 237 mailfail : list of str 238 Email list for sending error log files from ECMWF servers. 239 The email addresses should be seperated by a comma. 240 Default value is ['${USER}']. 241 242 mailops : list of str 243 Email list for sending operational log files from ECMWF servers. 244 The email addresses should be seperated by a comma. 245 Default value is ['${USER}']. 246 247 grib2flexpart : int 0 248 Switch to select generation of preprocessed FLEXPART files ".fp". 249 If it is selected, the program grib2flexpart will try 250 to convert the flex_extract output files into ".fp" format. 251 252 ecstorage : int 253 Switch to select storage of FLEXPART ready output files 254 in the ECFS file system. Default value is 0. 255 256 ectrans : int 257 Switch to select the transfer of FLEXPART ready output files 258 to the gateway server. Default value is 0. 259 260 inputdir : str 261 Path to the temporary directory for the retrieval grib files and 262 other processing files. Default value is _config.PATH_INPUT_DIR. 263 264 outputdir : str 265 Path to the final directory where the final FLEXPART ready input 266 files are stored. Default value is None. 267 268 flexextractdir : str 269 Path to the flex_extract root directory. Default value is 270 _config.PATH_FLEXEXTRACT_DIR. 271 272 exedir : str 273 Path to the FORTRAN executable file. Default value is 274 _config.PATH_FORTRAN_SRC. 275 276 flexpartdir : str 277 Path to a FLEXPART root directory. Default value is None. 278 279 makefile : str 280 Name of the makefile to be used for the Fortran program. 281 Default value is 'Makefile.gfortran'. 282 283 destination : str 284 The remote destination which is used to transfer files 285 from ECMWF server to local gateway server. Default value is None. 286 287 gateway : str 288 The gateway server the user is using. Default value is None. 289 290 ecuid : str 291 The user id on ECMWF server. Default value is None. 292 293 ecgid : str 294 The group id on ECMWF server. Default value is None. 295 296 install_target : str 297 Defines the location where the installation is to be done. 298 Default value is None. 299 300 debug : int 301 Switch to keep temporary files at the end of postprocessing (1) or 302 to delete all temporary files except the final output files (0). 303 Default value is 0. 304 305 request : int 306 Switch to select between just retrieving the data (0), writing the mars 307 parameter values to a csv file (1) or doing both (2). 308 Default value is 0. 309 310 public : int 311 Switch to select kind of ECMWF Web Api access and the 312 possible data sets. Public data sets (1) and Memberstate data sets (0). 313 Default value is 0. 314 315 ecapi : boolean 316 Tells wether the ECMWF Web APi was able to load or not. 317 Default value is None. 318 319 purefc : int 320 Switch to decide wether the job is a pure forecast retrieval or 321 coupled with analysis data. Default value is 0. 322 323 rrint: int 324 Switch to select between old precipitation disaggregation method (0) 325 or the new IA3 disaggegration method (1). Default value is 0. 326 327 logicals : list of str 328 List of the names of logical switches which controls the flow 329 of the program. Default list is ['gauss', 'omega', 'omegadiff', 'eta', 330 'etadiff', 'dpdeta', 'cwc', 'wrf', 'grib2flexpart', 'ecstorage', 331 'ectrans', 'debug', 'request', 'public', 'purefc', 'rrint'] 78 332 ''' 79 333 … … 85 339 Parameters 86 340 ---------- 87 filename : :obj:`string`341 filename : str 88 342 Name of CONTROL file. 89 343 … … 98 352 self.end_date = None 99 353 self.date_chunk = 3 354 self.job_chunk = None 100 355 self.dtime = None 101 356 self.basetime = None … … 273 528 Parameters 274 529 ---------- 275 args : :obj:`Namespace`530 args : Namespace 276 531 Contains the commandline arguments from script/program call. 277 532 … … 298 553 Parameters 299 554 ---------- 300 envs : :obj:`dictionary` of :obj:`strings`555 envs : dict of str 301 556 Contains the ECMWF environment parameternames "ECUID", "ECGID", 302 557 "DESTINATION" and "GATEWAY" with its corresponding values. … … 320 575 Parameters 321 576 ---------- 322 queue : :obj:`string`577 queue : str 323 578 Name of the queue if submitted to the ECMWF servers. 324 579 Used to check if ecuid, ecgid, gateway and destination … … 397 652 Return 398 653 ------ 399 l : :obj:`list`654 l : list of * 400 655 A sorted list of the all ControlFile class attributes with 401 656 their values except the attributes "_expanded", "exedir", -
source/python/classes/EcFlexpart.py
rbf48c8a r6f951ca 41 41 # 42 42 # @License: 43 # (C) Copyright 2014-2018. 43 # (C) Copyright 2014-2019. 44 # Anne Philipp, Leopold Haimberger 44 45 # 45 # This software is licensed under the terms of the Apache Licence Version 2.0 46 # which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. 47 # 48 # @Class Description: 49 # FLEXPART needs grib files in a specifc format. All necessary data fields 50 # for one time step are stored in a single file. The class represents an 51 # instance with all the parameter and settings necessary for retrieving 52 # MARS data and modifing them so they are fitting FLEXPART need. The class 53 # is able to disaggregate the fluxes and convert grid types to the one needed 54 # by FLEXPART, therefore using the FORTRAN program. 55 # 56 # @Class Content: 57 # - __init__ 58 # - write_namelist 59 # - retrieve 60 # - process_output 61 # - create 62 # - deacc_fluxes 63 # 64 # @Class Attributes: 65 # 66 # TODO 67 # 46 # This work is licensed under the Creative Commons Attribution 4.0 47 # International License. To view a copy of this license, visit 48 # http://creativecommons.org/licenses/by/4.0/ or send a letter to 49 # Creative Commons, PO Box 1866, Mountain View, CA 94042, USA. 68 50 #******************************************************************************* 69 51 #pylint: disable=unsupported-assignment-operation … … 102 84 class EcFlexpart(object): 103 85 ''' 104 Class to retrieve FLEXPART specific ECMWF data. 86 Class to represent FLEXPART specific ECMWF data. 87 88 FLEXPART needs grib files in a specifc format. All necessary data fields 89 for one time step are stored in a single file. The class represents an 90 instance with all the parameter and settings necessary for retrieving 91 MARS data and modifing them so they are fitting FLEXPART needs. The class 92 is able to disaggregate the fluxes and convert grid types to the one needed 93 by FLEXPART, therefore using the FORTRAN program. 94 95 Attributes 96 ---------- 97 mreq_count : int 98 Counter for the number of generated mars requests. 99 100 inputdir : str 101 Path to the directory where the retrieved data is stored. 102 103 dataset : str 104 For public datasets there is the specific naming and parameter 105 dataset which has to be used to characterize the type of 106 data. 107 108 basetime : str 109 The time for a half day retrieval. The 12 hours upfront are to be 110 retrieved. 111 112 dtime : str 113 Time step in hours. 114 115 acctype : str 116 The field type for the accumulated forecast fields. 117 118 acctime : str 119 The starting time from the accumulated forecasts. 120 121 accmaxstep : str 122 The maximum forecast step for the accumulated forecast fields. 123 124 marsclass : str 125 Characterisation of dataset. 126 127 stream : str 128 Identifies the forecasting system used to generate the data. 129 130 number : str 131 Selects the member in ensemble forecast run. 132 133 resol : str 134 Specifies the desired triangular truncation of retrieved data, 135 before carrying out any other selected post-processing. 136 137 accuracy : str 138 Specifies the number of bits per value to be used in the 139 generated GRIB coded fields. 140 141 addpar : str 142 List of additional parameters to be retrieved. 143 144 level : str 145 Specifies the maximum level. 146 147 expver : str 148 The version of the dataset. 149 150 levelist : str 151 Specifies the required levels. 152 153 glevelist : str 154 Specifies the required levels for gaussian grids. 155 156 gaussian : str 157 This parameter is deprecated and should no longer be used. 158 Specifies the desired type of Gaussian grid for the output. 159 160 grid : str 161 Specifies the output grid which can be either a Gaussian grid 162 or a Latitude/Longitude grid. 163 164 area : str 165 Specifies the desired sub-area of data to be extracted. 166 167 purefc : int 168 Switch for definition of pure forecast mode or not. 169 170 outputfilelist : list of str 171 The final list of FLEXPART ready input files. 172 173 types : dictionary 174 Determines the combination of type of fields, time and forecast step 175 to be retrieved. 176 177 params : dictionary 178 Collection of grid types and their corresponding parameters, 179 levels, level types and the grid definition. 180 181 server : ECMWFService or ECMWFDataServer 182 This is the connection to the ECMWF data servers. 183 184 public : int 185 Decides which Web API Server version is used. 186 187 dates : str 188 Contains start and end date of the retrieval in the format 189 "YYYYMMDD/to/YYYYMMDD" 105 190 ''' 191 106 192 # -------------------------------------------------------------------------- 107 193 # CLASS FUNCTIONS … … 113 199 Parameters: 114 200 ----------- 115 c : :obj:`ControlFile`201 c : ControlFile 116 202 Contains all the parameters of CONTROL file and 117 203 command line. 118 204 119 fluxes : :obj:`boolean`, optional205 fluxes : boolean, optional 120 206 Decides if the flux parameter settings are stored or 121 207 the rest of the parameter list. … … 174 260 Parameters: 175 261 ----------- 176 ftype : :obj:`list` of :obj:`string`262 ftype : list of str 177 263 List of field types. 178 264 179 ftime : :obj:`list` of :obj:`string`265 ftime : list of str 180 266 The time in hours of the field. 181 267 182 fstep : :obj:`string`268 fstep : str 183 269 Specifies the forecast time step from forecast base time. 184 270 Valid values are hours (HH) from forecast base time. … … 267 353 Parameters: 268 354 ----------- 269 gauss : :obj:`integer`355 gauss : int 270 356 Gaussian grid is retrieved. 271 357 272 eta : :obj:`integer`358 eta : int 273 359 Etadot parameter will be directly retrieved. 274 360 275 omega : :obj:`integer`361 omega : int 276 362 The omega paramterwill be retrieved. 277 363 278 cwc : :obj:`integer`364 cwc : int 279 365 The cloud liquid and ice water content will be retrieved. 280 366 281 wrf : :obj:`integer`367 wrf : int 282 368 Additional model level and surface level data will be retrieved for 283 369 WRF/FLEXPART-WRF simulations. … … 374 460 Parameters 375 461 ---------- 376 ftype : :obj:`string`462 ftype : str 377 463 Shortcut name of the type of the field. E.g. AN, FC, PF, ... 378 464 379 param : :obj:`string`465 param : str 380 466 Shortcut of the grid type. E.g. SH__ML, SH__SL, GG__ML, 381 467 GG__SL, OG__ML, OG__SL, OG_OROLSM_SL, OG_acc_SL 382 468 383 date : :obj:`string`469 date : str 384 470 The date period of the grib data to be stored in this file. 385 471 386 472 Return 387 473 ------ 388 targetname : :obj:`string`474 targetname : str 389 475 The target filename for the grib data. 390 476 ''' … … 401 487 Parameters 402 488 ---------- 403 request : :obj:`integer`489 request : int 404 490 Selects the mode of retrieval. 405 491 0: Retrieves the data from ECMWF. … … 407 493 2: Retrieves the data and prints the mars request. 408 494 409 par_dict : :obj:`dictionary`495 par_dict : dictionary 410 496 Contains all parameter which have to be set for creating the 411 497 Mars Retrievals. The parameter are: … … 463 549 Parameters 464 550 ---------- 465 keys : :obj:`dictionary`551 keys : dictionary 466 552 List of parameter names which serves as index. 467 553 468 inputfiles : :obj:`UioFiles`554 inputfiles : UioFiles 469 555 Contains a list of files. 470 556 471 557 Return 472 558 ------ 473 iid : :obj:`codes_index`559 iid : codes_index 474 560 This is a grib specific index structure to access 475 561 messages in a file. 476 562 477 index_vals : :obj:`list`563 index_vals : list of list of str 478 564 Contains the values from the keys used for a distinct selection 479 565 of grib messages in processing the grib files. … … 517 603 Parameters 518 604 ---------- 519 server : :obj:`ECMWFService` or :obj:`ECMWFDataServer`605 server : ECMWFService or ECMWFDataServer 520 606 The connection to the ECMWF server. This is different 521 607 for member state users which have full access and non … … 525 611 for member state users its False (ECMWFService) 526 612 527 dates : :obj:`string`613 dates : str 528 614 Contains start and end date of the retrieval in the format 529 615 "YYYYMMDD/to/YYYYMMDD" 530 616 531 request : :obj:`integer`617 request : int 532 618 Selects the mode of retrieval. 533 619 0: Retrieves the data from ECMWF. … … 535 621 2: Retrieves the data and prints the mars request. 536 622 537 inputdir : :obj:`string`, optional623 inputdir : str, optional 538 624 Path to the directory where the retrieved data is about 539 625 to be stored. The default is the current directory ('.'). … … 723 809 Parameters 724 810 ---------- 725 c : :obj:`ControlFile`811 c : ControlFile 726 812 Contains all the parameters of CONTROL file and 727 813 command line. 728 814 729 filename : :obj:`string`815 filename : str 730 816 Name of the namelist file. 731 817 … … 810 896 Parameters 811 897 ---------- 812 inputfiles : :obj:`UioFiles`898 inputfiles : UioFiles 813 899 Contains the list of files that contain flux data. 814 900 815 c : :obj:`ControlFile`901 c : ControlFile 816 902 Contains all the parameters of CONTROL file and 817 903 command line. … … 837 923 # index_vals[0]: ('20171106', '20171107', '20171108') ; date 838 924 # index_vals[1]: ('0', '600', '1200', '1800') ; time 839 # index_vals[2]: ('0', ' 12', '3', '6', '9') ; stepRange925 # index_vals[2]: ('0', '3', '6', '9', '12') ; stepRange 840 926 841 927 if c.rrint: … … 1096 1182 Parameters 1097 1183 ---------- 1098 ni : :obj:`integer`1184 ni : int 1099 1185 Amount of zonal grid points. 1100 1186 1101 nj : :obj:`integer`1187 nj : int 1102 1188 Amount of meridional grid points. 1103 1189 1104 nt : :obj:`integer`1190 nt : int 1105 1191 Number of time steps. 1106 1192 1107 lsp_np : :obj:`numpy array` of :obj:`float`1193 lsp_np : numpy array of float 1108 1194 The large scale precipitation fields for each time step. 1109 1195 Shape (ni * nj, nt). 1110 1196 1111 cp_np : :obj:`numpy array` of :obj:`float`1197 cp_np : numpy array of float 1112 1198 The convective precipitation fields for each time step. 1113 1199 Shape (ni * nj, nt). 1114 1200 1115 date_list : :obj:`list` of :obj:`datetime`1201 date_list : list of datetime 1116 1202 The list of dates for which the disaggregation is to be done. 1117 1203 1118 step_list : :obj:`list` of :obj:`integer`1204 step_list : list of int 1119 1205 The list of steps for a single forecast time. 1120 1206 Only necessary for pure forecasts. 1121 1207 1122 c : :obj:`ControlFile`1208 c : ControlFile 1123 1209 Contains all the parameters of CONTROL file and 1124 1210 command line. … … 1217 1303 Parameters 1218 1304 ---------- 1219 ifile : :obj:`string`1305 ifile : str 1220 1306 Filename of the input file to read the grib messages from. 1221 1307 1222 inputdir : :obj:`string`, optional1308 inputdir : str, optional 1223 1309 Path to the directory where the retrieved data is stored. 1224 1310 … … 1255 1341 Parameters 1256 1342 ---------- 1257 inputfiles : :obj:`UioFiles`1343 inputfiles : UioFiles 1258 1344 Contains a list of files. 1259 1345 1260 c : :obj:`ControlFile`1346 c : ControlFile 1261 1347 Contains all the parameters of CONTROL file and 1262 1348 command line. … … 1486 1572 Parameters 1487 1573 ---------- 1488 c : :obj:`ControlFile`1574 c : ControlFile 1489 1575 Contains all the parameters of CONTROL file and 1490 1576 command line. … … 1545 1631 Parameters 1546 1632 ---------- 1547 c : :obj:`ControlFile`1633 c : ControlFile 1548 1634 Contains all the parameters of CONTROL file and 1549 1635 command line. -
source/python/classes/MarsRetrieval.py
rbc1720f r6f951ca 17 17 # - added documentation 18 18 # - applied some minor modifications in programming style/structure 19 # - added writing of mars request attributes to a csv file 19 20 # 20 21 # @License: 21 # (C) Copyright 2015-2018. 22 # (C) Copyright 2014-2019. 23 # Anne Philipp, Leopold Haimberger 22 24 # 23 # This software is licensed under the terms of the Apache Licence Version 2.0 24 # which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. 25 # 26 # @Class Description: 27 # A MARS revtrieval has a specific syntax with a selection of keywords and 28 # their corresponding values. This class provides the necessary functions 29 # by displaying the selected parameters and their values and the actual 30 # retrievement of the data through a mars request or a Python web api 31 # interface. The initialization already expects all the keyword values. 32 # 33 # @Class Content: 34 # - __init__ 35 # - display_info 36 # - data_retrieve 37 # 38 # @Class Attributes: 39 # - server 40 # - marsclass 41 # - dtype 42 # - levtype 43 # - levelist 44 # - repres 45 # - date 46 # - resol 47 # - stream 48 # - area 49 # - time 50 # - step 51 # - expver 52 # - number 53 # - accuracy 54 # - grid 55 # - gaussian 56 # - target 57 # - param 58 # 25 # This work is licensed under the Creative Commons Attribution 4.0 26 # International License. To view a copy of this license, visit 27 # http://creativecommons.org/licenses/by/4.0/ or send a letter to 28 # Creative Commons, PO Box 1866, Mountain View, CA 94042, USA. 59 29 #******************************************************************************* 60 30 … … 73 43 # ------------------------------------------------------------------------------ 74 44 class MarsRetrieval(object): 75 '''Class for submitting MARS retrievals. 45 '''Specific syntax and content for submission of MARS retrievals. 46 47 A MARS revtrieval has a specific syntax with a selection of keywords and 48 their corresponding values. This class provides the necessary functions 49 by displaying the selected parameters and their values and the actual 50 retrievement of the data through a mars request or a Python web api 51 interface. The initialization already expects all the keyword values. 76 52 77 53 A description of MARS keywords/arguments and examples of their … … 79 55 https://software.ecmwf.int/wiki/display/UDOC/\ 80 56 Identification+keywords#Identificationkeywords-class 57 58 Attributes 59 ---------- 60 server : ECMWFService or ECMWFDataServer 61 This is the connection to the ECMWF data servers. 62 63 public : int 64 Decides which Web API Server version is used. 65 66 marsclass : str, optional 67 Characterisation of dataset. 68 69 dataset : str, optional 70 For public datasets there is the specific naming and parameter 71 dataset which has to be used to characterize the type of 72 data. 73 74 type : str, optional 75 Determines the type of fields to be retrieved. 76 77 levtype : str, optional 78 Denotes type of level. 79 80 levelist : str, optional 81 Specifies the required levels. 82 83 repres : str, optional 84 Selects the representation of the archived data. 85 86 date : str, optional 87 Specifies the Analysis date, the Forecast base date or 88 Observations date. 89 90 resol : str, optional 91 Specifies the desired triangular truncation of retrieved data, 92 before carrying out any other selected post-processing. 93 94 stream : str, optional 95 Identifies the forecasting system used to generate the data. 96 97 area : str, optional 98 Specifies the desired sub-area of data to be extracted. 99 100 time : str, optional 101 Specifies the time of the data in hours and minutes. 102 103 step : str, optional 104 Specifies the forecast time step from forecast base time. 105 106 expver : str, optional 107 The version of the dataset. 108 109 number : str, optional 110 Selects the member in ensemble forecast run. 111 112 accuracy : str, optional 113 Specifies the number of bits per value to be used in the 114 generated GRIB coded fields. 115 116 grid : str, optional 117 Specifies the output grid which can be either a Gaussian grid 118 or a Latitude/Longitude grid. 119 120 gaussian : str, optional 121 This parameter is deprecated and should no longer be used. 122 Specifies the desired type of Gaussian grid for the output. 123 124 target : str, optional 125 Specifies a file into which data is to be written after 126 retrieval or manipulation. 127 128 param : str, optional 129 Specifies the meteorological parameter. 81 130 ''' 82 131 … … 96 145 Parameters 97 146 ---------- 98 server : :obj:`ECMWFService`147 server : ECMWFService or ECMWFDataServer 99 148 This is the connection to the ECMWF data servers. 100 149 It is needed for the pythonic access of ECMWF data. 101 150 102 public : :obj:`integer`151 public : int 103 152 Decides which Web API version is used: 104 153 0: member-state users and full archive access … … 107 156 Default is "0" and for member-state users. 108 157 109 marsclass : :obj:`string`, optional158 marsclass : str, optional 110 159 Characterisation of dataset. E.g. EI (ERA-Interim), 111 160 E4 (ERA40), OD (Operational archive), ea (ERA5). 112 161 Default is the ERA-Interim dataset "ei". 113 162 114 dataset : :obj:`string`, optional163 dataset : str, optional 115 164 For public datasets there is the specific naming and parameter 116 165 dataset which has to be used to characterize the type of … … 119 168 Default is an empty string. 120 169 121 type : :obj:`string`, optional170 type : str, optional 122 171 Determines the type of fields to be retrieved. 123 172 Selects between observations, images or fields. … … 126 175 Default is an empty string. 127 176 128 levtype : :obj:`string`, optional177 levtype : str, optional 129 178 Denotes type of level. Has a direct implication on valid 130 179 levelist values! … … 134 183 Default is an empty string. 135 184 136 levelist : :obj:`string`, optional185 levelist : str, optional 137 186 Specifies the required levels. It has to have a valid 138 187 correspondence to the selected levtype. … … 140 189 Default is an empty string. 141 190 142 repres : :obj:`string`, optional191 repres : str, optional 143 192 Selects the representation of the archived data. 144 193 E.g. sh - spherical harmonics, gg - Gaussian grid, … … 146 195 Default is an empty string. 147 196 148 date : :obj:`string`, optional197 date : str, optional 149 198 Specifies the Analysis date, the Forecast base date or 150 199 Observations date. Valid formats are: … … 152 201 Default is an empty string. 153 202 154 resol : :obj:`string`, optional203 resol : str, optional 155 204 Specifies the desired triangular truncation of retrieved data, 156 205 before carrying out any other selected post-processing. … … 169 218 Default is an empty string. 170 219 171 stream : :obj:`string`, optional220 stream : str, optional 172 221 Identifies the forecasting system used to generate the data. 173 222 E.g. oper (Atmospheric model), enfo (Ensemble forecats), ... 174 223 Default is an empty string. 175 224 176 area : :obj:`string`, optional225 area : str, optional 177 226 Specifies the desired sub-area of data to be extracted. 178 227 Areas can be defined to wrap around the globe. … … 192 241 Default is an empty string. 193 242 194 time : :obj:`string`, optional243 time : str, optional 195 244 Specifies the time of the data in hours and minutes. 196 245 Valid values depend on the type of data: Analysis time, … … 202 251 Default is an empty string. 203 252 204 step : :obj:`string`, optional253 step : str, optional 205 254 Specifies the forecast time step from forecast base time. 206 255 Valid values are hours (HH) from forecast base time. It also … … 210 259 Default is an empty string. 211 260 212 expver : :obj:`string`, optional261 expver : str, optional 213 262 The version of the dataset. Each experiment is assigned a 214 263 unique code (version). Production data is assigned 1 or 2, … … 218 267 Default is "1". 219 268 220 number : :obj:`string`, optional269 number : str, optional 221 270 Selects the member in ensemble forecast run. (Only then it 222 271 is necessary.) It has a different meaning depending on … … 225 274 Default is an empty string. 226 275 227 accuracy : :obj:`string`, optional276 accuracy : str, optional 228 277 Specifies the number of bits per value to be used in the 229 278 generated GRIB coded fields. … … 236 285 Default is an empty string. 237 286 238 grid : :obj:`string`, optional287 grid : str, optional 239 288 Specifies the output grid which can be either a Gaussian grid 240 289 or a Latitude/Longitude grid. MARS requests specifying … … 257 306 Default is an empty string. 258 307 259 gaussian : :obj:`string`, optional308 gaussian : str, optional 260 309 This parameter is deprecated and should no longer be used. 261 310 Specifies the desired type of Gaussian grid for the output. … … 265 314 Default is an empty string. 266 315 267 target : :obj:`string`, optional316 target : str, optional 268 317 Specifies a file into which data is to be written after 269 318 retrieval or manipulation. Path names should always be … … 279 328 Default is an empty string. 280 329 281 param : :obj:`string`, optional330 param : str, optional 282 331 Specifies the meteorological parameter. 283 332 The list of meteorological parameters in MARS is extensive. … … 352 401 Parameters 353 402 ---------- 354 inputdir : :obj:`string`403 inputdir : str 355 404 The path where all data from the retrievals are stored. 356 405 357 request_number : :obj:`integer`406 request_number : int 358 407 Number of mars requests for flux and non-flux data. 359 408 -
source/python/classes/UioFiles.py
r5cb0eaa r6f951ca 12 12 # - added pattern search in method list_files 13 13 # 14 # February 2018 - Anne Philipp (University of Vienna):14 # February - December 2018 - Anne Philipp (University of Vienna): 15 15 # - applied PEP8 style guide 16 16 # - added documentation … … 20 20 # - modified the class so that it is initiated with a pattern instead 21 21 # of suffixes. Gives more precision in selection of files. 22 # - added delete method 22 23 # 23 24 # @License: 24 # (C) Copyright 2014-2018. 25 # (C) Copyright 2014-2019. 26 # Anne Philipp, Leopold Haimberger 25 27 # 26 # This software is licensed under the terms of the Apache Licence Version 2.0 27 # which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. 28 # 29 # @Class Decription: 30 # The class is for file manipulation. It is initiated with a regular 31 # expression pattern for this instance and can produce a list of Files 32 # from the given file pattern. These files can be deleted. 33 # 34 # @Class Content: 35 # - __init__ 36 # - __str__ 37 # - __list_files__ 38 # - delete_files 39 # 40 # @Class Attributes: 41 # - pattern 42 # - files 43 # 28 # This work is licensed under the Creative Commons Attribution 4.0 29 # International License. To view a copy of this license, visit 30 # http://creativecommons.org/licenses/by/4.0/ or send a letter to 31 # Creative Commons, PO Box 1866, Mountain View, CA 94042, USA. 44 32 #******************************************************************************* 45 33 … … 51 39 import fnmatch 52 40 53 # software specific module from flex_extract41 # software specific modules from flex_extract 54 42 sys.path.append('../') 55 #import profiling56 43 from mods.tools import silent_remove, get_list_as_string 57 44 … … 61 48 62 49 class UioFiles(object): 63 '''Class to manipulate files. At initialisation it has the pattern 64 which stores a regular expression pattern for the files, the path 65 to the files and the files already. 50 '''Collection of files matching a specific pattern. 51 52 The pattern can contain regular expressions for the files. 53 The files are listed and can be transformed to a single string or 54 they can be deleted. 55 56 Attributes 57 ---------- 58 path : str 59 Directory where to list the files. 60 61 pattern : str 62 Regular expression pattern. For example: '\*.grb' 63 64 files : list of str 65 List of files matching the pattern in the path. 66 66 ''' 67 67 # -------------------------------------------------------------------------- 68 # CLASS FUNCTIONS68 # CLASS METHODS 69 69 # -------------------------------------------------------------------------- 70 70 def __init__(self, path, pattern): … … 73 73 Parameters 74 74 ---------- 75 path : :obj:`string`75 path : str 76 76 Directory where to list the files. 77 77 78 pattern : :obj:`string`78 pattern : str 79 79 Regular expression pattern. For example: '\*.grb' 80 80 … … 92 92 return 93 93 94 #@profiling.timefn 94 95 95 def _list_files(self, path): 96 96 '''Lists all files in the directory with the matching … … 99 99 Parameters 100 100 ---------- 101 path : :obj:`string`101 path : str 102 102 Path to the files. 103 103 … … 116 116 return 117 117 118 118 119 def __str__(self): 119 120 '''Converts the list of files into a single string. … … 125 126 Return 126 127 ------ 127 files_string : :obj:`string`128 files_string : str 128 129 The content of the list as a single string. 129 130 ''' … … 133 134 134 135 return files_string 136 135 137 136 138 def delete_files(self): -
source/python/install.py
r1eca806 r6f951ca 14 14 # - splitted code in smaller functions 15 15 # - delete convert build files in here instead of compile job script 16 # - changed static path names to Variables from config file 16 # - changed static path names to variables from config file 17 # - splitted install function into several smaller pieces 18 # - use of tarfile package in python 17 19 # 18 20 # @License: 19 # (C) Copyright 2015-2018. 21 # (C) Copyright 2014-2019. 22 # Anne Philipp, Leopold Haimberger 20 23 # 21 # This software is licensed under the terms of the Apache Licence Version 2.0 22 # which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. 24 # This work is licensed under the Creative Commons Attribution 4.0 25 # International License. To view a copy of this license, visit 26 # http://creativecommons.org/licenses/by/4.0/ or send a letter to 27 # Creative Commons, PO Box 1866, Mountain View, CA 94042, USA. 23 28 # 24 # @Program Functionality: 25 # Depending on the selected installation environment (locally or on the 26 # ECMWF server ecgate or cca) the program extracts the commandline 27 # arguments and the CONTROL file parameter and prepares the corresponding 28 # environment. The necessary files are collected in a tar-ball and placed 29 # at the target location. There its untared, the environment variables will 30 # be set and the Fortran code will be compiled. If the ECMWF environment is 31 # selected a job script is prepared and submitted for the remaining 32 # configurations after putting the tar-ball to the target ECMWF server. 33 # 34 # @Program Content: 35 # - main 36 # - get_install_cmdline_args 37 # - install_via_gateway 38 # - mk_tarball 39 # - un_tarball 40 # - mk_env_vars 41 # - mk_compilejob 42 # - mk_job_template 43 # - del_convert_build 44 # - mk_convert_build 29 # @Methods: 30 # main 31 # get_install_cmdline_args 32 # install_via_gateway 33 # check_install_conditions 34 # mk_tarball 35 # un_tarball 36 # mk_env_vars 37 # mk_compilejob 38 # mk_job_template 39 # del_convert_build 40 # mk_convert_build 45 41 # 46 42 #******************************************************************************* 43 '''This script installs the flex_extract program. 44 45 Depending on the selected installation environment (locally or on the 46 ECMWF server ecgate or cca) the program extracts the commandline 47 arguments and the CONTROL file parameter and prepares the corresponding 48 environment. 49 The necessary files are collected in a tar-ball and placed 50 at the target location. There its untared, the environment variables will 51 be set and the Fortran code will be compiled. 52 If the ECMWF environment is selected a job script is prepared and submitted 53 for the remaining configurations after putting the tar-ball to the 54 target ECMWF server. 55 56 Type: install.py --help 57 to get information about command line parameters. 58 Read the documentation for usage instructions. 59 ''' 47 60 48 61 # ------------------------------------------------------------------------------ … … 96 109 Return 97 110 ------ 98 args : :obj:`Namespace`111 args : Namespace 99 112 Contains the commandline arguments from script/program call. 100 113 ''' … … 144 157 Parameters 145 158 ---------- 146 c : :obj:`ControlFile`159 c : ControlFile 147 160 Contains all the parameters of CONTROL file and 148 161 command line. … … 229 242 Parameters 230 243 ---------- 231 c : :obj:`ControlFile`244 c : ControlFile 232 245 Contains all the parameters of CONTROL file and 233 246 command line. … … 283 296 Parameters 284 297 ---------- 285 tarball_path : :obj:`string`298 tarball_path : str 286 299 The complete path to the tar file which will contain all 287 300 relevant data for flex_extract. 288 301 289 target : :obj:`string`302 target : str 290 303 The queue where the job is submitted to. 291 304 … … 374 387 Parameters 375 388 ---------- 376 tarball_path : :obj:`string`389 tarball_path : str 377 390 The complete path to the tar file which will contain all 378 391 relevant data for flex_extract. … … 407 420 Parameters 408 421 ---------- 409 ecuid : :obj:`string`422 ecuid : str 410 423 The user id on ECMWF server. 411 424 412 ecgid : :obj:`string`425 ecgid : str 413 426 The group id on ECMWF server. 414 427 415 gateway : :obj:`string`428 gateway : str 416 429 The gateway server the user is using. 417 430 418 destination : :obj:`string`431 destination : str 419 432 The remote destination which is used to transfer files 420 433 from ECMWF server to local gateway server. … … 469 482 Parameters 470 483 ---------- 471 makefile : :obj:`string`484 makefile : str 472 485 Name of the makefile which should be used to compile FORTRAN 473 486 CONVERT2 program. 474 487 475 target : :obj:`string`488 target : str 476 489 The target where the installation should be done, e.g. the queue. 477 490 478 ecuid : :obj:`string`491 ecuid : str 479 492 The user id on ECMWF server. 480 493 481 ecgid : :obj:`string`494 ecgid : str 482 495 The group id on ECMWF server. 483 496 484 fp_root : :obj:`string`497 fp_root : str 485 498 Path to the root directory of FLEXPART environment or flex_extract 486 499 environment. … … 544 557 Parameters 545 558 ---------- 546 ecuid : :obj:`string`559 ecuid : str 547 560 The user id on ECMWF server. 548 561 549 ecgid : :obj:`string`562 ecgid : str 550 563 The group id on ECMWF server. 551 564 552 gateway : :obj:`string`565 gateway : str 553 566 The gateway server the user is using. 554 567 555 destination : :obj:`string`568 destination : str 556 569 The remote destination which is used to transfer files 557 570 from ECMWF server to local gateway server. 558 571 559 fp_root : :obj:`string`572 fp_root : str 560 573 Path to the root directory of FLEXPART environment or flex_extract 561 574 environment. … … 622 635 Parameters 623 636 ---------- 624 src_path : :obj:`string`637 src_path : str 625 638 Path to the fortran source directory. 626 639 … … 645 658 Parameters 646 659 ---------- 647 src_path : :obj:`string`660 src_path : str 648 661 Path to the fortran source directory. 649 662 650 makefile : :obj:`string`663 makefile : str 651 664 The name of the makefile which should be used. 652 665 -
source/python/mods/checks.py
rf2616a3 r6f951ca 1 1 #!/usr/bin/env python 2 2 # -*- coding: utf-8 -*- 3 # #*******************************************************************************3 #******************************************************************************* 4 4 # @Author: Anne Philipp (University of Vienna) 5 5 # … … 9 9 # 10 10 # @License: 11 # (C) Copyright 2014-2018. 11 # (C) Copyright 2014-2019. 12 # Anne Philipp, Leopold Haimberger 12 13 # 13 # This software is licensed under the terms of the Apache Licence Version 2.0 14 # which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. 15 # 16 # @Modul Description: 17 # 18 # 19 # @Module Content: 20 21 # 14 # This work is licensed under the Creative Commons Attribution 4.0 15 # International License. To view a copy of this license, visit 16 # http://creativecommons.org/licenses/by/4.0/ or send a letter to 17 # Creative Commons, PO Box 1866, Mountain View, CA 94042, USA. 22 18 #******************************************************************************* 19 '''This module contains check methods for the CONTROL paramaters. 20 ''' 23 21 24 22 # ------------------------------------------------------------------------------ … … 27 25 28 26 import os 27 import sys 29 28 import _config 30 29 import exceptions … … 40 39 Parameters 41 40 ---------- 42 c : :obj:`ControlFile`41 c : ControlFile 43 42 Contains all the parameters of CONTROL file and 44 43 command line. 45 44 46 logicals : :obj:`list` of (:obj:`string` or :obj:`integer`)45 logicals : list of (str or int) 47 46 Names of the switches that are used to control the flow of the 48 47 program. … … 67 66 Parameters 68 67 ---------- 69 grid : :obj:`string`68 grid : str 70 69 Contains grid information 71 70 72 71 Return 73 72 ------ 74 grid : :obj:``string`73 grid : str 75 74 Contains grid in format Lat/lon. E.g. 0.1/0.1 76 75 ''' … … 108 107 Parameters 109 108 ---------- 110 grid : :obj:`string`109 grid : str 111 110 Contains grid information. 112 111 113 area : :obj:`string`112 area : str 114 113 Contains area informtion. 115 114 116 upper : :obj:`string`115 upper : str 117 116 The northern most latitude. 118 117 119 lower : :obj:`string`118 lower : str 120 119 The souther most latitude. 121 120 122 left : :obj:`string`121 left : str 123 122 The western most longitude. 124 123 125 right : :obj:`string`124 right : str 126 125 The eastern most longiude. 127 126 128 127 Return 129 128 ------ 130 grid : :obj:``string`129 grid : str 131 130 Contains grid in format Lat/lon. E.g. 0.1/0.1 132 131 ''' … … 173 172 Parameters 174 173 ---------- 175 levelist : :obj:`string`174 levelist : str 176 175 Specifies the level list. 177 176 Examples: model level: 1/to/137, pressure levels: 500/to/1000 178 177 179 level : :obj:`string`178 level : str 180 179 Specifies the maximum level. 181 180 182 181 Return 183 182 ------ 184 levelist : :obj:`string`183 levelist : str 185 184 Specifies the required levels. It has to have a valid 186 185 correspondence to the selected levtype. 187 186 Examples: model level: 1/to/137, pressure levels: 500/to/1000 188 187 189 level : :obj:`string`188 level : str 190 189 Specifies the maximum level. It has to be one of the 191 190 available maximum level number as contained in the variable … … 221 220 Parameters 222 221 ---------- 223 c : :obj:`ControlFile`222 c : ControlFile 224 223 Contains all the parameters of CONTROL file and 225 224 command line. 226 225 227 ppid : :obj:`int` or :obj:`None`226 ppid : int or None 228 227 Contains the ppid number provided by the command line parameter 229 228 of is None otherwise. … … 247 246 Parameters 248 247 ---------- 249 ftype : :obj:`list` of :obj:`string`248 ftype : list of str 250 249 List of field types. 251 250 … … 273 272 Parameters 274 273 ---------- 275 step : :obj:`list` of :obj:`string` or :obj:`string`274 step : list of str or str 276 275 Specifies the forecast time step from forecast base time. 277 276 Valid values are hours (HH) from forecast base time. 278 277 279 mailfail : :obj:`list` of :obj:``string`278 mailfail : list of str 280 279 Contains all email addresses which should be notified. 281 280 It might also contain just the ecmwf user name which will trigger … … 284 283 Return 285 284 ------ 286 step : :obj:`list` of :obj:`string`285 step : list of str 287 286 List of forecast steps in format e.g. [001, 002, ...] 288 287 ''' … … 313 312 Parameters 314 313 ---------- 315 ftype : :obj:`list` of :obj:`string` or :obj:`string`314 ftype : list of str or str 316 315 List of field types. 317 316 318 steps : :obj:`string`317 steps : str 319 318 Specifies the forecast time step from forecast base time. 320 319 Valid values are hours (HH) from forecast base time. … … 322 321 Return 323 322 ------ 324 ftype : :obj:`list` of :obj:`string`323 ftype : list of str 325 324 List of field types. 326 325 ''' … … 340 339 Parameters 341 340 ---------- 342 ftime : :obj:`list` of :obj:`string` or :obj:`string`341 ftime : list of str or str 343 342 The time in hours of the field. 344 343 345 344 Return 346 345 ------ 347 ftime : :obj:`list` of :obj:`string`346 ftime : list of str 348 347 The time in hours of the field. 349 348 ''' … … 358 357 Parameters 359 358 ---------- 360 ftype : :obj:`list` of :obj:`string`359 ftype : list of str 361 360 List of field types. 362 361 363 ftime : :obj:`list` of :obj:`string` or :obj:`string`362 ftime : list of str or str 364 363 The time in hours of the field. 365 364 366 steps : :obj:`string`365 steps : str 367 366 Specifies the forecast time step from forecast base time. 368 367 Valid values are hours (HH) from forecast base time. 369 368 370 maxstep : :obj:`integer`369 maxstep : int 371 370 The maximum forecast time step in hours from the forecast base time. 372 371 This is the maximum step for non flux (accumulated) forecast data. 373 372 374 purefc : :obj:`integer`373 purefc : int 375 374 Switch for definition of pure forecast mode or not. 376 375 377 376 Return 378 377 ------ 379 ftype : :obj:`list` of :obj:`string`378 ftype : list of str 380 379 List of field types. 381 380 382 ftime : :obj:`list` of :obj:`string`381 ftime : list of str 383 382 The time in hours of the field. 384 383 385 steps : :obj:`string`384 steps : str 386 385 Specifies the forecast time step from forecast base time. 387 386 Valid values are hours (HH) from forecast base time. … … 410 409 Parameters 411 410 ---------- 412 mail : :obj:`list` of :obj:`string` or :obj:`string`411 mail : list of str or str 413 412 Contains email addresses for notifications. 414 413 It might also contain just the ecmwf user name which will trigger … … 417 416 Return 418 417 ------ 419 mail : :obj:`list` of :obj:``string`418 mail : list of str 420 419 Contains email addresses for notifications. 421 420 It might also contain just the ecmwf user name which will trigger … … 439 438 Parameters 440 439 ---------- 441 queue : :obj:`string`440 queue : str 442 441 Name of the queue if submitted to the ECMWF servers. 443 442 Used to check if ecuid, ecgid, gateway and destination 444 443 are set correctly and are not empty. 445 444 446 gateway : :obj:`string`445 gateway : str 447 446 The address of the gateway server. 448 447 449 destination : :obj:`string`448 destination : str 450 449 The name of the destination of the gateway server for data 451 450 transfer through ectrans. E.g. name@genericSftp 452 451 453 ecuid : :obj:`string`452 ecuid : str 454 453 ECMWF user id. 455 454 456 ecgid : :obj:`string`455 ecgid : str 457 456 ECMWF group id. 458 457 … … 475 474 Parameters 476 475 ---------- 477 idir : :obj:`string`476 idir : str 478 477 Path to the temporary directory for MARS retrieval data. 479 478 480 odir : :obj:`string`479 odir : str 481 480 Path to the final output directory where the FLEXPART input files 482 481 will be stored. 483 482 484 fpdir : :obj:`string`483 fpdir : str 485 484 Path to FLEXPART root directory. 486 485 487 fedir : :obj:`string`486 fedir : str 488 487 Path to flex_extract root directory. 489 488 490 489 Return 491 490 ------ 492 odir : :obj:`string`491 odir : str 493 492 Path to the final output directory where the FLEXPART input files 494 493 will be stored. 495 494 496 fpdir : :obj:`string`495 fpdir : str 497 496 Path to FLEXPART root directory. 498 497 … … 513 512 Parameters 514 513 ---------- 515 start : :obj:`string`514 start : str 516 515 The start date of the retrieval job. 517 516 518 end : :obj:`string`517 end : str 519 518 The end date of the retrieval job. 520 519 521 520 Return 522 521 ------ 523 start : :obj:`string`522 start : str 524 523 The start date of the retrieval job. 525 524 526 end : :obj:`string`525 end : str 527 526 The end date of the retrieval job. 528 527 … … 555 554 Parameters 556 555 ---------- 557 maxstep : :obj:`string`556 maxstep : str 558 557 The maximum forecast time step in hours from the forecast base time. 559 558 This is the maximum step for non flux (accumulated) forecast data. 560 559 561 steps : :obj:`string`560 steps : str 562 561 Specifies the forecast time step from forecast base time. 563 562 Valid values are hours (HH) from forecast base time. … … 565 564 Return 566 565 ------ 567 maxstep : :obj:`integer`566 maxstep : int 568 567 The maximum forecast time step in hours from the forecast base time. 569 568 This is the maximum step for non flux (accumulated) forecast data. … … 588 587 Parameters 589 588 ---------- 590 basetime : :obj:``589 basetime : str 591 590 The time for a half day retrieval. The 12 hours upfront are to be 592 591 retrieved. … … 607 606 Parameters 608 607 ---------- 609 request : :obj:`integer`608 request : int 610 609 Selects the mode of retrieval. 611 610 0: Retrieves the data from ECMWF. … … 613 612 2: Retrieves the data and prints the mars request. 614 613 615 marsfile : :obj:`string`614 marsfile : str 616 615 Path to the mars request file. 617 616 … … 631 630 Parameters 632 631 ---------- 633 public : :obj:`ìnteger`632 public : int 634 633 Specifies if public data are to be retrieved or not. 635 634 636 dataset : :obj:`string`635 dataset : str 637 636 Specific name which identifies the public dataset. 638 637 … … 653 652 Parameters 654 653 ---------- 655 acctype : :obj:`string`654 acctype : str 656 655 The field type for the accumulated forecast fields. 657 656 658 ftype : :obj:`list` of :obj:`string`657 ftype : list of str 659 658 List of field types. 660 659 661 660 Return 662 661 ------ 663 acctype : :obj:`string`662 acctype : str 664 663 The field type for the accumulated forecast fields. 665 664 ''' … … 690 689 Parameters 691 690 ---------- 692 acctime : :obj:`string`691 acctime : str 693 692 The starting time from the accumulated forecasts. 694 693 695 acctype : :obj:`string`694 acctype : str 696 695 The field type for the accumulated forecast fields. 697 696 698 purefc : :obj:`integer`697 purefc : int 699 698 Switch for definition of pure forecast mode or not. 700 699 701 700 Return 702 701 ------ 703 acctime : :obj:`string`702 acctime : str 704 703 The starting time from the accumulated forecasts. 705 704 ''' 706 705 if not acctime: 707 706 print('... Control parameter ACCTIME was not defined.') 708 print('... Value will be set depending on field type: '707 print('... Value will be set depending on field type:\n ' 709 708 '\t\t EA=06/18\n\t\t EI/OD=00/12\n\t\t EP=18') 710 709 if acctype.upper() == 'EA': # Era 5 … … 726 725 Parameters 727 726 ---------- 728 accmaxstep : :obj:`string`727 accmaxstep : str 729 728 The maximum forecast step for the accumulated forecast fields. 730 729 731 acctype : :obj:`string`730 acctype : str 732 731 The field type for the accumulated forecast fields. 733 732 734 purefc : :obj:`integer`733 purefc : int 735 734 Switch for definition of pure forecast mode or not. 736 735 737 maxstep : :obj:`string`736 maxstep : str 738 737 The maximum forecast time step in hours from the forecast base time. 739 738 This is the maximum step for non flux (accumulated) forecast data. … … 741 740 Return 742 741 ------ 743 accmaxstep : :obj:`string`742 accmaxstep : str 744 743 The maximum forecast step for the accumulated forecast fields. 745 744 ''' … … 776 775 Parameters 777 776 ---------- 778 addpar : :obj:`string` or :obj:'list' of :obj:'string'777 addpar : str or list of str 779 778 List of additional parameters to be retrieved. 780 779 781 780 Return 782 781 ------ 783 addpar : :obj:'string'782 addpar : str 784 783 List of additional parameters to be retrieved. 785 784 ''' … … 798 797 799 798 def check_job_chunk(job_chunk): 800 '''Checks that the job chunknumber is positive and non zero.801 802 Parameters 803 ---------- 804 job_chunk : :obj:`integer`799 '''Checks that if job chunk is set, the number is positive and non zero. 800 801 Parameters 802 ---------- 803 job_chunk : int 805 804 The number of days for a single job script. 806 805 807 806 Return 808 807 ------ 809 job_chunk : :obj:`integer`808 job_chunk : int 810 809 The number of days for a single job script. 811 810 ''' 811 if not job_chunk: 812 return job_chunk 813 812 814 if job_chunk < 0: 813 815 raise ValueError('ERROR: The number of job chunk is negative!\n' -
source/python/mods/disaggregation.py
r0e2f93e r6f951ca 17 17 # - outsourced the disaggregation functions dapoly and darain 18 18 # to a new module named disaggregation 19 # - added the new disaggregation method for precipitation 19 20 # 20 21 # @License: 21 # (C) Copyright 2015-2018. 22 # 23 # This software is licensed under the terms of the Apache Licence Version 2.0 24 # which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. 25 # 26 # @Module Description: 27 # disaggregation of deaccumulated flux data from an ECMWF model FG field. 28 # Initially the flux data to be concerned are: 29 # - large-scale precipitation 30 # - convective precipitation 31 # - surface sensible heat flux 32 # - surface solar radiation 33 # - u stress 34 # - v stress 35 # Different versions of disaggregation is provided for rainfall 36 # data (darain, modified linear) and the surface fluxes and 37 # stress data (dapoly, cubic polynomial). 38 # 39 # @Module Content: 22 # (C) Copyright 2014-2019. 23 # Anne Philipp, Leopold Haimberger 24 # 25 # This work is licensed under the Creative Commons Attribution 4.0 26 # International License. To view a copy of this license, visit 27 # http://creativecommons.org/licenses/by/4.0/ or send a letter to 28 # Creative Commons, PO Box 1866, Mountain View, CA 94042, USA. 29 # 30 # @Methods: 40 31 # - dapoly 41 32 # - darain 42 33 # - IA3 43 #44 34 #******************************************************************************* 35 '''Disaggregation of deaccumulated flux data from an ECMWF model FG field. 36 37 Initially the flux data to be concerned are: 38 - large-scale precipitation 39 - convective precipitation 40 - surface sensible heat flux 41 - surface solar radiation 42 - u stress 43 - v stress 44 45 Different versions of disaggregation is provided for rainfall 46 data (darain, modified linear) and the surface fluxes and 47 stress data (dapoly, cubic polynomial). 48 ''' 45 49 46 50 # ------------------------------------------------------------------------------ … … 65 69 Parameters 66 70 ---------- 67 alist : :obj:`list` of :obj:`array` of :obj:`float`71 alist : list of array of float 68 72 List of 4 timespans as 2-dimensional, horizontal fields. 69 73 E.g. [[array_t1], [array_t2], [array_t3], [array_t4]] … … 71 75 Return 72 76 ------ 73 nfield : :obj:`array` of :obj:`float`77 nfield : array of float 74 78 Interpolated flux at central point of accumulation timespan. 75 79 … … 109 113 Parameters 110 114 ---------- 111 alist : :obj:`list` of :obj:`array` of :obj:`float`115 alist : list of array of float 112 116 List of 4 timespans as 2-dimensional, horizontal fields. 113 117 E.g. [[array_t1], [array_t2], [array_t3], [array_t4]] … … 115 119 Return 116 120 ------ 117 nfield : :obj:`array` of :obj:`float`121 nfield : array of float 118 122 Interpolated flux at central point of accumulation timespan. 119 123 … … 160 164 Note 161 165 ---- 162 Copyright 2017166 (C) Copyright 2017-2019 163 167 Sabine Hittmeir, Anne Philipp, Petra Seibert 164 168 … … 170 174 Parameters 171 175 ---------- 172 g : :obj:`list` of :obj:`float`176 g : list of float 173 177 Complete data series that will be interpolated having 174 178 the dimension of the original raw series. … … 176 180 Return 177 181 ------ 178 f : :obj:`list` of :obj:`float`182 f : list of float 179 183 The interpolated data series with additional subgrid points. 180 184 Its dimension is equal to the length of the input data series -
source/python/mods/get_mars_data.py
r9aefaad r6f951ca 22 22 # online documentation) 23 23 # - use of UIFiles class for file selection and deletion 24 # 24 # - seperated get_mars_data function into several smaller pieces: 25 # write_reqheader, mk_server, mk_dates, remove_old, do_retrievment 25 26 # 26 27 # @License: 27 # (C) Copyright 2014-2018. 28 # 29 # This software is licensed under the terms of the Apache Licence Version 2.0 30 # which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. 31 # 32 # @Program Functionality: 33 # This program can be used as a module in the whole flex_extract process 34 # or can be run by itself to just extract MARS data from ECMWF. To do so, 35 # a couple of necessary parameters has to be passed with the program call. 36 # See documentation for more details. 37 # 38 # @Program Content: 39 # - main 40 # - get_mars_data 41 # - do_retrievement 42 # 28 # (C) Copyright 2014-2019. 29 # Anne Philipp, Leopold Haimberger 30 # 31 # This work is licensed under the Creative Commons Attribution 4.0 32 # International License. To view a copy of this license, visit 33 # http://creativecommons.org/licenses/by/4.0/ or send a letter to 34 # Creative Commons, PO Box 1866, Mountain View, CA 94042, USA. 43 35 #******************************************************************************* 44 """ToDo: Name of litte program 45 46 ToDo: Add desccription 47 48 ToDo: Add Conditions 49 50 This script requires that `...` be installed within the Python 51 environment you are running this script in. 36 '''This script extracts MARS data from ECMWF servers. 37 38 At first, the necessary parameters from command line and CONTROL files are 39 extracted. They define the data set to be extracted from MARS. 52 40 53 41 This file can also be imported as a module and contains the following 54 42 functions: 55 43 56 * get_mars_data -57 * do_retrievement -58 44 * main - the main function of the script 59 """ 45 * get_mars_data - overall control of ECMWF data retrievment 46 * write_reqheader - writes the header into the mars_request file 47 * mk_server - creates the server connection to ECMWF servers 48 * mk_dates - defines the start and end date 49 * remove_old - deletes old retrieved grib files 50 * do_retrievement - creates individual retrievals 51 52 Type: get_mars_data.py --help 53 to get information about command line parameters. 54 Read the documentation for usage instructions. 55 ''' 60 56 # ------------------------------------------------------------------------------ 61 57 # MODULES … … 67 63 68 64 # software specific classes and modules from flex_extract 65 # add path to local main python path for flex_extract to get full access 69 66 sys.path.append(os.path.dirname(os.path.abspath( 70 67 inspect.getfile(inspect.currentframe()))) + '/../') … … 120 117 Parameters 121 118 ---------- 122 c : :obj:`ControlFile`119 c : ControlFile 123 120 Contains all the parameters of CONTROL file and 124 121 command line. … … 166 163 Parameters 167 164 ---------- 168 marsfile : :obj:`string`165 marsfile : str 169 166 Path to the mars request file. 170 167 … … 189 186 Parameters 190 187 ---------- 191 c : :obj:`ControlFile`188 c : ControlFile 192 189 Contains all the parameters of CONTROL file and 193 190 command line. … … 195 192 Return 196 193 ------ 197 server : :obj:`ECMWFDataServer` or :obj:`ECMWFService`194 server : ECMWFDataServer or ECMWFService 198 195 Connection to ECMWF server via python interface ECMWF WebAPI. 199 196 … … 230 227 Parameters 231 228 ---------- 232 c : :obj:`ControlFile`229 c : ControlFile 233 230 Contains all the parameters of CONTROL file and 234 231 command line. 235 232 236 fluxes : :obj:`boolean`, optional233 fluxes : boolean, optional 237 234 Decides if the flux parameter settings are stored or 238 235 the rest of the parameter list. … … 241 238 Return 242 239 ------ 243 start : :obj:`datetime`240 start : datetime 244 241 The start date of the retrieving data set. 245 242 246 end : :obj:`datetime`243 end : datetime 247 244 The end date of the retrieving data set. 248 245 249 chunk : :obj:`datetime`246 chunk : datetime 250 247 Time period in days for one single mars retrieval. 251 248 … … 271 268 Parameters 272 269 ---------- 273 pattern : :obj:`string`270 pattern : str 274 271 The sub string pattern which identifies the files to be deleted. 275 272 276 inputdir : :obj:`string`, optional273 inputdir : str, optional 277 274 Path to the directory where the retrieved data is stored. 278 275 … … 295 292 Parameters 296 293 ---------- 297 c : :obj:`ControlFile`294 c : ControlFile 298 295 Contains all the parameters of CONTROL file and 299 296 command line. 300 297 301 server : :obj:`ECMWFService`298 server : ECMWFService or ECMWFDataServer 302 299 The server connection to ECMWF. 303 300 304 start : :obj:`datetime`301 start : datetime 305 302 The start date of the retrieval. 306 303 307 end : :obj:`datetime`304 end : datetime 308 305 The end date of the retrieval. 309 306 310 delta_t : :obj:`datetime`307 delta_t : datetime 311 308 Delta_t + 1 is the maximal time period of a single 312 309 retrieval. 313 310 314 fluxes : :obj:`boolean`, optional311 fluxes : boolean, optional 315 312 Decides if the flux parameters are to be retrieved or 316 313 the rest of the parameter list. -
source/python/mods/prepare_flexpart.py
r9aefaad r6f951ca 22 22 # - added documentation 23 23 # - minor changes in programming style for consistence 24 # - BUG : removed call of clean_up-Function after call of24 # - BUGFIX: removed call of clean_up-Function after call of 25 25 # prepareFlexpart in main since it is already called in 26 26 # prepareFlexpart at the end! … … 29 29 # 30 30 # @License: 31 # (C) Copyright 2014-2018. 31 # (C) Copyright 2014-2019. 32 # Anne Philipp, Leopold Haimberger 32 33 # 33 # This software is licensed under the terms of the Apache Licence Version 2.0 34 # which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. 35 # 36 # @Program Functionality: 37 # This program prepares the final version of the grib files which are 38 # then used by FLEXPART. It converts the bunch of grib files extracted 39 # via get_mars_data by doing for example the necessary conversion to get 40 # consistent grids or the disaggregation of flux data. Finally, the 41 # program combines the data fields in files per available hour with the 42 # naming convention xxYYMMDDHH, where xx should be 2 arbitrary letters 43 # (mostly xx is chosen to be "EN"). 44 # 45 # @Program Content: 46 # - main 47 # - prepare_flexpart 48 # 34 # This work is licensed under the Creative Commons Attribution 4.0 35 # International License. To view a copy of this license, visit 36 # http://creativecommons.org/licenses/by/4.0/ or send a letter to 37 # Creative Commons, PO Box 1866, Mountain View, CA 94042, USA. 49 38 #******************************************************************************* 39 '''This script prepares the final version of the grib files which are 40 then used by FLEXPART. 41 42 It converts the bunch of grib files extracted via get_mars_data before, 43 by doing the necessary conversion to get consistent grids or the 44 disaggregation of flux data. Finally, the data fields are combined 45 in files per available hour with the naming convention xxYYMMDDHH, 46 where xx should be 2 arbitrary letters (mostly xx is chosen to be "EN"). 47 48 This file can also be imported as a module which then contains the following 49 functions: 50 51 * main 52 * prepare_flexpart 53 54 Type: prepare_flexpart.py --help 55 to get information about command line parameters. 56 Read the documentation for usage instructions. 57 ''' 50 58 51 59 # ------------------------------------------------------------------------------ … … 59 67 60 68 # software specific classes and modules from flex_extract 61 69 # add path to local main python path for flex_extract to get full access 62 70 sys.path.append(os.path.dirname(os.path.abspath( 63 71 inspect.getfile(inspect.currentframe()))) + '/../') … … 115 123 Parameters 116 124 ---------- 117 ppid : :obj:`int`125 ppid : int 118 126 Contains the ppid number of the current ECMWF job. It will be None if 119 127 the method was called within this module. 120 128 121 c : :obj:`ControlFile`129 c : ControlFile 122 130 Contains all the parameters of CONTROL file and 123 131 command line. -
source/python/mods/tools.py
rf2616a3 r6f951ca 14 14 # my_error, normal_exit, init128, to_param_id 15 15 # 16 # April 2018 - Anne Philipp (University of Vienna):16 # April - December 2018 - Anne Philipp (University of Vienna): 17 17 # - applied PEP8 style guide 18 18 # - added documentation 19 # - moved all functions from file Flexparttools to this file tools 20 # - added function get_list_as_string 19 # - moved all non class methods from former file Flexparttools in here 21 20 # - seperated args and control interpretation 21 # - added functions get_list_as_string, read_ecenv, send_mail, make_dir, 22 # put_file_to_ecserver, submit_job_to_ecserver, get_informations, 23 # get_dimensions, execute_subprocess, none_or_int, none_or_str 22 24 # 23 25 # @License: 24 # (C) Copyright 2014-2018. 26 # (C) Copyright 2014-2019. 27 # Anne Philipp, Leopold Haimberger 25 28 # 26 # This software is licensed under the terms of the Apache Licence Version 2.0 27 # which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. 29 # This work is licensed under the Creative Commons Attribution 4.0 30 # International License. To view a copy of this license, visit 31 # http://creativecommons.org/licenses/by/4.0/ or send a letter to 32 # Creative Commons, PO Box 1866, Mountain View, CA 94042, USA. 28 33 # 29 # @Modul Description: 30 # This module contains a couple of helpful functions which are 31 # used in different places in flex_extract. 32 # 33 # @Module Content: 34 # - get_cmdline_args 35 # - clean_up 36 # - my_error 37 # - normal_exit 38 # - product 39 # - silent_remove 40 # - init128 41 # - to_param_id 42 # - get_list_as_string 43 # - make_dir 44 # 34 # @Methods: 35 # none_or_str 36 # none_or_int 37 # get_cmdline_args 38 # read_ecenv 39 # clean_up 40 # my_error 41 # send_mail 42 # normal_exit 43 # product 44 # silent_remove 45 # init128 46 # to_param_id 47 # get_list_as_string 48 # make_dir 49 # put_file_to_ecserver 50 # submit_job_to_ecserver 51 # get_informations 52 # get_dimensions 53 # execute_subprocess 45 54 #******************************************************************************* 55 '''This module contains a collection of diverse tasks within flex_extract. 56 ''' 46 57 47 58 # ------------------------------------------------------------------------------ … … 59 70 60 71 # ------------------------------------------------------------------------------ 61 # FUNCTIONS72 # METHODS 62 73 # ------------------------------------------------------------------------------ 63 74 … … 68 79 Parameters 69 80 ---------- 70 value : :obj:`string`81 value : str 71 82 String to be checked for the "None" word. 72 83 … … 87 98 Parameters 88 99 ---------- 89 value : :obj:`string`100 value : str 90 101 String to be checked for the "None" word. 91 102 … … 110 121 Return 111 122 ------ 112 args : :obj:`Namespace`123 args : Namespace 113 124 Contains the commandline arguments from script/program call. 114 125 ''' … … 201 212 Parameters 202 213 ---------- 203 filepath : :obj:`string`214 filepath : str 204 215 Path to file where the ECMWF environment parameters are stored. 205 216 206 217 Return 207 218 ------ 208 envs : :obj:`dictionary`219 envs : dict 209 220 Contains the environment parameter ecuid, ecgid, gateway 210 221 and destination for ECMWF server environments. … … 233 244 Parameters 234 245 ---------- 235 c : :obj:`ControlFile`246 c : ControlFile 236 247 Contains all the parameters of CONTROL file and 237 248 command line. … … 265 276 Parameters 266 277 ---------- 267 user : :obj:`list` of :obj:`string`278 user : list of str 268 279 Contains all email addresses which should be notified. 269 280 It might also contain just the ecmwf user name which wil trigger 270 281 mailing to the associated email address for this user. 271 282 272 message : :obj:`string`, optional283 message : str, optional 273 284 Error message. Default value is "ERROR". 274 285 … … 295 306 Parameters 296 307 ---------- 297 users : :obj:`list` of :obj:`string`308 users : list of str 298 309 Contains all email addresses which should be notified. 299 310 It might also contain just the ecmwf user name which wil trigger 300 311 mailing to the associated email address for this user. 301 312 302 success_mode : :obj:``string`313 success_mode : str 303 314 States the exit mode of the program to put into 304 315 the mail subject line. 305 316 306 message : :obj:`string`, optional317 message : str, optional 307 318 Message for exiting program. Default value is "Done!". 308 319 … … 341 352 Parameters 342 353 ---------- 343 message : :obj:`string`, optional354 message : str, optional 344 355 Message for exiting program. Default value is "Done!". 345 356 … … 374 385 Parameters 375 386 ---------- 376 \*args : :obj:`list` or :obj:`string`387 \*args : list or str 377 388 Positional arguments (arbitrary number). 378 389 379 \*\*kwds : :obj:`dictionary`390 \*\*kwds : dict 380 391 Contains all the keyword arguments from \*args. 381 392 … … 405 416 Parameters 406 417 ---------- 407 filename : :obj:`string`418 filename : str 408 419 The name of the file to be removed without notification. 409 420 … … 429 440 Parameters 430 441 ---------- 431 filepath : :obj:`string`442 filepath : str 432 443 Path to file of ECMWF grib table number 128. 433 444 434 445 Return 435 446 ------ 436 table128 : :obj:`dictionary`447 table128 : dict 437 448 Contains the ECMWF grib table 128 information. 438 449 The key is the parameter number and the value is the … … 462 473 Parameters 463 474 ---------- 464 pars : :obj:`string`475 pars : str 465 476 Addpar argument from CONTROL file in the format of 466 477 parameter names instead of ids. The parameter short … … 468 479 one single string. 469 480 470 table : :obj:`dictionary`481 table : dict 471 482 Contains the ECMWF grib table 128 information. 472 483 The key is the parameter number and the value is the … … 475 486 Return 476 487 ------ 477 ipar : :obj:`list` of :obj:`integer`488 ipar : list of int 478 489 List of addpar parameters from CONTROL file transformed to 479 490 parameter ids in the format of integer. … … 501 512 Parameters 502 513 ---------- 503 list_obj : :obj:`list`514 list_obj : list of * 504 515 A list with arbitrary content. 505 516 506 concatenate_sign : :obj:`string`, optional517 concatenate_sign : str, optional 507 518 A string which is used to concatenate the single 508 519 list elements. Default value is ", ". … … 510 521 Return 511 522 ------ 512 str_of_list : :obj:`string`523 str_of_list : str 513 524 The content of the list as a single string. 514 525 ''' … … 528 539 Parameters 529 540 ---------- 530 directory : :obj:`string`541 directory : str 531 542 The path to directory which should be created. 532 543 … … 557 568 Parameters 558 569 ---------- 559 ecd : :obj:`string`570 ecd : str 560 571 The path were the file is stored. 561 572 562 filename : :obj:`string`573 filename : str 563 574 The name of the file to send to the ECMWF server. 564 575 565 target : :obj:`string`576 target : str 566 577 The target queue where the file should be sent to. 567 578 568 ecuid : :obj:`string`579 ecuid : str 569 580 The user id on ECMWF server. 570 581 571 ecgid : :obj:`string`582 ecgid : str 572 583 The group id on ECMWF server. 573 584 … … 610 621 Parameters 611 622 ---------- 612 target : :obj:`string`623 target : str 613 624 The target where the file should be sent to, e.g. the queue. 614 625 615 jobname : :obj:`string`626 jobname : str 616 627 The name of the jobfile to be submitted to the ECMWF server. 617 628 618 629 Return 619 630 ------ 620 job_id : :obj:`int`631 job_id : int 621 632 The id number of the job as a reference at the ecmwf server. 622 633 ''' … … 650 661 Parameters 651 662 ---------- 652 filename : :obj:`string`663 filename : str 653 664 Name of the file which will be opened to extract basic information. 654 665 655 666 Return 656 667 ------ 657 data : :obj:`dictionary`668 data : dict 658 669 Contains basic informations of the ECMWF grib files, e.g. 659 670 'Ni', 'Nj', 'latitudeOfFirstGridPointInDegrees', … … 702 713 Parameters 703 714 ---------- 704 info : :obj:`dictionary`715 info : dict 705 716 Contains basic informations of the ECMWF grib files, e.g. 706 717 'Ni', 'Nj', 'latitudeOfFirstGridPointInDegrees', … … 709 720 'iDirectionIncrementInDegrees', 'missingValue' 710 721 711 purefc : :obj:`integer`722 purefc : int 712 723 Switch for definition of pure forecast mode or not. 713 724 714 dtime : :obj:`string`725 dtime : str 715 726 Time step in hours. 716 727 717 index_vals : :obj:`list`728 index_vals : list of list of str 718 729 Contains the values from the keys used for a distinct selection 719 730 of grib messages in processing the grib files. … … 723 734 index_vals[2]: ('0', '12', '3', '6', '9') ; stepRange 724 735 725 start_date : :obj:`string`736 start_date : str 726 737 The start date of the retrieval job. 727 738 728 end_date : :obj:`string`739 end_date : str 729 740 The end date of the retrieval job. 730 741 731 742 Return 732 743 ------ 733 (ix, jy, it) : :obj:`tuple` of :obj:`integer`744 (ix, jy, it) : tuple of int 734 745 Dimension in x-direction, y-direction and in time. 735 746 ''' … … 755 766 Parameters 756 767 ---------- 757 cmd_list : :obj:`list` of `:obj:`string`768 cmd_list : list of str 758 769 A list of the components for the command line execution. Each 759 770 list entry is a single part of the command which is seperated from … … 763 774 Return 764 775 ------ 765 error_msg : :obj:`string`, optional776 error_msg : str, optional 766 777 The possible error message if the subprocess failed. 767 778 By default it will just tell "SUBPROCESS FAILED!". -
source/python/submit.py
r0629ba8 r6f951ca 18 18 # - changed path names to variables from config file 19 19 # - added option for writing mars requests to extra file 20 # additionally,as option without submitting the mars jobs 20 # additionally, as option without submitting the mars jobs 21 # - splitted submit function to use genshi templates for the 22 # job script and avoid code duplication 21 23 # 22 24 # @License: 23 # (C) Copyright 2014-2018. 24 # 25 # This software is licensed under the terms of the Apache Licence Version 2.0 26 # which can be obtained at http://www.apache.org/licenses/LICENSE-2.0. 27 # 28 # @Program Functionality: 29 # This program is the main program of flex_extract and controls the 30 # program flow. 31 # If it is supposed to work locally then it works through the necessary 32 # functions get_mars_data and prepareFlexpart. Otherwise it prepares 33 # a shell job script which will do the necessary work on the 34 # ECMWF server and is submitted via ecaccess-job-submit. 35 # 36 # @Program Content: 37 # - main 38 # - submit 39 # 25 # (C) Copyright 2014-2019. 26 # Anne Philipp, Leopold Haimberger 27 # 28 # This work is licensed under the Creative Commons Attribution 4.0 29 # International License. To view a copy of this license, visit 30 # http://creativecommons.org/licenses/by/4.0/ or send a letter to 31 # Creative Commons, PO Box 1866, Mountain View, CA 94042, USA. 40 32 #******************************************************************************* 33 '''This script allows the user to extract meteorological fields from the ECMWF. 34 35 It prepares the settings for retrieving the data from ECMWF servers and 36 process the resulting files to prepare them for the use with FLEXPART or 37 FLEXTRA. 38 39 If it is supposed to work locally then it works through the necessary 40 functions get_mars_data and prepare_flexpart. Otherwise it prepares 41 a job script (korn shell) which will do the necessary work on the 42 ECMWF server. This script will de submitted via the ecaccess command 43 ecaccess-job-submit. 44 45 This file can also be imported as a module which then contains the following 46 functions: 47 48 * main - the main function of the script 49 * submit - calls mk_jobscript depending on operation mode and submits its 50 * mk_jobscript - creates the job script from a template 51 52 Type: submit.py --help 53 to get information about command line parameters. 54 Read the documentation for usage instructions. 55 ''' 41 56 42 57 # ------------------------------------------------------------------------------ … … 59 74 60 75 # ------------------------------------------------------------------------------ 61 # FUNCTIONS76 # METHODS 62 77 # ------------------------------------------------------------------------------ 63 78 … … 112 127 Parameters 113 128 ---------- 114 jtemplate : :obj:`string`129 jtemplate : str 115 130 Job template file from sub-directory "_templates" for 116 131 submission to ECMWF. It contains all necessary … … 119 134 Default is "job.temp". 120 135 121 c : :obj:`ControlFile`136 c : ControlFile 122 137 Contains all the parameters of CONTROL file and 123 138 command line. 124 139 125 queue : :obj:`string`140 queue : str 126 141 Name of queue for submission to ECMWF (e.g. ecgate or cca ) 127 142 … … 201 216 Parameters 202 217 ---------- 203 jtemplate : :obj:`string`218 jtemplate : str 204 219 Job template file from sub-directory "_templates" for 205 220 submission to ECMWF. It contains all necessary … … 208 223 Default is "job.temp". 209 224 210 job_file : :obj:`string`225 job_file : str 211 226 Path to the job script file. 212 227 213 clist : :obj:`list` of :obj:`string`228 clist : list of str 214 229 Contains all necessary parameters for ECMWF CONTROL file. 215 230
Note: See TracChangeset
for help on using the changeset viewer.