Changeset 2625ca8 in flex_extract.git for documentation/Sphinx/build/html/_modules/get_mars_data.html
- Timestamp:
- Jan 15, 2019, 1:03:37 AM (5 years ago)
- Branches:
- master, ctbto, dev
- Children:
- 6f951ca
- Parents:
- 0629ba8
- File:
-
- 1 edited
Legend:
- Unmodified
- Added
- Removed
-
documentation/Sphinx/build/html/_modules/get_mars_data.html
r274f9ef r2625ca8 175 175 <span class="c1"># online documentation)</span> 176 176 <span class="c1"># - use of UIFiles class for file selection and deletion</span> 177 <span class="c1">#</span> 177 <span class="c1"># - seperated get_mars_data function into several smaller pieces:</span> 178 <span class="c1"># write_reqheader, mk_server, mk_dates, remove_old, do_retrievment</span> 178 179 <span class="c1">#</span> 179 180 <span class="c1"># @License:</span> 180 <span class="c1"># (C) Copyright 2014-2018.</span> 181 <span class="c1"># (C) Copyright 2014-2019.</span> 182 <span class="c1"># Anne Philipp, Leopold Haimberger</span> 181 183 <span class="c1">#</span> 182 <span class="c1"># This software is licensed under the terms of the Apache Licence Version 2.0</span> 183 <span class="c1"># which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.</span> 184 <span class="c1">#</span> 185 <span class="c1"># @Program Functionality:</span> 186 <span class="c1"># This program can be used as a module in the whole flex_extract process</span> 187 <span class="c1"># or can be run by itself to just extract MARS data from ECMWF. To do so,</span> 188 <span class="c1"># a couple of necessary parameters has to be passed with the program call.</span> 189 <span class="c1"># See documentation for more details.</span> 190 <span class="c1">#</span> 191 <span class="c1"># @Program Content:</span> 192 <span class="c1"># - main</span> 193 <span class="c1"># - get_mars_data</span> 194 <span class="c1"># - do_retrievement</span> 195 <span class="c1">#</span> 184 <span class="c1"># This work is licensed under the Creative Commons Attribution 4.0</span> 185 <span class="c1"># International License. To view a copy of this license, visit</span> 186 <span class="c1"># http://creativecommons.org/licenses/by/4.0/ or send a letter to</span> 187 <span class="c1"># Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.</span> 196 188 <span class="c1">#*******************************************************************************</span> 197 189 <span class="sd">'''This script extracts MARS data from ECMWF servers.</span> 190 191 <span class="sd">At first, the necessary parameters from command line and CONTROL files are</span> 192 <span class="sd">extracted. They define the data set to be extracted from MARS.</span> 193 194 <span class="sd">This file can also be imported as a module and contains the following</span> 195 <span class="sd">functions:</span> 196 197 <span class="sd"> * main - the main function of the script</span> 198 <span class="sd"> * get_mars_data - overall control of ECMWF data retrievment</span> 199 <span class="sd"> * write_reqheader - writes the header into the mars_request file</span> 200 <span class="sd"> * mk_server - creates the server connection to ECMWF servers</span> 201 <span class="sd"> * mk_dates - defines the start and end date</span> 202 <span class="sd"> * remove_old - deletes old retrieved grib files</span> 203 <span class="sd"> * do_retrievement - creates individual retrievals</span> 204 205 <span class="sd">Type: get_mars_data.py --help</span> 206 <span class="sd">to get information about command line parameters.</span> 207 <span class="sd">Read the documentation for usage instructions.</span> 208 <span class="sd">'''</span> 198 209 <span class="c1"># ------------------------------------------------------------------------------</span> 199 210 <span class="c1"># MODULES</span> … … 205 216 206 217 <span class="c1"># software specific classes and modules from flex_extract</span> 218 <span class="c1"># add path to local main python path for flex_extract to get full access</span> 207 219 <span class="n">sys</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">dirname</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">abspath</span><span class="p">(</span> 208 220 <span class="n">inspect</span><span class="o">.</span><span class="n">getfile</span><span class="p">(</span><span class="n">inspect</span><span class="o">.</span><span class="n">currentframe</span><span class="p">())))</span> <span class="o">+</span> <span class="s1">'/../'</span><span class="p">)</span> 209 221 <span class="kn">import</span> <span class="nn">_config</span> 210 <span class="kn">from</span> <span class="nn">tools</span> <span class="k">import</span> <span class="p">(</span><span class="n">my_error</span><span class="p">,</span> <span class="n">normal_exit</span><span class="p">,</span> <span class="n">get_cmdline_arg uments</span><span class="p">,</span>222 <span class="kn">from</span> <span class="nn">tools</span> <span class="k">import</span> <span class="p">(</span><span class="n">my_error</span><span class="p">,</span> <span class="n">normal_exit</span><span class="p">,</span> <span class="n">get_cmdline_args</span><span class="p">,</span> 211 223 <span class="n">read_ecenv</span><span class="p">,</span> <span class="n">make_dir</span><span class="p">)</span> 212 224 <span class="kn">from</span> <span class="nn">classes.EcFlexpart</span> <span class="k">import</span> <span class="n">EcFlexpart</span> 213 225 <span class="kn">from</span> <span class="nn">classes.UioFiles</span> <span class="k">import</span> <span class="n">UioFiles</span> 226 <span class="kn">from</span> <span class="nn">classes.MarsRetrieval</span> <span class="k">import</span> <span class="n">MarsRetrieval</span> 214 227 215 228 <span class="k">try</span><span class="p">:</span> … … 235 248 <span class="sd"> '''</span> 236 249 237 <span class="n">args</span> <span class="o">=</span> <span class="n">get_cmdline_arg uments</span><span class="p">()</span>250 <span class="n">args</span> <span class="o">=</span> <span class="n">get_cmdline_args</span><span class="p">()</span> 238 251 <span class="n">c</span> <span class="o">=</span> <span class="n">ControlFile</span><span class="p">(</span><span class="n">args</span><span class="o">.</span><span class="n">controlfile</span><span class="p">)</span> 239 252 … … 244 257 245 258 <span class="n">get_mars_data</span><span class="p">(</span><span class="n">c</span><span class="p">)</span> 246 <span class="n">normal_exit</span><span class="p">(</span><span class="n">c</span><span class="o">.</span><span class="n">mail fail</span><span class="p">,</span> <span class="s1">'Done!'</span><span class="p">)</span>259 <span class="n">normal_exit</span><span class="p">(</span><span class="n">c</span><span class="o">.</span><span class="n">mailops</span><span class="p">,</span> <span class="n">c</span><span class="o">.</span><span class="n">queue</span><span class="p">,</span> <span class="s1">'Done!'</span><span class="p">)</span> 247 260 248 261 <span class="k">return</span></div> … … 257 270 <span class="sd"> Parameters</span> 258 271 <span class="sd"> ----------</span> 259 <span class="sd"> c : :obj:`ControlFile`</span>272 <span class="sd"> c : ControlFile</span> 260 273 <span class="sd"> Contains all the parameters of CONTROL file and</span> 261 274 <span class="sd"> command line.</span> … … 265 278 266 279 <span class="sd"> '''</span> 280 <span class="n">c</span><span class="o">.</span><span class="n">ecapi</span> <span class="o">=</span> <span class="n">ecapi</span> 267 281 268 282 <span class="k">if</span> <span class="ow">not</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">exists</span><span class="p">(</span><span class="n">c</span><span class="o">.</span><span class="n">inputdir</span><span class="p">):</span> 269 283 <span class="n">make_dir</span><span class="p">(</span><span class="n">c</span><span class="o">.</span><span class="n">inputdir</span><span class="p">)</span> 270 284 271 <span class="k">if</span> <span class="n">c</span><span class="o">.</span><span class="n">request</span> <span class="o">==</span> <span class="mi">0</span> <span class="ow">or</span> <span class="n">c</span><span class="o">.</span><span class="n">request</span> <span class="o">==</span> <span class="mi">2</span><span class="p">:</span>285 <span class="k">if</span> <span class="n">c</span><span class="o">.</span><span class="n">request</span> <span class="o">==</span> <span class="mi">0</span><span class="p">:</span> 272 286 <span class="nb">print</span><span class="p">(</span><span class="s2">"Retrieving EC data!"</span><span class="p">)</span> 273 <span class="k">elif</span> <span class="n">c</span><span class="o">.</span><span class="n">request</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span> 274 <span class="nb">print</span><span class="p">(</span><span class="s2">"Printing mars requests!"</span><span class="p">)</span> 287 <span class="k">else</span><span class="p">:</span> 288 <span class="k">if</span> <span class="n">c</span><span class="o">.</span><span class="n">request</span> <span class="o">==</span> <span class="mi">1</span><span class="p">:</span> 289 <span class="nb">print</span><span class="p">(</span><span class="s2">"Printing mars requests!"</span><span class="p">)</span> 290 <span class="k">elif</span> <span class="n">c</span><span class="o">.</span><span class="n">request</span> <span class="o">==</span> <span class="mi">2</span><span class="p">:</span> 291 <span class="nb">print</span><span class="p">(</span><span class="s2">"Retrieving EC data and printing mars request!"</span><span class="p">)</span> 292 <span class="n">write_reqheader</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">c</span><span class="o">.</span><span class="n">inputdir</span><span class="p">,</span> <span class="n">_config</span><span class="o">.</span><span class="n">FILE_MARS_REQUESTS</span><span class="p">))</span> 275 293 276 294 <span class="nb">print</span><span class="p">(</span><span class="s2">"start date </span><span class="si">%s</span><span class="s2"> "</span> <span class="o">%</span> <span class="p">(</span><span class="n">c</span><span class="o">.</span><span class="n">start_date</span><span class="p">))</span> 277 295 <span class="nb">print</span><span class="p">(</span><span class="s2">"end date </span><span class="si">%s</span><span class="s2"> "</span> <span class="o">%</span> <span class="p">(</span><span class="n">c</span><span class="o">.</span><span class="n">end_date</span><span class="p">))</span> 278 296 279 <span class="k">if</span> <span class="n">ecapi</span><span class="p">:</span> 297 <span class="n">server</span> <span class="o">=</span> <span class="n">mk_server</span><span class="p">(</span><span class="n">c</span><span class="p">)</span> 298 299 <span class="c1"># if data are to be retrieved, clean up any old grib files</span> 300 <span class="k">if</span> <span class="n">c</span><span class="o">.</span><span class="n">request</span> <span class="o">==</span> <span class="mi">0</span> <span class="ow">or</span> <span class="n">c</span><span class="o">.</span><span class="n">request</span> <span class="o">==</span> <span class="mi">2</span><span class="p">:</span> 301 <span class="n">remove_old</span><span class="p">(</span><span class="s1">'*grb'</span><span class="p">,</span> <span class="n">c</span><span class="o">.</span><span class="n">inputdir</span><span class="p">)</span> 302 303 <span class="c1"># -------------- flux data ------------------------------------------------</span> 304 <span class="n">start</span><span class="p">,</span> <span class="n">end</span><span class="p">,</span> <span class="n">datechunk</span> <span class="o">=</span> <span class="n">mk_dates</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">fluxes</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span> 305 <span class="n">do_retrievement</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">server</span><span class="p">,</span> <span class="n">start</span><span class="p">,</span> <span class="n">end</span><span class="p">,</span> <span class="n">datechunk</span><span class="p">,</span> <span class="n">fluxes</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span> 306 307 <span class="c1"># -------------- non flux data --------------------------------------------</span> 308 <span class="n">start</span><span class="p">,</span> <span class="n">end</span><span class="p">,</span> <span class="n">datechunk</span> <span class="o">=</span> <span class="n">mk_dates</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">fluxes</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span> 309 <span class="n">do_retrievement</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">server</span><span class="p">,</span> <span class="n">start</span><span class="p">,</span> <span class="n">end</span><span class="p">,</span> <span class="n">datechunk</span><span class="p">,</span> <span class="n">fluxes</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span> 310 311 <span class="k">return</span></div> 312 313 <div class="viewcode-block" id="write_reqheader"><a class="viewcode-back" href="../api.html#get_mars_data.write_reqheader">[docs]</a><span class="k">def</span> <span class="nf">write_reqheader</span><span class="p">(</span><span class="n">marsfile</span><span class="p">):</span> 314 <span class="sd">'''Writes header with column names into mars request file.</span> 315 316 <span class="sd"> Parameters</span> 317 <span class="sd"> ----------</span> 318 <span class="sd"> marsfile : str</span> 319 <span class="sd"> Path to the mars request file.</span> 320 321 <span class="sd"> Return</span> 322 <span class="sd"> ------</span> 323 324 <span class="sd"> '''</span> 325 <span class="n">MR</span> <span class="o">=</span> <span class="n">MarsRetrieval</span><span class="p">(</span><span class="kc">None</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span> 326 <span class="n">attrs</span> <span class="o">=</span> <span class="nb">vars</span><span class="p">(</span><span class="n">MR</span><span class="p">)</span><span class="o">.</span><span class="n">copy</span><span class="p">()</span> 327 <span class="k">del</span> <span class="n">attrs</span><span class="p">[</span><span class="s1">'server'</span><span class="p">]</span> 328 <span class="k">del</span> <span class="n">attrs</span><span class="p">[</span><span class="s1">'public'</span><span class="p">]</span> 329 <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">marsfile</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span> 330 <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'request_number'</span> <span class="o">+</span> <span class="s1">', '</span><span class="p">)</span> 331 <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">', '</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">key</span><span class="p">)</span> <span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">attrs</span><span class="o">.</span><span class="n">iterkeys</span><span class="p">())))</span> 332 <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="p">)</span> 333 334 <span class="k">return</span></div> 335 336 <div class="viewcode-block" id="mk_server"><a class="viewcode-back" href="../api.html#get_mars_data.mk_server">[docs]</a><span class="k">def</span> <span class="nf">mk_server</span><span class="p">(</span><span class="n">c</span><span class="p">):</span> 337 <span class="sd">'''Creates server connection if ECMWF WebAPI is available.</span> 338 339 <span class="sd"> Parameters</span> 340 <span class="sd"> ----------</span> 341 <span class="sd"> c : ControlFile</span> 342 <span class="sd"> Contains all the parameters of CONTROL file and</span> 343 <span class="sd"> command line.</span> 344 345 <span class="sd"> Return</span> 346 <span class="sd"> ------</span> 347 <span class="sd"> server : ECMWFDataServer or ECMWFService</span> 348 <span class="sd"> Connection to ECMWF server via python interface ECMWF WebAPI.</span> 349 350 <span class="sd"> '''</span> 351 <span class="k">if</span> <span class="n">c</span><span class="o">.</span><span class="n">ecapi</span><span class="p">:</span> 280 352 <span class="k">if</span> <span class="n">c</span><span class="o">.</span><span class="n">public</span><span class="p">:</span> 281 353 <span class="n">server</span> <span class="o">=</span> <span class="n">ecmwfapi</span><span class="o">.</span><span class="n">ECMWFDataServer</span><span class="p">()</span> … … 285 357 <span class="n">server</span> <span class="o">=</span> <span class="kc">False</span> 286 358 287 <span class="n">c</span><span class="o">.</span><span class="n">ecapi</span> <span class="o">=</span> <span class="n">ecapi</span>288 359 <span class="nb">print</span><span class="p">(</span><span class="s1">'Using ECMWF WebAPI: '</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">c</span><span class="o">.</span><span class="n">ecapi</span><span class="p">))</span> 289 360 290 <span class="c1"># basetime geht rückwärts</span> 291 292 <span class="c1"># if basetime 00</span> 293 <span class="c1"># dann wird von 12 am vortag bis 00 am start tag geholt</span> 294 <span class="c1"># aber ohne 12 selbst sondern 12 + step</span> 295 296 <span class="c1"># if basetime 12</span> 297 <span class="c1"># dann wird von 00 + step bis 12 am start tag geholt</span> 298 299 <span class="c1"># purer forecast wird vorwärts bestimmt.</span> 300 <span class="c1"># purer forecast mode ist dann wenn größer 24 stunden</span> 301 <span class="c1"># wie kann das noch festgestellt werden ????</span> 302 <span class="c1"># nur FC und steps mehr als 24 ?</span> 303 <span class="c1"># die einzige problematik beim reinen forecast ist die benennung der files!</span> 304 <span class="c1"># also sobald es Tagesüberschneidungen gibt</span> 305 <span class="c1"># allerdings ist das relevant und ersichtlich an den NICHT FLUSS DATEN</span> 306 361 <span class="k">return</span> <span class="n">server</span></div> 362 363 364 <div class="viewcode-block" id="mk_dates"><a class="viewcode-back" href="../api.html#get_mars_data.mk_dates">[docs]</a><span class="k">def</span> <span class="nf">mk_dates</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">fluxes</span><span class="p">):</span> 365 <span class="sd">'''Prepares start and end date depending on flux or non flux data.</span> 366 367 <span class="sd"> If forecast for maximum one day (upto 24h) are to be retrieved, then</span> 368 <span class="sd"> collect accumulation data (flux data) with additional days in the</span> 369 <span class="sd"> beginning and at the end (used for complete disaggregation of</span> 370 <span class="sd"> original period)</span> 371 372 <span class="sd"> If forecast data longer than 24h are to be retrieved, then</span> 373 <span class="sd"> collect accumulation data (flux data) with the exact start and end date</span> 374 <span class="sd"> (disaggregation will be done for the exact time period with</span> 375 <span class="sd"> boundary conditions)</span> 376 377 <span class="sd"> Since for basetime the extraction contains the 12 hours upfront,</span> 378 <span class="sd"> if basetime is 0, the starting date has to be the day before and</span> 379 380 <span class="sd"> Parameters</span> 381 <span class="sd"> ----------</span> 382 <span class="sd"> c : ControlFile</span> 383 <span class="sd"> Contains all the parameters of CONTROL file and</span> 384 <span class="sd"> command line.</span> 385 386 <span class="sd"> fluxes : boolean, optional</span> 387 <span class="sd"> Decides if the flux parameter settings are stored or</span> 388 <span class="sd"> the rest of the parameter list.</span> 389 <span class="sd"> Default value is False.</span> 390 391 <span class="sd"> Return</span> 392 <span class="sd"> ------</span> 393 <span class="sd"> start : datetime</span> 394 <span class="sd"> The start date of the retrieving data set.</span> 395 396 <span class="sd"> end : datetime</span> 397 <span class="sd"> The end date of the retrieving data set.</span> 398 399 <span class="sd"> chunk : datetime</span> 400 <span class="sd"> Time period in days for one single mars retrieval.</span> 401 402 <span class="sd"> '''</span> 307 403 <span class="n">start</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">strptime</span><span class="p">(</span><span class="n">c</span><span class="o">.</span><span class="n">start_date</span><span class="p">,</span> <span class="s1">'%Y%m</span><span class="si">%d</span><span class="s1">'</span><span class="p">)</span> 308 404 <span class="n">end</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">strptime</span><span class="p">(</span><span class="n">c</span><span class="o">.</span><span class="n">end_date</span><span class="p">,</span> <span class="s1">'%Y%m</span><span class="si">%d</span><span class="s1">'</span><span class="p">)</span> 309 <span class="c1"># time period for one single retrieval</span> 310 <span class="n">datechunk</span> <span class="o">=</span> <span class="n">timedelta</span><span class="p">(</span><span class="n">days</span><span class="o">=</span><span class="nb">int</span><span class="p">(</span><span class="n">c</span><span class="o">.</span><span class="n">date_chunk</span><span class="p">))</span> 311 312 <span class="k">if</span> <span class="n">c</span><span class="o">.</span><span class="n">basetime</span> <span class="o">==</span> <span class="s1">'00'</span><span class="p">:</span> 405 <span class="n">chunk</span> <span class="o">=</span> <span class="n">timedelta</span><span class="p">(</span><span class="n">days</span><span class="o">=</span><span class="nb">int</span><span class="p">(</span><span class="n">c</span><span class="o">.</span><span class="n">date_chunk</span><span class="p">))</span> 406 407 <span class="k">if</span> <span class="n">c</span><span class="o">.</span><span class="n">basetime</span><span class="p">:</span> 408 <span class="k">if</span> <span class="n">c</span><span class="o">.</span><span class="n">basetime</span> <span class="o">==</span> <span class="s1">'00'</span><span class="p">:</span> 409 <span class="n">start</span> <span class="o">=</span> <span class="n">start</span> <span class="o">-</span> <span class="n">timedelta</span><span class="p">(</span><span class="n">days</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> 410 411 <span class="k">if</span> <span class="ow">not</span> <span class="n">c</span><span class="o">.</span><span class="n">purefc</span> <span class="ow">and</span> <span class="n">fluxes</span><span class="p">:</span> 313 412 <span class="n">start</span> <span class="o">=</span> <span class="n">start</span> <span class="o">-</span> <span class="n">timedelta</span><span class="p">(</span><span class="n">days</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> 314 315 <span class="k">if</span> <span class="n">c</span><span class="o">.</span><span class="n">maxstep</span> <span class="o"><=</span> <span class="mi">24</span><span class="p">:</span> 316 <span class="n">startm1</span> <span class="o">=</span> <span class="n">start</span> <span class="o">-</span> <span class="n">timedelta</span><span class="p">(</span><span class="n">days</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> 317 318 <span class="k">if</span> <span class="n">c</span><span class="o">.</span><span class="n">basetime</span> <span class="o">==</span> <span class="s1">'00'</span> <span class="ow">or</span> <span class="n">c</span><span class="o">.</span><span class="n">basetime</span> <span class="o">==</span> <span class="s1">'12'</span><span class="p">:</span> 319 <span class="c1"># endp1 = end + timedelta(days=1)</span> 320 <span class="n">endp1</span> <span class="o">=</span> <span class="n">end</span> 321 <span class="k">else</span><span class="p">:</span> 322 <span class="c1"># endp1 = end + timedelta(days=2)</span> 323 <span class="n">endp1</span> <span class="o">=</span> <span class="n">end</span> <span class="o">+</span> <span class="n">timedelta</span><span class="p">(</span><span class="n">days</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> 324 325 <span class="c1"># -------------- flux data ------------------------------------------------</span> 326 <span class="k">if</span> <span class="n">c</span><span class="o">.</span><span class="n">request</span> <span class="o">==</span> <span class="mi">0</span> <span class="ow">or</span> <span class="n">c</span><span class="o">.</span><span class="n">request</span> <span class="o">==</span> <span class="mi">2</span><span class="p">:</span> 327 <span class="nb">print</span><span class="p">(</span><span class="s1">'... removing old flux content of '</span> <span class="o">+</span> <span class="n">c</span><span class="o">.</span><span class="n">inputdir</span><span class="p">)</span> 328 <span class="n">tobecleaned</span> <span class="o">=</span> <span class="n">UioFiles</span><span class="p">(</span><span class="n">c</span><span class="o">.</span><span class="n">inputdir</span><span class="p">,</span> 329 <span class="s1">'*_acc_*.'</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getppid</span><span class="p">())</span> <span class="o">+</span> <span class="s1">'.*.grb'</span><span class="p">)</span> 330 <span class="n">tobecleaned</span><span class="o">.</span><span class="n">delete_files</span><span class="p">()</span> 331 332 <span class="c1"># if forecast for maximum one day (upto 24h) are to be retrieved,</span> 333 <span class="c1"># collect accumulation data (flux data)</span> 334 <span class="c1"># with additional days in the beginning and at the end</span> 335 <span class="c1"># (used for complete disaggregation of original period)</span> 336 <span class="k">if</span> <span class="n">c</span><span class="o">.</span><span class="n">maxstep</span> <span class="o"><=</span> <span class="mi">24</span><span class="p">:</span> 337 <span class="n">do_retrievement</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">server</span><span class="p">,</span> <span class="n">startm1</span><span class="p">,</span> <span class="n">endp1</span><span class="p">,</span> <span class="n">datechunk</span><span class="p">,</span> <span class="n">fluxes</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span> 338 339 <span class="c1"># if forecast data longer than 24h are to be retrieved,</span> 340 <span class="c1"># collect accumulation data (flux data)</span> 341 <span class="c1"># with the exact start and end date</span> 342 <span class="c1"># (disaggregation will be done for the</span> 343 <span class="c1"># exact time period with boundary conditions)</span> 344 <span class="k">else</span><span class="p">:</span> 345 <span class="n">do_retrievement</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">server</span><span class="p">,</span> <span class="n">start</span><span class="p">,</span> <span class="n">end</span><span class="p">,</span> <span class="n">datechunk</span><span class="p">,</span> <span class="n">fluxes</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span> 346 347 <span class="c1"># -------------- non flux data --------------------------------------------</span> 348 <span class="k">if</span> <span class="n">c</span><span class="o">.</span><span class="n">request</span> <span class="o">==</span> <span class="mi">0</span> <span class="ow">or</span> <span class="n">c</span><span class="o">.</span><span class="n">request</span> <span class="o">==</span> <span class="mi">2</span><span class="p">:</span> 349 <span class="nb">print</span><span class="p">(</span><span class="s1">'... removing old non flux content of '</span> <span class="o">+</span> <span class="n">c</span><span class="o">.</span><span class="n">inputdir</span><span class="p">)</span> 350 <span class="n">tobecleaned</span> <span class="o">=</span> <span class="n">UioFiles</span><span class="p">(</span><span class="n">c</span><span class="o">.</span><span class="n">inputdir</span><span class="p">,</span> 351 <span class="s1">'*__*.'</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">getppid</span><span class="p">())</span> <span class="o">+</span> <span class="s1">'.*.grb'</span><span class="p">)</span> 352 <span class="n">tobecleaned</span><span class="o">.</span><span class="n">delete_files</span><span class="p">()</span> 353 354 <span class="n">do_retrievement</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">server</span><span class="p">,</span> <span class="n">start</span><span class="p">,</span> <span class="n">end</span><span class="p">,</span> <span class="n">datechunk</span><span class="p">,</span> <span class="n">fluxes</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span> 413 <span class="n">end</span> <span class="o">=</span> <span class="n">end</span> <span class="o">+</span> <span class="n">timedelta</span><span class="p">(</span><span class="n">days</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> 414 415 <span class="k">return</span> <span class="n">start</span><span class="p">,</span> <span class="n">end</span><span class="p">,</span> <span class="n">chunk</span></div> 416 417 <div class="viewcode-block" id="remove_old"><a class="viewcode-back" href="../api.html#get_mars_data.remove_old">[docs]</a><span class="k">def</span> <span class="nf">remove_old</span><span class="p">(</span><span class="n">pattern</span><span class="p">,</span> <span class="n">inputdir</span><span class="p">):</span> 418 <span class="sd">'''Deletes old retrieval files from current input directory</span> 419 <span class="sd"> matching the pattern.</span> 420 421 <span class="sd"> Parameters</span> 422 <span class="sd"> ----------</span> 423 <span class="sd"> pattern : str</span> 424 <span class="sd"> The sub string pattern which identifies the files to be deleted.</span> 425 426 <span class="sd"> inputdir : str, optional</span> 427 <span class="sd"> Path to the directory where the retrieved data is stored.</span> 428 429 <span class="sd"> Return</span> 430 <span class="sd"> ------</span> 431 432 <span class="sd"> '''</span> 433 <span class="nb">print</span><span class="p">(</span><span class="s1">'... removing old content of '</span> <span class="o">+</span> <span class="n">inputdir</span><span class="p">)</span> 434 435 <span class="n">tobecleaned</span> <span class="o">=</span> <span class="n">UioFiles</span><span class="p">(</span><span class="n">inputdir</span><span class="p">,</span> <span class="n">pattern</span><span class="p">)</span> 436 <span class="n">tobecleaned</span><span class="o">.</span><span class="n">delete_files</span><span class="p">()</span> 355 437 356 438 <span class="k">return</span></div> 439 357 440 358 441 <div class="viewcode-block" id="do_retrievement"><a class="viewcode-back" href="../api.html#get_mars_data.do_retrievement">[docs]</a><span class="k">def</span> <span class="nf">do_retrievement</span><span class="p">(</span><span class="n">c</span><span class="p">,</span> <span class="n">server</span><span class="p">,</span> <span class="n">start</span><span class="p">,</span> <span class="n">end</span><span class="p">,</span> <span class="n">delta_t</span><span class="p">,</span> <span class="n">fluxes</span><span class="o">=</span><span class="kc">False</span><span class="p">):</span> … … 362 445 <span class="sd"> Parameters</span> 363 446 <span class="sd"> ----------</span> 364 <span class="sd"> c : :obj:`ControlFile`</span>447 <span class="sd"> c : ControlFile</span> 365 448 <span class="sd"> Contains all the parameters of CONTROL file and</span> 366 449 <span class="sd"> command line.</span> 367 450 368 <span class="sd"> server : :obj:`ECMWFService`</span>451 <span class="sd"> server : ECMWFService or ECMWFDataServer</span> 369 452 <span class="sd"> The server connection to ECMWF.</span> 370 453 371 <span class="sd"> start : :obj:`datetime`</span>454 <span class="sd"> start : datetime</span> 372 455 <span class="sd"> The start date of the retrieval.</span> 373 456 374 <span class="sd"> end : :obj:`datetime`</span>457 <span class="sd"> end : datetime</span> 375 458 <span class="sd"> The end date of the retrieval.</span> 376 459 377 <span class="sd"> delta_t : :obj:`datetime`</span>460 <span class="sd"> delta_t : datetime</span> 378 461 <span class="sd"> Delta_t + 1 is the maximal time period of a single</span> 379 462 <span class="sd"> retrieval.</span> 380 463 381 <span class="sd"> fluxes : :obj:`boolean`, optional</span>464 <span class="sd"> fluxes : boolean, optional</span> 382 465 <span class="sd"> Decides if the flux parameters are to be retrieved or</span> 383 466 <span class="sd"> the rest of the parameter list.</span> … … 430 513 <div role="contentinfo"> 431 514 <p> 432 © Copyright 2018, Anne Philipp and Leopold Haimberger .515 © Copyright 2018, Anne Philipp and Leopold Haimberger 433 516 434 517 </p> … … 449 532 450 533 451 <script type="text/javascript"> 452 var DOCUMENTATION_OPTIONS = { 453 URL_ROOT:'../', 454 VERSION:'7.1 alpha', 455 LANGUAGE:'None', 456 COLLAPSE_INDEX:false, 457 FILE_SUFFIX:'.html', 458 HAS_SOURCE: true, 459 SOURCELINK_SUFFIX: '.txt' 460 }; 461 </script> 462 <script type="text/javascript" src="../_static/jquery.js"></script> 463 <script type="text/javascript" src="../_static/underscore.js"></script> 464 <script type="text/javascript" src="../_static/doctools.js"></script> 465 <script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script> 534 535 536 <script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script> 537 <script type="text/javascript" src="../_static/jquery.js"></script> 538 <script type="text/javascript" src="../_static/underscore.js"></script> 539 <script type="text/javascript" src="../_static/doctools.js"></script> 540 466 541 467 542
Note: See TracChangeset
for help on using the changeset viewer.