Changeset b1674ed in flex_extract.git for Documentation/html/quick_start.html
- Timestamp:
- Apr 1, 2020, 5:03:13 PM (4 years ago)
- Branches:
- master, ctbto, dev
- Children:
- d78851d, e0005c9
- Parents:
- 50f9ca6
- File:
-
- 1 edited
-
Documentation/html/quick_start.html (modified) (21 diffs)
Legend:
- Unmodified
- Added
- Removed
-
Documentation/html/quick_start.html
r30f7911 rb1674ed 22 22 23 23 <script type="text/javascript" id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script> 24 <script type="text/javascript"src="_static/jquery.js"></script>25 <script type="text/javascript"src="_static/underscore.js"></script>26 <script type="text/javascript"src="_static/doctools.js"></script>27 <script type="text/javascript"src="_static/language_data.js"></script>28 <script async="async" type="text/javascript"src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/latest.js?config=TeX-AMS-MML_HTMLorMML"></script>24 <script src="_static/jquery.js"></script> 25 <script src="_static/underscore.js"></script> 26 <script src="_static/doctools.js"></script> 27 <script src="_static/language_data.js"></script> 28 <script async="async" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/latest.js?config=TeX-AMS-MML_HTMLorMML"></script> 29 29 30 30 <script type="text/javascript" src="_static/js/theme.js"></script> … … 207 207 </div> 208 208 <p>The <code class="docutils literal notranslate"><span class="pre">Jobscripts</span></code> directory is used to store the Korn shell job scripts generated by a <code class="docutils literal notranslate"><span class="pre">flex_extract</span></code> run in the <strong>Remote</strong> or <strong>Gateway</strong> mode. They are used to submit the setup information to the ECMWF server and start the jobs in ECMWF’s batch mode. The typical user must not touch these files. They will be generated from template files which are stored in the <code class="docutils literal notranslate"><span class="pre">Templates</span></code> directory under <code class="docutils literal notranslate"><span class="pre">flex_extract_vX.X</span></code>. Usually there will be a <code class="docutils literal notranslate"><span class="pre">compilejob.ksh</span></code> and a <code class="docutils literal notranslate"><span class="pre">job.ksh</span></code> script which are explained in the section <a class="reference internal" href="Documentation/input.html"><span class="doc">Control & Input Data</span></a>. In the rare case of operational data extraction there will be a <code class="docutils literal notranslate"><span class="pre">joboper.ksh</span></code> which is designed to get the time parameters from environment variables at the ECMWF servers.</p> 209 <p>The <code class="docutils literal notranslate"><span class="pre">Controls</span></code> directory contains a number of example <code class="docutils literal notranslate"><span class="pre">CONTROL</span></code> files. These ``CONTROL``files represent the current range of possible dataset retrievals with <code class="docutils literal notranslate"><span class="pre">flex_extract</span></code>. Some parameters in the <code class="docutils literal notranslate"><span class="pre">CONTROL</span></code> files can be adapted and some others should not be changed. In this <a class="reference internal" href="#"><span class="doc">Quick Start</span></a> guide we explain how an extraction with <code class="docutils literal notranslate"><span class="pre">flex_extract</span></code> can be started in the different <a class="reference internal" href="Documentation/Overview/app_modes.html"><span class="doc">Application Modes</span></a> and point out some specifics of each dataset and <code class="docutils literal notranslate"><span class="pre">CONTROL</span></code> file.</p>209 <p>The <code class="docutils literal notranslate"><span class="pre">Controls</span></code> directory contains a number of example <code class="docutils literal notranslate"><span class="pre">CONTROL</span></code> files. These <code class="docutils literal notranslate"><span class="pre">CONTROL</span></code> files represent the current range of possible dataset retrievals with <code class="docutils literal notranslate"><span class="pre">flex_extract</span></code>. Some parameters in the <code class="docutils literal notranslate"><span class="pre">CONTROL</span></code> files can be adapted and some others should not be changed. In this <a class="reference internal" href="#"><span class="doc">Quick Start</span></a> guide we explain how an extraction with <code class="docutils literal notranslate"><span class="pre">flex_extract</span></code> can be started in the different <a class="reference internal" href="Documentation/Overview/app_modes.html"><span class="doc">Application Modes</span></a> and point out some specifics of each dataset and <code class="docutils literal notranslate"><span class="pre">CONTROL</span></code> file.</p> 210 210 <p>Directly under <code class="docutils literal notranslate"><span class="pre">Run</span></code> you find the files <code class="docutils literal notranslate"><span class="pre">run.sh</span></code> and <code class="docutils literal notranslate"><span class="pre">run_local.sh</span></code> and according to your selected <a class="reference internal" href="Documentation/Overview/app_modes.html"><span class="doc">Application Modes</span></a> there might also be a file named <code class="docutils literal notranslate"><span class="pre">ECMWF_ENV</span></code> for the user credentials to quickly and automatically access ECMWF servers.</p> 211 211 <p>From version 7.1 on, the <code class="docutils literal notranslate"><span class="pre">run.sh</span></code> (or <code class="docutils literal notranslate"><span class="pre">run_local.sh</span></code>) script is the main access point to <code class="docutils literal notranslate"><span class="pre">flex_extract</span></code>.</p> 212 212 <div class="admonition note"> 213 <p class=" firstadmonition-title">Note</p>214 <p class="last">Note that, for experienced users (or users of older versions), it is still possible to access <code class="docutils literal notranslate"><span class="pre">flex_extract</span></code> directly via the <code class="docutils literal notranslate"><span class="pre">submit.py</span></code> script which can be found in the directory <code class="docutils literal notranslate"><span class="pre">flex_extract_vX.X/Source/Python</span></code>.</p>213 <p class="admonition-title">Note</p> 214 <p>Note that, for experienced users (or users of older versions), it is still possible to access <code class="docutils literal notranslate"><span class="pre">flex_extract</span></code> directly via the <code class="docutils literal notranslate"><span class="pre">submit.py</span></code> script which can be found in the directory <code class="docutils literal notranslate"><span class="pre">flex_extract_vX.X/Source/Python</span></code>.</p> 215 215 </div> 216 216 <div class="section" id="job-preparation"> … … 220 220 <h3>Remote and gateway modes<a class="headerlink" href="#remote-and-gateway-modes" title="Permalink to this headline">¶</a></h3> 221 221 <p>For member state users it is recommended to use the <em>remote</em> or <em>gateway</em> mode, especially for more demanding tasks, to retrieve and convert data on ECMWF machines and to transfer only the final output files to the local host.</p> 222 <dl class="docutils"> 223 <dt>Remote mode</dt> 224 <dd><p class="first">The only difference between both modes is the users working location. In the <em>remote</em> mode you have to login to the ECMWF server and then go to the <code class="docutils literal notranslate"><span class="pre">Run</span></code> directory as shown above. At ECMWF servers <code class="docutils literal notranslate"><span class="pre">flex_extract</span></code> is installed in the <code class="docutils literal notranslate"><span class="pre">$HOME</span></code> directory. However, to be able to start the program you have to load the <code class="docutils literal notranslate"><span class="pre">Python3</span></code> environment with the module system first.</p> 222 <dl> 223 <dt>Remote mode</dt><dd><p>The only difference between both modes is the users working location. In the <em>remote</em> mode you have to login to the ECMWF server and then go to the <code class="docutils literal notranslate"><span class="pre">Run</span></code> directory as shown above. At ECMWF servers <code class="docutils literal notranslate"><span class="pre">flex_extract</span></code> is installed in the <code class="docutils literal notranslate"><span class="pre">$HOME</span></code> directory. However, to be able to start the program you have to load the <code class="docutils literal notranslate"><span class="pre">Python3</span></code> environment with the module system first.</p> 225 224 <div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># Remote mode</span> 226 225 ssh -X <ecuid>@ecaccess.ecmwf.int 227 226 </pre></div> 228 227 </div> 229 <div class=" lasthighlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># On ECMWF server</span>228 <div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># On ECMWF server</span> 230 229 <span class="o">[</span><ecuid>@ecgb11 ~<span class="o">]</span>$ module load python3 231 230 <span class="o">[</span><ecuid>@ecgb11 ~<span class="o">]</span>$ <span class="nb">cd</span> flex_extract_vX.X/Run … … 233 232 </div> 234 233 </dd> 235 <dt>Gateway mode</dt> 236 <dd><p class="first">For the gateway mode you have to log in on the gateway server and go to the <code class="docutils literal notranslate"><span class="pre">Run</span></code> directory of <code class="docutils literal notranslate"><span class="pre">flex_extract</span></code>:</p> 237 <div class="last highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># Gateway mode</span> 234 <dt>Gateway mode</dt><dd><p>For the gateway mode you have to log in on the gateway server and go to the <code class="docutils literal notranslate"><span class="pre">Run</span></code> directory of <code class="docutils literal notranslate"><span class="pre">flex_extract</span></code>:</p> 235 <div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># Gateway mode</span> 238 236 ssh <user>@<gatewayserver> 239 237 <span class="nb">cd</span> <path-to-flex_extract_vX.X>/Run … … 299 297 </div> 300 298 </div> 301 <p>If the job was submitted to the HPC ( <code class="docutils literal notranslate"><span class="pre">queue=cca</span></code> ) you may login to the HPC and look into the directory <code class="docutils literal notranslate"><span class="pre">/scratch/ms/ECGID/ECUID/.ecaccess_do_not_remove</span></code> for job logs. The working directories are deleted after job failure and thus normally cannot be accessed.</p>299 <p>If the job was submitted to the HPC ( <code class="docutils literal notranslate"><span class="pre">queue=cca</span></code> or <code class="docutils literal notranslate"><span class="pre">queue=ccb</span></code> ) you may login to the HPC and look into the directory <code class="docutils literal notranslate"><span class="pre">/scratch/ms/ECGID/ECUID/.ecaccess_do_not_remove</span></code> for job logs. The working directories are deleted after job failure and thus normally cannot be accessed.</p> 302 300 <p>To check if the resulting files are still transferred to local gateway server you can use the command <code class="docutils literal notranslate"><span class="pre">ecaccess-ectrans-list</span></code> or check the destination path for resulting files on your local gateway server.</p> 303 301 </div> … … 306 304 <p>To get to know the working process and to start your first submission you could use one of the example <code class="docutils literal notranslate"><span class="pre">CONTROL</span></code> files stored in the <code class="docutils literal notranslate"><span class="pre">Control</span></code> directory as they are. For quick results and for testing reasons it is recommended to extract <em>CERA-20C</em> data.</p> 307 305 <p>Open the <code class="docutils literal notranslate"><span class="pre">run_local.sh</span></code> file and modify the parameter block marked in the file as shown below. The differences are highlighted.</p> 308 <table border="1" class="docutils">306 <table class="docutils align-default"> 309 307 <colgroup> 310 <col width="50%" />311 <col width="50%" />308 <col style="width: 50%" /> 309 <col style="width: 50%" /> 312 310 </colgroup> 313 <tbody valign="top">314 <tr class="row-odd"><td> Take this for <strong>member-state user</strong></td>315 <td> Take this for <strong>public user</strong></td>311 <tbody> 312 <tr class="row-odd"><td><p>Take this for <strong>member-state user</strong></p></td> 313 <td><p>Take this for <strong>public user</strong></p></td> 316 314 </tr> 317 <tr class="row-even"><td><div class=" first lasthighlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># -----------------------------------------</span>315 <tr class="row-even"><td><div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># -----------------------------------------</span> 318 316 <span class="c1"># AVAILABLE COMMANDLINE ARGUMENTS TO SET</span> 319 317 <span class="c1">#</span> … … 341 339 </div> 342 340 </td> 343 <td><div class=" first lasthighlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># -----------------------------------------</span>341 <td><div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># -----------------------------------------</span> 344 342 <span class="c1"># AVAILABLE COMMANDLINE ARGUMENTS TO SET</span> 345 343 <span class="c1">#</span> … … 378 376 <blockquote> 379 377 <div><ol class="arabic simple"> 380 <li> There is a maximum size of 20GB for single retrieval via ECMWF Web API. Normally this is not a problem but for global fields with T1279 resolution and hourly time steps the limit may already apply.</li>381 <li> If the retrieved MARS files are large but the resulting files are relative small (small local domain) then the retrieval to the local host may be inefficient since all data must be transferred via the Internet. This scenario applies most notably if <code class="docutils literal notranslate"><span class="pre">etadot</span></code> has to be calculated via the continuity equation as this requires global fields even if the domain is local. In this case job submission via ecgate might be a better choice. It really depends on the use patterns and also on the internet connection speed.</li>378 <li><p>There is a maximum size of 20GB for single retrieval via ECMWF Web API. Normally this is not a problem but for global fields with T1279 resolution and hourly time steps the limit may already apply.</p></li> 379 <li><p>If the retrieved MARS files are large but the resulting files are relative small (small local domain) then the retrieval to the local host may be inefficient since all data must be transferred via the Internet. This scenario applies most notably if <code class="docutils literal notranslate"><span class="pre">etadot</span></code> has to be calculated via the continuity equation as this requires global fields even if the domain is local. In this case job submission via ecgate might be a better choice. It really depends on the use patterns and also on the internet connection speed.</p></li> 382 380 </ol> 383 381 </div></blockquote> … … 390 388 <blockquote> 391 389 <div><ul class="simple"> 392 <li>< strong>Public users</strong> can use a web mask to check on data or list available data at this <a class="reference external" href="https://apps.ecmwf.int/datasets/">Public datasets web interface</a>.</li>393 <li>< strong>Member state users</strong> can check availability of data online in the <a class="reference external" href="https://apps.ecmwf.int/mars-catalogue/">MARS catalogue</a>.</li>390 <li><p><strong>Public users</strong> can use a web mask to check on data or list available data at this <a class="reference external" href="https://apps.ecmwf.int/datasets/">Public datasets web interface</a>.</p></li> 391 <li><p><strong>Member state users</strong> can check availability of data online in the <a class="reference external" href="https://apps.ecmwf.int/mars-catalogue/">MARS catalogue</a>.</p></li> 394 392 </ul> 395 393 </div></blockquote> 396 394 <p>There you can select step by step what data suits your needs. This would be the most straightforeward way of checking for available data and therefore limit the possibility of <code class="docutils literal notranslate"><span class="pre">flex_extract</span></code> to fail. The following figure gives an example how the web interface would look like:</p> 397 <div class="figure " id="ref-fig-mars-catalogue-ss">395 <div class="figure align-default" id="ref-fig-mars-catalogue-ss"> 398 396 <img alt="_images/MARS_catalogue_snapshot.png" src="_images/MARS_catalogue_snapshot.png" /> 399 397 </div> 400 <p>!!!!!!!! ADD HERE ANOTHER SCREENSHOT OF THE PARAMETER SELECTION AREA ( HAS TO BE DONE AT HOME )</p>401 398 <p>Additionally, you can find a lot of helpful links to dataset documentations, direct links to specific dataset web catalogues or further general information at the <a class="reference external" href="Ecmwf/ec-links.html">link collection</a> in the ECMWF data section.</p> 402 399 <p><code class="docutils literal notranslate"><span class="pre">Flex_extract</span></code> is specialised to retrieve a limited number of datasets, namely <em>ERA-Interim</em>, <em>CERA-20C</em>, <em>ERA5</em> and <em>HRES (operational data)</em> as well as the <em>ENS (operational data, 15-day forecast)</em>. The limitation relates mainly to the dataset itself, the stream (what kind of forecast or what subset of dataset) and the experiment number. Mostly, the experiment number is equal to <code class="docutils literal notranslate"><span class="pre">1</span></code> to signal that the actual version should be used.</p> … … 430 427 </div> 431 428 <p>The main differences and features in the datasets are listed in the table shown below:</p> 432 <div class="figure" id="id5"> 433 <span id="ref-tab-dataset-cmp"></span><img alt="_images/dataset_cmp_table.png" src="_images/dataset_cmp_table.png" /> 434 <p class="caption"><span class="caption-text">DO THIS TABLE AGAIN BY HAND!</span></p> 429 <div class="figure align-default" id="ref-tab-dataset-cmp"> 430 <img alt="_images/dataset_cmp_table.png" src="_images/dataset_cmp_table.png" /> 435 431 </div> 436 432 <p>A common problem for beginners in retrieving ECMWF datasets is the mismatch in the definition of these parameters. For example, if you would like to retrieve operational data before <code class="docutils literal notranslate"><span class="pre">June</span> <span class="pre">25th</span> <span class="pre">2013</span></code> and set the maximum level to <code class="docutils literal notranslate"><span class="pre">137</span></code> you will get an error because this number of levels was first introduced at this effective day. So, be cautious in the combination of space and time resolution as well as the field types which are not available all the time.</p> 437 433 <div class="admonition note"> 438 <p class=" firstadmonition-title">Note</p>439 <p class="last">Sometimes it might not be clear how specific parameters in the control file must be set in terms of format. Please see the description of the parameters in section <a class="reference external" href="Documentation/Input/control_params.html">CONTROL parameters</a> or have a look at the ECMWF user documentation for <a class="reference external" href="https://confluence.ecmwf.int/display/UDOC/MARS+keywords">MARS keywords</a></p>434 <p class="admonition-title">Note</p> 435 <p>Sometimes it might not be clear how specific parameters in the control file must be set in terms of format. Please see the description of the parameters in section <a class="reference external" href="Documentation/Input/control_params.html">CONTROL parameters</a> or have a look at the ECMWF user documentation for <a class="reference external" href="https://confluence.ecmwf.int/display/UDOC/MARS+keywords">MARS keywords</a></p> 440 436 </div> 441 437 <p>In the following we shortly discuss the main retrieval opportunities of the different datasets and categoize the <code class="docutils literal notranslate"><span class="pre">CONTROL</span></code> files.</p> … … 450 446 <p>For <em>CERA-20C</em> it seems that there are no differences in the dataset against the full dataset, while the <em>public ERA-Interim</em> has only analysis fields every 6 hour without filling forecasts in between for model levels. Therefore it is only possible to retrieve 6-hourly data for <em>public ERA-Interim</em>.</p> 451 447 <div class="admonition note"> 452 <p class=" firstadmonition-title">Note</p>453 <p class="last">In general, <em>ERA5</em> is a public dataset. However, since the model levels are not yet publicly available, it is not possible to retrieve <em>ERA5</em> data to drive the <code class="docutils literal notranslate"><span class="pre">FLEXPART</span></code> model. As soon as this is possible it will be announced at the community website and per newsletter.</p>448 <p class="admonition-title">Note</p> 449 <p>In general, <em>ERA5</em> is a public dataset. However, since the model levels are not yet publicly available, it is not possible to retrieve <em>ERA5</em> data to drive the <code class="docutils literal notranslate"><span class="pre">FLEXPART</span></code> model. As soon as this is possible it will be announced at the community website and per newsletter.</p> 454 450 </div> 455 451 </div> … … 466 462 The forecast starting time is <code class="docutils literal notranslate"><span class="pre">06/18</span> <span class="pre">UTC</span></code> which is important for the flux data. This should be set in the <code class="docutils literal notranslate"><span class="pre">CONTROL</span></code> file via the <code class="docutils literal notranslate"><span class="pre">ACCTIME</span> <span class="pre">06/18</span></code> parameter in correspondence with <code class="docutils literal notranslate"><span class="pre">ACCMAXSTEP</span> <span class="pre">12</span></code> and <code class="docutils literal notranslate"><span class="pre">ACCTYPE</span> <span class="pre">FC</span></code>.</p> 467 463 <div class="admonition note"> 468 <p class=" firstadmonition-title">Note</p>469 <p class="last">We know that <em>ERA5</em> also has an ensemble data assimilation system but this is not yet retrievable with <code class="docutils literal notranslate"><span class="pre">flex_extract</span></code> since the deaccumulation of the flux fields works differently in this stream. Ensemble retrieval for <em>ERA5</em> is a future ToDo.</p>464 <p class="admonition-title">Note</p> 465 <p>We know that <em>ERA5</em> also has an ensemble data assimilation system but this is not yet retrievable with <code class="docutils literal notranslate"><span class="pre">flex_extract</span></code> since the deaccumulation of the flux fields works differently in this stream. Ensemble retrieval for <em>ERA5</em> is a future ToDo.</p> 470 466 </div> 471 467 </div> … … 473 469 <h3>ERA-Interim<a class="headerlink" href="#era-interim" title="Permalink to this headline">¶</a></h3> 474 470 <p>This re-analysis dataset will exceed its end of production at 31st August 2019! 475 It is then available from 1st January 1979 to 31st August 2019. The <code class="docutils literal notranslate"><span class="pre">etadot</span></code> is not available in this dataset. Therefore <code class="docutils literal notranslate"><span class="pre">flex_extract</span></code> must select the <code class="docutils literal notranslate"><span class="pre">GAUSS</span></code> parameter to retrieve the divergence field in addition. The vertical velocity is the calculated with the continuity equation in the Fortran program <code class="docutils literal notranslate"><span class="pre">CONVERT2</span></code>. Since the analysis fields are only available for every 6th hour, the dataset can be made 3 hourly by adding forecast fields in between. No ensemble members are available.</p> 476 <div class="admonition-todo admonition" id="index-0"> 477 <p class="first admonition-title">Todo</p> 478 <p class="last">@LEO: please check the complete description and functionality of the CONTROL FILEs</p> 479 </div> 471 It is then available from 1st January 1979 to 31st August 2019. The <code class="docutils literal notranslate"><span class="pre">etadot</span></code> is not available in this dataset. Therefore <code class="docutils literal notranslate"><span class="pre">flex_extract</span></code> must select the <code class="docutils literal notranslate"><span class="pre">GAUSS</span></code> parameter to retrieve the divergence field in addition. The vertical velocity is the calculated with the continuity equation in the Fortran program <code class="docutils literal notranslate"><span class="pre">calc_etadot</span></code>. Since the analysis fields are only available for every 6th hour, the dataset can be made 3 hourly by adding forecast fields in between. No ensemble members are available.</p> 480 472 </div> 481 473 <div class="section" id="operational-data"> 482 474 <h3>Operational data<a class="headerlink" href="#operational-data" title="Permalink to this headline">¶</a></h3> 483 <p>This is the real time atmospheric model in high resolution with a 10-day forecast. This means it underwent regular adaptations and improvements over the years. Hence, retrieving data from this dataset needs extra attention in selecting correct settings of parameter. See < a class="reference internal" href="#ref-tab-dataset-cmp"><span class="std std-ref">DO THIS TABLE AGAIN BY HAND!</span></a> for the most important parameters.475 <p>This is the real time atmospheric model in high resolution with a 10-day forecast. This means it underwent regular adaptations and improvements over the years. Hence, retrieving data from this dataset needs extra attention in selecting correct settings of parameter. See <span class="xref std std-ref">ref-tab-dataset-cmp</span> for the most important parameters. 484 476 Nowadays, it is available 1 hourly by filling the gaps of the 6 hourly analysis fields with 1 hourly forecast fields. Since 4th June 2008 the eta coordinate is directly available so that <code class="docutils literal notranslate"><span class="pre">ETA</span></code> should be set to <code class="docutils literal notranslate"><span class="pre">1</span></code> to save computation time. The horizontal resolution can be up to <code class="docutils literal notranslate"><span class="pre">0.1°</span></code> and in combination with <code class="docutils literal notranslate"><span class="pre">137</span></code> vertical levels can lead to troubles in retrieving this high resolution dataset in terms of job duration and quota exceedence. 485 477 It is recommended to submit such high resolution cases for single day retrievals (see <code class="docutils literal notranslate"><span class="pre">JOB_CHUNK</span></code> parameter in <code class="docutils literal notranslate"><span class="pre">run.sh</span></code> script) to avoid job failures due to exceeding limits.</p> … … 492 484 </div> 493 485 <p>These files defines the minimum number of parameters necessary to retrieve a daily subset. The setup of field types is optimal and should only be changed if the user understands what he does. The grid, domain and temporal resolution can be changed according to availability.</p> 494 <div class="admonition-todo admonition" id="index-1">495 <p class="first admonition-title">Todo</p>496 <p class="last">@LEO - explain the setup with 4V fields! Who can extract it, when would this be useful?</p>497 </div>498 486 <div class="admonition note"> 499 <p class="first admonition-title">Note</p> 500 <p class="last">Please see <a class="reference external" href="https://confluence.ecmwf.int/display/UDOC/Retrieve#Retrieve-Retrievalefficiency">Information about MARS retrievement</a> to get to know hints about retrieval efficiency and troubleshooting.</p> 501 </div> 502 <dl class="docutils"> 503 <dt>Pure forecast</dt> 504 <dd><p class="first">It is possible to retrieve pure forecasts exceeding a day. The forecast period available depends on the date and forecast field type. Please use MARS catalogue to check the availability. Below are some examples for 36 hour forecast of <em>Forecast (FC)</em>, <em>Control forecast (CF)</em> and <em>Calibration/Validation forecast (CV)</em>. 487 <p class="admonition-title">Note</p> 488 <p>Please see <a class="reference external" href="https://confluence.ecmwf.int/display/UDOC/Retrieve#Retrieve-Retrievalefficiency">Information about MARS retrievement</a> to get to know hints about retrieval efficiency and troubleshooting.</p> 489 </div> 490 <dl> 491 <dt>Pure forecast</dt><dd><p>It is possible to retrieve pure forecasts exceeding a day. The forecast period available depends on the date and forecast field type. Please use MARS catalogue to check the availability. Below are some examples for 36 hour forecast of <em>Forecast (FC)</em>, <em>Control forecast (CF)</em> and <em>Calibration/Validation forecast (CV)</em>. 505 492 The <em>CV</em> field type was only available 3-hourly from 2006 up to 2016. It is recommended to use the <em>CF</em> type since this is available from 1992 (3-hourly) on up to today in 1-hourly temporal resolution. <em>CV</em> and <em>CF</em> field types belong to the <em>Ensemble prediction system (ENFO)</em> which contain 50 ensemble members. 506 493 Please be aware that in this case it is necessary to set the specific type for flux fields explicitly, otherwise it could select a default value which might be different from what you expect!</p> 507 <div class=" lasthighlight-bash notranslate"><div class="highlight"><pre><span></span>CONTROL_OD.ENFO.CF.36hours494 <div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>CONTROL_OD.ENFO.CF.36hours 508 495 CONTROL_OD.ENFO.CV.36hours 509 496 CONTROL_OD.OPER.FC.36hours … … 511 498 </div> 512 499 </dd> 513 <dt>Half-day retrievals</dt> 514 <dd><p class="first">If a forecast for just half a day is wanted it can be done by substituting the analysis fields also by forecast fields as shown in files with <code class="docutils literal notranslate"><span class="pre">twiceaday</span></code> in it. They produce a full day retrieval with pure 12 hour forecasts twice a day. It is also possible to use the operational version which would get the time information from ECMWF’s environmental variables and therefore get the newest forecast per day. This version uses a <code class="docutils literal notranslate"><span class="pre">BASETIME</span></code> parameter which tells MARS to extract the exact 12 hours upfront to the selected date. If the <code class="docutils literal notranslate"><span class="pre">CONTROL</span></code> file with <code class="docutils literal notranslate"><span class="pre">basetime</span></code> in the filename is used this can be done for any other date too.</p> 515 <div class="last highlight-bash notranslate"><div class="highlight"><pre><span></span>CONTROL_OD.OPER.FC.eta.basetime 500 <dt>Half-day retrievals</dt><dd><p>If a forecast for just half a day is wanted it can be done by substituting the analysis fields also by forecast fields as shown in files with <code class="docutils literal notranslate"><span class="pre">twiceaday</span></code> in it. They produce a full day retrieval with pure 12 hour forecasts twice a day. It is also possible to use the operational version which would get the time information from ECMWF’s environmental variables and therefore get the newest forecast per day. This version uses a <code class="docutils literal notranslate"><span class="pre">BASETIME</span></code> parameter which tells MARS to extract the exact 12 hours upfront to the selected date. If the <code class="docutils literal notranslate"><span class="pre">CONTROL</span></code> file with <code class="docutils literal notranslate"><span class="pre">basetime</span></code> in the filename is used this can be done for any other date too.</p> 501 <div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>CONTROL_OD.OPER.FC.eta.basetime 516 502 CONTROL_OD.OPER.FC.operational 517 503 CONTROL_OD.OPER.FC.twiceaday.1hourly … … 520 506 </div> 521 507 </dd> 522 <dt>Ensemble members</dt> 523 <dd><p class="first">The retrieval of ensemble members were already mentioned in the pure forecast section and for <em>CERA-20C</em> data. 508 <dt>Ensemble members</dt><dd><p>The retrieval of ensemble members were already mentioned in the pure forecast section and for <em>CERA-20C</em> data. 524 509 In this <code class="docutils literal notranslate"><span class="pre">flex_extract</span></code> version there is an additional possibility to retrieve the <em>Ensemble Long window Data Assimilation (ELDA)</em> stream from the real-time dataset. This model version has (up to May 2019) 25 ensemble members and a control run (<code class="docutils literal notranslate"><span class="pre">number</span> <span class="pre">0</span></code>). Starting from June 2019 it has 50 ensemble members. Therefore we created the possibility to double up the 25 ensemble members (before June 2019) to 50 members by taking the original 25 members from MARS and subtracting 2 times the difference between the member value and the control value. This is done by selecting the parameter <code class="docutils literal notranslate"><span class="pre">DOUBLEELDA</span></code> and set it to <code class="docutils literal notranslate"><span class="pre">1</span></code>.</p> 525 <div class=" lasthighlight-bash notranslate"><div class="highlight"><pre><span></span>CONTROL_OD.ELDA.FC.eta.ens.double510 <div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>CONTROL_OD.ELDA.FC.eta.ens.double 526 511 CONTROL_OD.ENFO.PF.ens 527 512 </pre></div> … … 529 514 </dd> 530 515 </dl> 531 <div class="admonition-todo admonition" id="index-2">532 <p class="first admonition-title">Todo</p>533 <p class="last">@LEO: Please tell me why perturbed forecast are possible? Is it because of some user rights? I have no opportunity of retrieve PF data.</p>534 </div>535 516 </div> 536 517 <div class="section" id="specific-features"> 537 518 <h3>Specific features<a class="headerlink" href="#specific-features" title="Permalink to this headline">¶</a></h3> 538 <dl class=" docutils">539 <dt>rrint</dt> 540 < dd>Decides if the precipitation flux data uses the old (<code class="docutils literal notranslate"><span class="pre">0</span></code>) or new (<code class="docutils literal notranslate"><span class="pre">1</span></code>) disaggregation scheme. See <a class="reference internal" href="Documentation/disagg.html"><span class="doc">Disaggregation of Flux Data</span></a> for explanaition.</dd>541 <dt>cwc</dt> 542 < dd>Decides if the total cloud water content will be retrieved (set to <code class="docutils literal notranslate"><span class="pre">1</span></code>) in addition. This is the sum of cloud liquid and cloud ice water content.</dd>543 <dt>addpar</dt> 544 < dd>With this parameter an additional list of 2-dimensional, non-flux parameters can be retrieved. Use format <code class="docutils literal notranslate"><span class="pre">param1/param2/.../paramx</span></code> to list the parameters. Please be consistent in using either the parameter IDs or the short names.</dd>545 <dt>doubleelda</dt> 546 < dd>Use this to double the ensemble member number by adding further disturbance to each member.</dd>547 <dt>debug</dt> 548 < dd>If set to <code class="docutils literal notranslate"><span class="pre">1</span></code> all temporary files were kept at the end. Otherwise everything except the final output files will be deleted.</dd>549 <dt>request</dt> 550 < dd>This produces an extra <em>csv</em> file <code class="docutils literal notranslate"><span class="pre">mars_requests.csv</span></code> where the content of each mars request of the job is stored. Useful for debugging and documentation.</dd>551 <dt>mailfail</dt> 552 < dd>At default the mail is send to the mail connected with the user account. Add additional email addresses if you want. But as soon as you enter a new mail, the default will be overwritten. If you would like to keep the mail from your user account, please add <code class="docutils literal notranslate"><span class="pre">${USER}</span></code> to the list ( comma seperated ) or mail addresses.</dd>519 <dl class="simple"> 520 <dt>rrint</dt><dd><p>Decides if the precipitation flux data uses the old (<code class="docutils literal notranslate"><span class="pre">0</span></code>) or new (<code class="docutils literal notranslate"><span class="pre">1</span></code>) disaggregation scheme. See <a class="reference internal" href="Documentation/disagg.html"><span class="doc">Disaggregation of Flux Data</span></a> for explanaition.</p> 521 </dd> 522 <dt>cwc</dt><dd><p>Decides if the total cloud water content will be retrieved (set to <code class="docutils literal notranslate"><span class="pre">1</span></code>) in addition. This is the sum of cloud liquid and cloud ice water content.</p> 523 </dd> 524 <dt>addpar</dt><dd><p>With this parameter an additional list of 2-dimensional, non-flux parameters can be retrieved. Use format <code class="docutils literal notranslate"><span class="pre">param1/param2/.../paramx</span></code> to list the parameters. Please be consistent in using either the parameter IDs or the short names.</p> 525 </dd> 526 <dt>doubleelda</dt><dd><p>Use this to double the ensemble member number by adding further disturbance to each member.</p> 527 </dd> 528 <dt>debug</dt><dd><p>If set to <code class="docutils literal notranslate"><span class="pre">1</span></code> all temporary files were kept at the end. Otherwise everything except the final output files will be deleted.</p> 529 </dd> 530 <dt>request</dt><dd><p>This produces an extra <em>csv</em> file <code class="docutils literal notranslate"><span class="pre">mars_requests.csv</span></code> where the content of each mars request of the job is stored. Useful for debugging and documentation.</p> 531 </dd> 532 <dt>mailfail</dt><dd><p>At default the mail is send to the mail connected with the user account. Add additional email addresses if you want. But as soon as you enter a new mail, the default will be overwritten. If you would like to keep the mail from your user account, please add <code class="docutils literal notranslate"><span class="pre">${USER}</span></code> to the list ( comma seperated ) or mail addresses.</p> 533 </dd> 553 534 </dl> 554 535 </div> 555 536 <div class="section" id="hints-for-definition-of-some-parameter-combinations"> 556 537 <h3>Hints for definition of some parameter combinations<a class="headerlink" href="#hints-for-definition-of-some-parameter-combinations" title="Permalink to this headline">¶</a></h3> 557 <dl class="docutils"> 558 <dt>Field types and times</dt> 559 <dd><p class="first">This combination is very important. It defines the temporal resolution and which field type is extracted per time step. 538 <dl> 539 <dt>Field types and times</dt><dd><p>This combination is very important. It defines the temporal resolution and which field type is extracted per time step. 560 540 The time declaration for analysis (AN) fields uses the times of the specific analysis and (forecast time) steps have to be <code class="docutils literal notranslate"><span class="pre">0</span></code>. The forecast field types (e.g. FC, CF, CV, PF) need to declare a combination of (forescast start) times and the (forecast) steps. Both of them together defines the actual time step. It is important to know the forecast starting times for the dataset to be retrieved, since they are different. In general it is enough to give information for the exact time steps, but it is also possible to have more time step combinations of <code class="docutils literal notranslate"><span class="pre">TYPE</span></code>, <code class="docutils literal notranslate"><span class="pre">TIME</span></code> and <code class="docutils literal notranslate"><span class="pre">STEP</span></code> because the temporal (hourly) resolution with the <code class="docutils literal notranslate"><span class="pre">DTIME</span></code> parameter will select the correct combinations.</p> 561 <div class="literal-block-wrapper last docutils container" id="id6">562 <div class="code-block-caption"><span class="caption-text">Example of a setting for the field types and temporal resolution.</span><a class="headerlink" href="#id 6" title="Permalink to this code">¶</a></div>541 <div class="literal-block-wrapper docutils container" id="id5"> 542 <div class="code-block-caption"><span class="caption-text">Example of a setting for the field types and temporal resolution.</span><a class="headerlink" href="#id5" title="Permalink to this code">¶</a></div> 563 543 <div class="highlight-bash notranslate"><div class="highlight"><pre><span></span> DTIME <span class="m">3</span> 564 544 TYPE AN FC FC FC AN FC FC FC … … 569 549 </div> 570 550 </dd> 571 <dt>Vertical velocity</dt> 572 <dd><p class="first">The vertical velocity for <code class="docutils literal notranslate"><span class="pre">FLEXPART</span></code> is not directly available from MARS. Therefore it has to be calculated. There are a couple of different options. The following parameters are responsible for the selection. See <a class="reference internal" href="Documentation/vertco.html"><span class="doc">Vertical Coordinate</span></a> for a detailed explanation. The <code class="docutils literal notranslate"><span class="pre">ETADIFF</span></code>, <code class="docutils literal notranslate"><span class="pre">OMEGA</span></code> and <code class="docutils literal notranslate"><span class="pre">OMEGADIFF</span></code> versions are only recommended for debugging and testing reasons. Usually it is a decision between <code class="docutils literal notranslate"><span class="pre">GAUSS</span></code> and <code class="docutils literal notranslate"><span class="pre">ETA</span></code>, where for <code class="docutils literal notranslate"><span class="pre">GAUSS</span></code> spectral fields of the horizontal wind fields and the divergence are to be retrieved and used with the continuity equation to calculate the vertical velocity. For <code class="docutils literal notranslate"><span class="pre">ETA</span></code> the latitude/longitude fields of horizontal wind fields and eta-coordinate are to be retrieved. It is recommended to use <code class="docutils literal notranslate"><span class="pre">ETA</span></code> where possible due to a reduced computation time.</p> 573 <div class="literal-block-wrapper last docutils container" id="id7"> 574 <div class="code-block-caption"><span class="caption-text">Example setting for the vertical coordinate retrieval.</span><a class="headerlink" href="#id7" title="Permalink to this code">¶</a></div> 551 <dt>Vertical velocity</dt><dd><p>The vertical velocity for <code class="docutils literal notranslate"><span class="pre">FLEXPART</span></code> is not directly available from MARS. Therefore it has to be calculated. There are a couple of different options. The following parameters are responsible for the selection. See <a class="reference internal" href="Documentation/vertco.html"><span class="doc">Vertical Coordinate</span></a> for a detailed explanation. The <code class="docutils literal notranslate"><span class="pre">ETADIFF</span></code>, <code class="docutils literal notranslate"><span class="pre">OMEGA</span></code> and <code class="docutils literal notranslate"><span class="pre">OMEGADIFF</span></code> versions are only recommended for debugging and testing reasons. Usually it is a decision between <code class="docutils literal notranslate"><span class="pre">GAUSS</span></code> and <code class="docutils literal notranslate"><span class="pre">ETA</span></code>, where for <code class="docutils literal notranslate"><span class="pre">GAUSS</span></code> spectral fields of the horizontal wind fields and the divergence are to be retrieved and used with the continuity equation to calculate the vertical velocity. For <code class="docutils literal notranslate"><span class="pre">ETA</span></code> the latitude/longitude fields of horizontal wind fields and eta-coordinate are to be retrieved. It is recommended to use <code class="docutils literal notranslate"><span class="pre">ETA</span></code> where possible due to a reduced computation time.</p> 552 <div class="literal-block-wrapper docutils container" id="id6"> 553 <div class="code-block-caption"><span class="caption-text">Example setting for the vertical coordinate retrieval.</span><a class="headerlink" href="#id6" title="Permalink to this code">¶</a></div> 575 554 <div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>GAUSS <span class="m">0</span> 576 555 ETA <span class="m">1</span> … … 583 562 </div> 584 563 </dd> 585 <dt>Grid resolution and domain</dt> 586 <dd><p class="first">The grid and domain selection depends on each other. The grid can be defined in the format of normal degrees (e.g. <code class="docutils literal notranslate"><span class="pre">1.</span></code>) or as in older versions by 1/1000. degrees (e.g. <code class="docutils literal notranslate"><span class="pre">1000</span></code> for <code class="docutils literal notranslate"><span class="pre">1°</span></code>). 564 <dt>Grid resolution and domain</dt><dd><p>The grid and domain selection depends on each other. The grid can be defined in the format of normal degrees (e.g. <code class="docutils literal notranslate"><span class="pre">1.</span></code>) or as in older versions by 1/1000. degrees (e.g. <code class="docutils literal notranslate"><span class="pre">1000</span></code> for <code class="docutils literal notranslate"><span class="pre">1°</span></code>). 587 565 After selecting the grid, the domain has to be defined in a way that the length of the domain in longitude or latitude direction must be an exact multiple of the grid. 588 566 The horizontal resolution for spectral fields will be set by the parameter <code class="docutils literal notranslate"><span class="pre">RESOL</span></code>. For information about how to select an appropriate value you can read the explanation of the MARS keyword <a class="reference external" href="https://confluence.ecmwf.int/display/UDOC/Post-processing+keywords#Post-processingkeywords-resol">here</a> and in <a class="reference external" href="https://confluence.ecmwf.int/display/UDOC/Retrieve#Retrieve-Truncationbeforeinterpolation">this table</a>.</p> 589 <div class="literal-block-wrapper last docutils container" id="id8">590 <div class="code-block-caption"><span class="caption-text">Example setting for a northern hemisphere domain with a grid of <code class="docutils literal notranslate"><span class="pre">0.25°</span></code>.</span><a class="headerlink" href="#id 8" title="Permalink to this code">¶</a></div>567 <div class="literal-block-wrapper docutils container" id="id7"> 568 <div class="code-block-caption"><span class="caption-text">Example setting for a northern hemisphere domain with a grid of <code class="docutils literal notranslate"><span class="pre">0.25°</span></code>.</span><a class="headerlink" href="#id7" title="Permalink to this code">¶</a></div> 591 569 <div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>GRID <span class="m">0</span>.25 592 570 RESOL <span class="m">799</span> … … 600 578 </div> 601 579 </dd> 602 <dt>Flux data</dt> 603 <dd><p class="first">The flux fields are accumulated forecast fields all the time. Since some re-analysis dataset nowadays have complete set of analysis fields in their temporal resolution it was important to define a new parameter set to define the flux fields since the information could not be taken from <code class="docutils literal notranslate"><span class="pre">TYPE</span></code>, <code class="docutils literal notranslate"><span class="pre">TIME</span></code> and <code class="docutils literal notranslate"><span class="pre">STEP</span></code> any longer. Select a forecast field type <code class="docutils literal notranslate"><span class="pre">ACCTYPE</span></code>, the forecast starting time <code class="docutils literal notranslate"><span class="pre">ACCTIME</span></code> and the maximum forecast step <code class="docutils literal notranslate"><span class="pre">ACCMAXSTEP</span></code>. The <code class="docutils literal notranslate"><span class="pre">DTIME</span></code> parameter defines the temporal resolution for the whole period.</p> 604 <div class="literal-block-wrapper last docutils container" id="id9"> 605 <div class="code-block-caption"><span class="caption-text">Example setting for the definition of flux fields.</span><a class="headerlink" href="#id9" title="Permalink to this code">¶</a></div> 580 <dt>Flux data</dt><dd><p>The flux fields are accumulated forecast fields all the time. Since some re-analysis dataset nowadays have complete set of analysis fields in their temporal resolution it was important to define a new parameter set to define the flux fields since the information could not be taken from <code class="docutils literal notranslate"><span class="pre">TYPE</span></code>, <code class="docutils literal notranslate"><span class="pre">TIME</span></code> and <code class="docutils literal notranslate"><span class="pre">STEP</span></code> any longer. Select a forecast field type <code class="docutils literal notranslate"><span class="pre">ACCTYPE</span></code>, the forecast starting time <code class="docutils literal notranslate"><span class="pre">ACCTIME</span></code> and the maximum forecast step <code class="docutils literal notranslate"><span class="pre">ACCMAXSTEP</span></code>. The <code class="docutils literal notranslate"><span class="pre">DTIME</span></code> parameter defines the temporal resolution for the whole period.</p> 581 <div class="literal-block-wrapper docutils container" id="id8"> 582 <div class="code-block-caption"><span class="caption-text">Example setting for the definition of flux fields.</span><a class="headerlink" href="#id8" title="Permalink to this code">¶</a></div> 606 583 <div class="highlight-bash notranslate"><div class="highlight"><pre><span></span> DTIME <span class="m">3</span> 607 584 ACCTYPE FC … … 639 616 <div role="contentinfo"> 640 617 <p> 641 © Copyright 20 19, Anne Philipp and Leopold Haimberger618 © Copyright 2020, Anne Philipp and Leopold Haimberger 642 619 643 620 </p>
Note: See TracChangeset
for help on using the changeset viewer.
