FLEXPART.EU: Ticket #286: BUG: Problem with local public ERA5 data retrieval in flex_extract v7.1.2
https://www.flexpart.eu/ticket/286
<p>
When working with flex_extract v7.1.2 locally to retrieve ERA5 data from the CS3 website with CDSAPI the requests, a user had problems to use the retrieved files with FLEXPART v10.4.
</p>
<p>
An example result file and CONTROL file are attached.
</p>
en-usFLEXPART.EUhttps://www.flexpart.eu/chrome/site/flexpart_banner.png
https://www.flexpart.eu/ticket/286
Trac 1.0.2anphiSun, 29 Nov 2020 21:26:11 GMTstatus changed
https://www.flexpart.eu/ticket/286#comment:1
https://www.flexpart.eu/ticket/286#comment:1
<ul>
<li><strong>status</strong>
changed from <em>new</em> to <em>accepted</em>
</li>
</ul>
TicketanphiSun, 29 Nov 2020 21:50:11 GMT
https://www.flexpart.eu/ticket/286#comment:2
https://www.flexpart.eu/ticket/286#comment:2
<p>
After a quick inspection of the file content, it was clear that there is a bug in retrieving the surface fields. See below for list of fields contained in the example file:
</p>
<pre class="wiki">1 ecmf 20180701 an regular_ll 0 surface 0 sd grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 msl grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 tcc grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 10u grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 10v grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 2t grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 2d grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 z grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 lsm grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 sdor grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 cvl grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 cvh grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 fsr grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 sdor grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 cvl grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 cvh grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 fsr grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 sdor grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 cvl grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 cvh grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 fsr grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 sdor grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 cvl grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 cvh grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 fsr grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 sdor grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 cvl grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 cvh grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 fsr grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 sdor grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 cvl grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 cvh grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 fsr grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 sdor grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 cvl grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 cvh grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 fsr grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 sdor grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 cvl grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 cvh grid_simple
1 ecmf 20180701 an regular_ll 0 surface 0 fsr grid_simple
</pre><p>
The fields for topography information cvl,cvh,fsr and sdor should appear one time each and not multiple times. And precipitation data is missing, at least for this specific time step.
</p>
<hr />
<p>
The problem occurs because I wanted to speed up the retrieval time for public users who want to retrieve ERA5 data from CS3 website. I, therefore, split the retrieval of surface and model level fields, since model level fields are still not available on CS3 and we need to retrieve them from the original MARS archive. The surface fields can be retrieved relatively fast from CS3, but they have different access keywords and storage structure.
</p>
<p>
There are actually two separate bugs. The first bug is in the conversion routine from MARS keywords to CS3 keywords. Surface fluxes are no longer accessed through the combination of time and step parameters but direct analysis times. The second bug is the accidental fixing of the dtime parameter to 3 in the surface field retrieval. Therefore all surface fields are retrieved every third hour, regardless of what was determined beforehand.
</p>
<p>
A bugfix will be provided as soon as possible.
</p>
TicketanphiSun, 29 Nov 2020 21:50:30 GMTsummary changed
https://www.flexpart.eu/ticket/286#comment:3
https://www.flexpart.eu/ticket/286#comment:3
<ul>
<li><strong>summary</strong>
changed from <em>Problem with local public ERA5 data retrieval in flex_extract v7.1.2</em> to <em>BUG: Problem with local public ERA5 data retrieval in flex_extract v7.1.2</em>
</li>
</ul>
TicketanphiTue, 01 Dec 2020 17:30:11 GMT
https://www.flexpart.eu/ticket/286#comment:4
https://www.flexpart.eu/ticket/286#comment:4
<p>
This bug is fixed in <tt>dev</tt> branch of our git repository now.
</p>
<p>
The exact changes can be viewed <a class="ext-link" href="https://www.flexpart.eu/changeset/e3c679d2518710a8d2b18af648ec468fbdfeb003/flex_extract.git"><span class="icon"></span>here</a>.
</p>
<p>
Until the fix is merged into the master branch, I will keep the ticket open.
</p>
Ticket