BUG: Problem with local public ERA5 data retrieval in flex_extract v7.1.2

[anphi]

When working with flex_extract v7.1.2 locally to retrieve ERA5 data from the CS3 website with CDSAPI the requests, a user had problems to use the retrieved files with FLEXPART v10.4.

An example result file and CONTROL file are attached.

[anphi]

After a quick inspection of the file content, it was clear that there is a bug in retrieving the surface fields. See below for list of fields contained in the example file:

1            ecmf         20180701     an           regular_ll   0            surface      0            sd           grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            msl          grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            tcc          grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            10u          grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            10v          grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            2t           grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            2d           grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            z            grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            lsm          grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            sdor         grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            cvl          grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            cvh          grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            fsr          grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            sdor         grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            cvl          grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            cvh          grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            fsr          grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            sdor         grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            cvl          grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            cvh          grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            fsr          grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            sdor         grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            cvl          grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            cvh          grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            fsr          grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            sdor         grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            cvl          grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            cvh          grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            fsr          grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            sdor         grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            cvl          grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            cvh          grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            fsr          grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            sdor         grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            cvl          grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            cvh          grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            fsr          grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            sdor         grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            cvl          grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            cvh          grid_simple 
1            ecmf         20180701     an           regular_ll   0            surface      0            fsr          grid_simple 

The fields for topography information cvl,cvh,fsr and sdor should appear one time each and not multiple times. And precipitation data is missing, at least for this specific time step.

---

The problem occurs because I wanted to speed up the retrieval time for public users who want to retrieve ERA5 data from CS3 website. I, therefore, split the retrieval of surface and model level fields, since model level fields are still not available on CS3 and we need to retrieve them from the original MARS archive. The surface fields can be retrieved relatively fast from CS3, but they have different access keywords and storage structure.

There are actually two separate bugs. The first bug is in the conversion routine from MARS keywords to CS3 keywords. Surface fluxes are no longer accessed through the combination of time and step parameters but direct analysis times. The second bug is the accidental fixing of the dtime parameter to 3 in the surface field retrieval. Therefore all surface fields are retrieved every third hour, regardless of what was determined beforehand.

A bugfix will be provided as soon as possible.

[anphi]

This bug is fixed in dev branch of our git repository now.

The exact changes can be viewed ​here.

Until the fix is merged into the master branch, I will keep the ticket open.