MARS request for ERA5 surface flux data returns "No data is available" error

[fcheng]

Dear FLEXPART community,

Recently I use flex_extract (v7.1.2) to download ERA5 data. I found that for some dates the request for ERA5 surface flux data will encounter an error as below:

2020-11-23 11:31:16,800 INFO Request is failed
2020-11-23 11:31:16,801 ERROR Message: no data is available within your requested subset
2020-11-23 11:31:16,801 ERROR Reason:  Request returned no data

Below is the mars request (generated by flex_extract) which triggers the error.

marsclass: EA
dataset: ERA5
type: AN
levtype: SFC
levelist: 1
repres:
date: 19951101
resol: 159
stream: OPER
area: 90.0/-179.0/-90.0/180.0
time: 00
step: 000
expver: 1
number: OFF
accuracy: 24
grid: 1.0/1.0
gaussian:
target: /disk/r059/ldaiad/flex_extract_farmers/farmer_511/Run/./Workspace/EA_temp/OG_OROLSM__SL.19951101.192867.192868.grb
param: 160.128/027.128/028.128/244.128

ERA5 data on some other dates can be download via the same script without any problem, but there are some *dates/periods* that the same download script gives the above error.

A minimal download script to reproduce this error can be as follows.

import cdsapi

c = cdsapi.Client()

c.retrieve('reanalysis-era5-single-levels',
{
  "product_type":'reanalysis',
  "area": [40.0,-10.0,30.0,10.0],
  "year": "1995",
  "month": "11",
  "day": ['1'],
  "grid": [1.,1.],
  "param": "160.128/027.128/028.128/244.128",
  #"variable": ['standard_deviation_of_orography', 'forecast_surface_roughness'],
  "time": ['00'],
  "format":'grib2',
}, 'download_era5_flux_cdsapi.grib')

Note that if using the variable names instead of param (i.e. use 'standard_deviation_of_orography' and 'forecast_surface_roughness' in the commented line), I can download the data without any problem. (but somehow I can't find the variables corresponding to parameters 027.128 and 028.128.)

Hence, I think this may not be a problem for flex_extract. Rather, I guess there are *some* ERA5 surface flux data *missing* in the MARS database.

So my question is how I can work around this error. Can I modify the source code of flex_extract so that I can just download the ERA5 surface flux data from CDS instead of MARS? If so, how can it be done?

Thank you for your time in reading this post. Any advice is much appreciated!

Best,
Fandy

[anphi]

Dear Fandy,

thank you for reporting the problem.

I will investigate this problem and will provide a fix!

Best regards,
Anne

[anphi]

Dear Fandy,

I just tested around a little bit with the minimal download script. In my opinion, this has to be an issue with CS3.

The one script you provided is working if you just modify it a little bit:

import cdsapi

c = cdsapi.Client()

c.retrieve('reanalysis-era5-single-levels',
{
  "product_type":'reanalysis',
  "area": [40.0,-10.0,30.0,10.0],
  "year": "1995",
  "month": "11",
  "day": ['1'],
  "grid": [1.,1.],
  "param": "160.128/027.128/028/244.128",
  "time": ['00'],
  "format":'grib2',
}, 'download_era5_flux_cdsapi.grib')

So it seems that there is a problem with explicitly selecting the variables from its corresponding table 128.

For now, I don't think that there should be a change in flex_extract in general. I should rather contact CS3 support and ask them whats going on. (I will do that and report back).

Until then, I have a suggestion to help you out with a quick solution to get your data:

• go to the Source/Python/Mods directory and open file tools.py.
• navigate to function to_param_id_with_tablenumber, should start in line 556
• in there, comment out the following block where the table number 128 is added:

     if par == k or par == v:
         spar.append(k + '.128')
         break
  

Then you could run flex_extract again and hopefully all data can be found.

Please note that I did not fully test this, as the retrieval takes a while to finish''

I also want to point out that there was a general bug found for the local public ERA5 retrieval. See ticket #286. A bugfix is in preparation.

[fcheng]

Dear Anne,

Thanks a lot for your explanation and help! However, when I commented out the code block in tools.py

#   if par == k or par == v:
#       spar.append(k + '.128')
#       break

I receive the following errors when retrieving ERA5

                Warning: par LSP not found in table 128
                Warning: par CP not found in table 128
                Warning: par SSHF not found in table 128
                Warning: par EWSS not found in table 128
                Warning: par NSSS not found in table 128
                Warning: par SSR not found in table 128
marsclass: EA
dataset: ERA5
type: FC
levtype: SFC
levelist: 1
repres:
date: 20180930/to/20181002
resol: 159
stream: OPER
area: 90.0/-179.0/-90.0/180.0
time: 06/18
step: 1/to/12/by/1
expver: 1
number: OFF
accuracy: 24
grid: 1.0/1.0
gaussian:
target: /disk/r034/ldaiad/flex_extract_farmers/farmer_010/Run/./Workspace/EA_temp/FCOG_acc_SL.20180930.183456.183457.grb
param:
target: /disk/r034/ldaiad/flex_extract_farmers/farmer_010/Run/./Workspace/EA_temp/FCOG_acc_SL.20180930.183456.183457.grb


MARS Request failed!
'param'
Traceback (most recent call last):
  File "/disk/r034/ldaiad/flex_extract_farmers/farmer_010/Source/Python/Classes/MarsRetrieval.py", line 575, in data_retrieve
    attrs = self._convert_to_cdsera5_sfc_request(attrs)
  File "/disk/r034/ldaiad/flex_extract_farmers/farmer_010/Source/Python/Classes/MarsRetrieval.py", line 512, in _convert_to_cdsera5_sfc_request
    newattrs['param'] = attrs['param'].split('/')
KeyError: 'param'

But if I modify the code line like the one below, the bug is gone.

if par == k or par == v:
       spar.append(k + '')
       break

Now the downloading process works pretty fine (unless I can retrieve those surface variables now). So I think this is a good workaround, although I notice the downloading speed seems a bit slower than before.

Also, thanks for reporting another bug. I really appreciate your effort in developing flex_extract!

Cheers,
Fandy

[anphi]

The bugfix for ticket #286 is now in the dev branch. Please update your version to be able to retrieve correct ERA5 data ( it only affects ERA5 data retrieval with the local application mode ).