Opened 3 years ago
Closed 3 years ago
#285 closed Support (fixed)
MARS request for ERA5 surface flux data returns "No data is available" error
Reported by: | fcheng | Owned by: | anphi |
---|---|---|---|
Priority: | major | Milestone: | flex_extract_v7.1.3 |
Component: | flex_extract | Version: | flex_extract_v7.1 |
Keywords: | Cc: |
Description
Dear FLEXPART community,
Recently I use flex_extract (v7.1.2) to download ERA5 data. I found that for some dates the request for ERA5 surface flux data will encounter an error as below:
2020-11-23 11:31:16,800 INFO Request is failed 2020-11-23 11:31:16,801 ERROR Message: no data is available within your requested subset 2020-11-23 11:31:16,801 ERROR Reason: Request returned no data
Below is the mars request (generated by flex_extract) which triggers the error.
marsclass: EA dataset: ERA5 type: AN levtype: SFC levelist: 1 repres: date: 19951101 resol: 159 stream: OPER area: 90.0/-179.0/-90.0/180.0 time: 00 step: 000 expver: 1 number: OFF accuracy: 24 grid: 1.0/1.0 gaussian: target: /disk/r059/ldaiad/flex_extract_farmers/farmer_511/Run/./Workspace/EA_temp/OG_OROLSM__SL.19951101.192867.192868.grb param: 160.128/027.128/028.128/244.128
ERA5 data on some other dates can be download via the same script without any problem, but there are some *dates/periods* that the same download script gives the above error.
A minimal download script to reproduce this error can be as follows.
import cdsapi c = cdsapi.Client() c.retrieve('reanalysis-era5-single-levels', { "product_type":'reanalysis', "area": [40.0,-10.0,30.0,10.0], "year": "1995", "month": "11", "day": ['1'], "grid": [1.,1.], "param": "160.128/027.128/028.128/244.128", #"variable": ['standard_deviation_of_orography', 'forecast_surface_roughness'], "time": ['00'], "format":'grib2', }, 'download_era5_flux_cdsapi.grib')
Note that if using the variable names instead of param (i.e. use 'standard_deviation_of_orography' and 'forecast_surface_roughness' in the commented line), I can download the data without any problem. (but somehow I can't find the variables corresponding to parameters 027.128 and 028.128.)
Hence, I think this may not be a problem for flex_extract. Rather, I guess there are *some* ERA5 surface flux data *missing* in the MARS database.
So my question is how I can work around this error. Can I modify the source code of flex_extract so that I can just download the ERA5 surface flux data from CDS instead of MARS? If so, how can it be done?
Thank you for your time in reading this post. Any advice is much appreciated!
Best,
Fandy
Change History (7)
comment:1 Changed 3 years ago by anphi
- Owner set to anphi
- Status changed from new to accepted
comment:2 Changed 3 years ago by anphi
- Component changed from FP input data to flex_extract
comment:3 Changed 3 years ago by anphi
comment:4 Changed 3 years ago by anphi
Dear Fandy,
I just tested around a little bit with the minimal download script. In my opinion, this has to be an issue with CS3.
The one script you provided is working if you just modify it a little bit:
import cdsapi c = cdsapi.Client() c.retrieve('reanalysis-era5-single-levels', { "product_type":'reanalysis', "area": [40.0,-10.0,30.0,10.0], "year": "1995", "month": "11", "day": ['1'], "grid": [1.,1.], "param": "160.128/027.128/028/244.128", "time": ['00'], "format":'grib2', }, 'download_era5_flux_cdsapi.grib')
So it seems that there is a problem with explicitly selecting the variables from its corresponding table 128.
For now, I don't think that there should be a change in flex_extract in general. I should rather contact CS3 support and ask them whats going on. (I will do that and report back).
Until then, I have a suggestion to help you out with a quick solution to get your data:
- go to the Source/Python/Mods directory and open file tools.py.
- navigate to function to_param_id_with_tablenumber, should start in line 556
- in there, comment out the following block where the table number 128 is added:
if par == k or par == v: spar.append(k + '.128') break
Then you could run flex_extract again and hopefully all data can be found.
Please note that I did not fully test this, as the retrieval takes a while to finish''
I also want to point out that there was a general bug found for the local public ERA5 retrieval. See ticket #286. A bugfix is in preparation.
comment:5 Changed 3 years ago by fcheng
Dear Anne,
Thanks a lot for your explanation and help! However, when I commented out the code block in tools.py
# if par == k or par == v: # spar.append(k + '.128') # break
I receive the following errors when retrieving ERA5
Warning: par LSP not found in table 128 Warning: par CP not found in table 128 Warning: par SSHF not found in table 128 Warning: par EWSS not found in table 128 Warning: par NSSS not found in table 128 Warning: par SSR not found in table 128 marsclass: EA dataset: ERA5 type: FC levtype: SFC levelist: 1 repres: date: 20180930/to/20181002 resol: 159 stream: OPER area: 90.0/-179.0/-90.0/180.0 time: 06/18 step: 1/to/12/by/1 expver: 1 number: OFF accuracy: 24 grid: 1.0/1.0 gaussian: target: /disk/r034/ldaiad/flex_extract_farmers/farmer_010/Run/./Workspace/EA_temp/FCOG_acc_SL.20180930.183456.183457.grb param: target: /disk/r034/ldaiad/flex_extract_farmers/farmer_010/Run/./Workspace/EA_temp/FCOG_acc_SL.20180930.183456.183457.grb MARS Request failed! 'param' Traceback (most recent call last): File "/disk/r034/ldaiad/flex_extract_farmers/farmer_010/Source/Python/Classes/MarsRetrieval.py", line 575, in data_retrieve attrs = self._convert_to_cdsera5_sfc_request(attrs) File "/disk/r034/ldaiad/flex_extract_farmers/farmer_010/Source/Python/Classes/MarsRetrieval.py", line 512, in _convert_to_cdsera5_sfc_request newattrs['param'] = attrs['param'].split('/') KeyError: 'param'
But if I modify the code line like the one below, the bug is gone.
if par == k or par == v: spar.append(k + '') break
Now the downloading process works pretty fine (at least I can retrieve those surface variables now). So I think this is a good workaround, although I notice the downloading speed seems a bit slower than before.
Also, thanks for reporting another bug. I really appreciate your effort in developing flex_extract!
Cheers,
Fandy
comment:6 Changed 3 years ago by anphi
The bugfix for ticket #286 is now in the dev branch. Please update your version to be able to retrieve correct ERA5 data ( it only affects ERA5 data retrieval with the local application mode ).
comment:7 Changed 3 years ago by anphi
- Milestone set to flex_extract_v7.1.3
- Resolution set to fixed
- Status changed from accepted to closed
Dear Fandy,
thank you for reporting the problem.
I will investigate this problem and will provide a fix!
Best regards,
Anne