Opened 4 years ago
Last modified 20 months ago
#297 accepted Defect
gridcheck_ecmwf.f90: fix of small bug and improvements
Reported by: | anphi | Owned by: | anphi |
---|---|---|---|
Priority: | critical | Milestone: | FLEXPART 10 |
Component: | FP coding/compilation | Version: | FLEXPART 10.4 |
Keywords: | GRIB2 GRIB1 | Cc: | pesei |
Description (last modified by pesei)
- check on field dimensions uses undefined variable nx
- If ECMWF grib files contain only GRIB2 formatted fields, FLEXPART stops with an error message due to missing parameters from GRIB1 messages.
***ERROR: input file needs to contain GRiB1 formatted messages
Update 2022-02-09: In v10.4 the problem is not that a GRIB1 field is required, but that this messages is emitted if no T2 field is found (which is used to extract certain parameters). Rewriting the error message is probably good enough as a solution.
Change History (22)
comment:1 Changed 3 years ago by anphi
- Status changed from new to accepted
comment:2 Changed 3 years ago by cluekenwinkels
comment:3 Changed 3 years ago by pesei
We'll try to offer at least a quick fix soon.
comment:4 Changed 3 years ago by pesei
Closely related: #199 gridcheck: checking array dimensions needs to be improved
comment:5 Changed 3 years ago by anphi
Dear cluekenwinkels,
without any more detail to your specific case I am unable to reproduce the error.
Just from checking the code, I have the suspicion that your EN files are missing the 2m temperature field. I am not going into much detail yet. Could you please provide me with more information?
I take it, you are using Flexpart 10.4? What flex_extract version do you use and do you still have the log file from the extraction of the EN files you are currently using? Please provide it to me (attaching it to this ticket would be good). And could you please send me the grib_ls output from at least one of your EN files please? I need to know what is actually in the files. Maybe you can attach one EN file to this ticket. That would be enough.
Thank you.
Anne
comment:6 Changed 3 years ago by pesei
If your EN file is big, you can upload it here: https://bokubox.boku.ac.at/#f6dfa649844127b8da86fa8856618ce4 (max size 15 GB)
(NB: only I will be able to download)
comment:7 Changed 3 years ago by cluekenwinkels
I followed the documentation for flex_extract 7.1.2, so this should be my version: https://www.flexpart.eu/flex_extract/installation.html#ref-download
The Flexpart version is 10.4.
The error only occurs using ERA5 data as I checked runs with ERA-Interim data which worked fine. The only flex_extract output I have left are the files with names EA########, all other files were deleted by flex_extract. Should I upload one of those? Where can I find the EN files?
(I tried it with both settings: FORMAT=GRIB1 and FORMAT=GRIB2, and the final error remains the same.)
comment:8 Changed 3 years ago by pesei
Thank you for the files. the EA-File is what we need (for many years we called the FLEXPART input files EN*, more recentely identiers specific to the data set are used, such as EA for ERA5, just old habit to say EN).
There are two uploads without file name - is this a stopped upload? They are not downloadable, so I deleted them.
comment:9 Changed 3 years ago by cluekenwinkels
Ok perfect. There should be 3 files: the EA*, one .o* and one .e* file (the output and error file respectively). I did not try to upload any other files.
comment:10 Changed 3 years ago by anphi
I need to tell you that the grib file looks completely wrong in the surface section. There are a couple of fields which are too much and others are completely missing. I do not remember seeing this specific combination of surface fields before and I cannot reproduce the full problem right now.
So, generally, there is a problem with flex_extract or your setup but not with FLEXPART.
Please upload the CONTROL file you used for the extraction and maybe you can also upload the run_local.sh file.
And I need to make sure that we are using the same version of flex_extract.
Did you use the tar file from the University of Vienna Phaidra website or did you clone the master branch of the git repository? And can you please confirm that you use 7.1.2 and not 7.1.1?
comment:11 Changed 3 years ago by cluekenwinkels
I used this download link: git clone --single-branch --branch master https://www.flexpart.eu/gitmob/flex_extract
I could not find the version number within the files, but it should yield the most recent one, right?
I cannot upload data anymore to the bokubox link, so here are the first lines of my CONTROL file:
START_DATE 20180809 DTIME 1 TYPE AN AN AN AN AN AN AN AN AN AN AN AN AN AN AN AN AN AN AN AN AN AN AN AN TIME 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 STEP 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ACCTYPE FC ACCTIME 06/18 ACCMAXSTEP 12 CLASS EA STREAM OPER GRID 0.28125 LEFT -24. LOWER 9.875 UPPER 74. RIGHT 60.375 LEVELIST 1/to/137 RESOL 799 ETA 1 FORMAT GRIB1 PREFIX EA CWC 1 RRINT 1 ECTRANS 1
This should be the first lines of the run_local.sh I used (I changed it since then but the parameters should be right):
QUEUE=' ' START_DATE='20170102' DATE_CHUNK=None JOB_CHUNK=None BASETIME=None STEP=None LEVELIST=None AREA=None INPUTDIR='./Workspace/path_to_output' OUTPUTDIR=None PP_ID=None JOB_TEMPLATE=' ' CONTROLFILE='/work/bb1170/RUN/b381737/software/flexpart_v10.4_3d7eebf/preprocess/flex_extract/Run/Control/CONTROL_EA5_grib1' DEBUG=0 REQUEST=0 PUBLIC=1
comment:12 Changed 3 years ago by anphi
Thank you! I need some time to figure out what the problem is. I am on it. I'll let you know as soon as it's solved and how you should proceed.
And by the way, you used RRINT=1. This is not yet supported in Flexpart10.4. I currently have a beta version under test so you need to either use RRINT=0 or wait for the release of my beta version.
Best,
Anne
comment:13 Changed 3 years ago by pesei
- Description modified (diff)
comment:14 Changed 3 years ago by anphi
I solved the issue. The new version is currently in the dev branch. With a git pull followed by a git checkout dev. You should get the dev branch and you should be able to restart your extraction. The file MarsRetrieval.py in the Source/Python/Classes directory should have a new timestamp.
Best
Anne
comment:15 Changed 3 years ago by cluekenwinkels
Hi, thanks already! I will try it out!
Edit: It the simulations work now. Thanks alot!
(there are still a few warnings that occur however it seems that the correct output is produced)
parId 3063 isec1(6) 143 ***WARNING: undefined GRiB2 message found! 0 1 254 1 parId 0 isec1(6) -1 ***WARNING: undefined GRiB2 message found! 192 128 27 1 parId 27 isec1(6) -1 ***WARNING: undefined GRiB2 message found! 192 128 28 1 parId 28 isec1(6) -1 ***WARNING: undefined GRiB2 message found! 192 128 244 1 parId 244 isec1(6) -1
comment:16 Changed 3 years ago by anphi
Thanks a lot for this feedback. I very much appreciate it.
A couple of these warnings are expected and we are aware that we need to adapt the output messages.
However, two warnings come unexpected to me.
First, the parameter with ID 254 shouldn't be there and I'd like to investigate this. Second, it seems that the convective precipitation is not recognized but Flexpart. I have a feeling this is due to GRIB1 vs GRIB2 differences, but I will check this out and report back later this week.
comment:17 follow-up: ↓ 18 Changed 2 years ago by mqzhang
Hi,
I meet the same problem.The FLEXPART stop with error *ERROR: input file needs to contain GRiB1 formattedmessages.I have read the entire ticket, but I still don;t understant how to solve due to the ticket post may be not detailed. So, Could anyone explain how to solve the problem? I am grateful for any help.
(Version 10.4 use CONTROL_EA5 in flex_extract 7.1.2, installed in 05/26/2022 by follow the instruction in https://gmd.copernicus.org/articles/13/5277/2020/#section7)
---Mitch Zhang. (Write in 05/27/2022)
comment:18 in reply to: ↑ 17 ; follow-up: ↓ 19 Changed 2 years ago by cluekenwinkels
Hi,
for me it worked to reinstall flexextract. In the instructions you referred to it is said that you should use Git clone --single-branch --branch master https://www.flexpart.eu/gitmob/flex_extract. This only downloads the last official release, which does not contain the fix which was done due to this ticket.
To get the fixed version just use git clone https://www.flexpart.eu/gitmob/flex_extract. To then change to the fixed version go into the downloaded folder and use git checkout dev as described comment 14.
Now you can continue the installation of flexextract as before, now with the new version. If you then use data which was downloaded with this new version in flexpart, it should work.
Replying to mqzhang:
Hi,
I meet the same problem.The FLEXPART stop with error *ERROR: input file needs to contain GRiB1 formattedmessages.I have read the entire ticket, but I still don;t understant how to solve due to the ticket post may be not detailed. So, Could anyone explain how to solve the problem? I am grateful for any help.
(Version 10.4 use CONTROL_EA5 in flex_extract 7.1.2, installed in 05/26/2022 by follow the instruction in https://gmd.copernicus.org/articles/13/5277/2020/#section7)
---Mitch Zhang. (Write in 05/27/2022)
comment:19 in reply to: ↑ 18 Changed 2 years ago by mqzhang
Thanks.It helps me and works.I appreciate it very much.
Replying to cluekenwinkels:
Hi,
for me it worked to reinstall flexextract. In the instructions you referred to it is said that you should use Git clone --single-branch --branch master https://www.flexpart.eu/gitmob/flex_extract. This only downloads the last official release, which does not contain the fix which was done due to this ticket.
To get the fixed version just use git clone https://www.flexpart.eu/gitmob/flex_extract. To then change to the fixed version go into the downloaded folder and use git checkout dev as described comment 14.
Now you can continue the installation of flexextract as before, now with the new version. If you then use data which was downloaded with this new version in flexpart, it should work.
Replying to mqzhang:
Hi,
I meet the same problem.The FLEXPART stop with error *ERROR: input file needs to contain GRiB1 formattedmessages.I have read the entire ticket, but I still don;t understant how to solve due to the ticket post may be not detailed. So, Could anyone explain how to solve the problem? I am grateful for any help.
(Version 10.4 use CONTROL_EA5 in flex_extract 7.1.2, installed in 05/26/2022 by follow the instruction in https://gmd.copernicus.org/articles/13/5277/2020/#section7)
---Mitch Zhang. (Write in 05/27/2022)
comment:20 follow-up: ↓ 21 Changed 21 months ago by myoshimura
Dear all,
I'm meeting the same problem and reinstalling flex_extract. I installed the version with git checkout dev and tried to fetch met data with run_local.sh. However, I got the following error message and I couldn't download the data.
MARS request failed File "/flex_extract/Run/../Source/Python/submit.py", line 272, in <module> main() File "/flex_extract/Run/../Source/Python/submit.py", line 108, in main get_mars_data(c) File "/flex_extract/Source/Python/Mods/get_mars_data.py", line 161, in get_mars_data do_retrievement(c, server, start, end, datechunk, fluxes=True) File "/flex_extract/Source/Python/Mods/get_mars_data.py", line 393, in do_retrievement my_error('MARS request failed') File "/flex_extract/Source/Python/Mods/tools.py", line 344, in my_error trace = '\n'.join(traceback.format_stack())
Could anyone help me how to resolve this issue?
Thank you
comment:21 in reply to: ↑ 20 ; follow-up: ↓ 22 Changed 21 months ago by jpianezze
Hi all,
@myoshimura : I had the same error when I tried to use flex_extract on my laptot to extract ERA5 data via CDSAPI. I fixed it by putting PUBLIC=1 in the script run_local.sh. Hope it will works for you too.
@anphi : Thank you for porting flex_extract to the new ECMWF servers and to fix some bugs ! It works well for me !
Cheers,
Joris
Replying to myoshimura:
Dear all,
I'm meeting the same problem and reinstalling flex_extract. I installed the version with git checkout dev and tried to fetch met data with run_local.sh. However, I got the following error message and I couldn't download the data.
MARS request failed File "/flex_extract/Run/../Source/Python/submit.py", line 272, in <module> main() File "/flex_extract/Run/../Source/Python/submit.py", line 108, in main get_mars_data(c) File "/flex_extract/Source/Python/Mods/get_mars_data.py", line 161, in get_mars_data do_retrievement(c, server, start, end, datechunk, fluxes=True) File "/flex_extract/Source/Python/Mods/get_mars_data.py", line 393, in do_retrievement my_error('MARS request failed') File "/flex_extract/Source/Python/Mods/tools.py", line 344, in my_error trace = '\n'.join(traceback.format_stack())Could anyone help me how to resolve this issue?
Thank you
comment:22 in reply to: ↑ 21 Changed 20 months ago by myoshimura
@jpianezze: Thank you so much for your advice! It helps me!
BTW, I sometimes encounter an error reported in #227 (I'm a public user and using local mode) when I change the data domain or start_date. And I sometimes get stuck during data retrieving. Curiously,
I cannot confirm how this happens (the latter issue may be due to my internet connection).
Replying to jpianezze:
Hi all,
@myoshimura : I had the same error when I tried to use flex_extract on my laptot to extract ERA5 data via CDSAPI. I fixed it by putting PUBLIC=1 in the script run_local.sh. Hope it will works for you too.
@anphi : Thank you for porting flex_extract to the new CEP servers and to fix some bugs ! It works well for me !
Cheers,
Joris
Replying to myoshimura:
Dear all,
I'm meeting the same problem and reinstalling flex_extract. I installed the version with git checkout dev and tried to fetch met data with run_local.sh. However, I got the following error message and I couldn't download the data.
MARS request failed File "/flex_extract/Run/../Source/Python/submit.py", line 272, in <module> main() File "/flex_extract/Run/../Source/Python/submit.py", line 108, in main get_mars_data(c) File "/flex_extract/Source/Python/Mods/get_mars_data.py", line 161, in get_mars_data do_retrievement(c, server, start, end, datechunk, fluxes=True) File "/flex_extract/Source/Python/Mods/get_mars_data.py", line 393, in do_retrievement my_error('MARS request failed') File "/flex_extract/Source/Python/Mods/tools.py", line 344, in my_error trace = '\n'.join(traceback.format_stack())Could anyone help me how to resolve this issue?
Thank you
Hello, I recently started using Flexpart and recieved the same ERROR in my text case. I followed the same steps as described in ticket 303. Is there a solution yet? Could you also give information about the flex_extact version you are using?