Opened 3 years ago

Last modified 14 months ago

#297 accepted Defect

gridcheck_ecmwf.f90: fix of small bug and improvements

Reported by: anphi Owned by: anphi
Priority: critical Milestone: FLEXPART 10
Component: FP coding/compilation Version: FLEXPART 10.4
Keywords: GRIB2 GRIB1 Cc: pesei

Description (last modified by pesei)

  1. check on field dimensions uses undefined variable nx
  1. If ECMWF grib files contain only GRIB2 formatted fields, FLEXPART stops with an error message due to missing parameters from GRIB1 messages.

***ERROR: input file needs to contain GRiB1 formatted messages

Update 2022-02-09: In v10.4 the problem is not that a GRIB1 field is required, but that this messages is emitted if no T2 field is found (which is used to extract certain parameters). Rewriting the error message is probably good enough as a solution.

Change History (22)

comment:1 Changed 3 years ago by anphi

  • Status changed from new to accepted

comment:2 Changed 2 years ago by cluekenwinkels

Hello, I recently started using Flexpart and recieved the same ERROR in my text case. I followed the same steps as described in ticket 303. Is there a solution yet? Could you also give information about the flex_extact version you are using?

comment:3 Changed 2 years ago by pesei

We'll try to offer at least a quick fix soon.

comment:4 Changed 2 years ago by pesei

Closely related: #199 gridcheck: checking array dimensions needs to be improved

comment:5 Changed 2 years ago by anphi

Dear cluekenwinkels,

without any more detail to your specific case I am unable to reproduce the error.
Just from checking the code, I have the suspicion that your EN files are missing the 2m temperature field. I am not going into much detail yet. Could you please provide me with more information?
I take it, you are using Flexpart 10.4? What flex_extract version do you use and do you still have the log file from the extraction of the EN files you are currently using? Please provide it to me (attaching it to this ticket would be good). And could you please send me the grib_ls output from at least one of your EN files please? I need to know what is actually in the files. Maybe you can attach one EN file to this ticket. That would be enough.

Thank you.
Anne

comment:6 Changed 2 years ago by pesei

If your EN file is big, you can upload it here: https://bokubox.boku.ac.at/#f6dfa649844127b8da86fa8856618ce4 (max size 15 GB)
(NB: only I will be able to download)

comment:7 Changed 2 years ago by cluekenwinkels

I followed the documentation for flex_extract 7.1.2, so this should be my version: https://www.flexpart.eu/flex_extract/installation.html#ref-download
The Flexpart version is 10.4.
The error only occurs using ERA5 data as I checked runs with ERA-Interim data which worked fine. The only flex_extract output I have left are the files with names EA########, all other files were deleted by flex_extract. I uploaded one of these files and the error and output files of the corresponding flex_extract run. Where can I find the EN files?
(I tried it with both settings: FORMAT=GRIB1 and FORMAT=GRIB2, and the final error remains the same.)

Last edited 2 years ago by cluekenwinkels (previous) (diff)

comment:8 Changed 2 years ago by pesei

Thank you for the files. the EA-File is what we need (for many years we called the FLEXPART input files EN*, more recentely identiers specific to the data set are used, such as EA for ERA5, just old habit to say EN).

There are two uploads without file name - is this a stopped upload? They are not downloadable, so I deleted them.

comment:9 Changed 2 years ago by cluekenwinkels

Ok perfect. There should be 3 files: the EA*, one .o* and one .e* file (the output and error file respectively). I did not try to upload any other files.

comment:10 Changed 2 years ago by anphi

I need to tell you that the grib file looks completely wrong in the surface section. There are a couple of fields which are too much and others are completely missing. I do not remember seeing this specific combination of surface fields before and I cannot reproduce the full problem right now.

So, generally, there is a problem with flex_extract or your setup but not with FLEXPART.

Please upload the CONTROL file you used for the extraction and maybe you can also upload the run_local.sh file.

And I need to make sure that we are using the same version of flex_extract.
Did you use the tar file from the University of Vienna Phaidra website or did you clone the master branch of the git repository? And can you please confirm that you use 7.1.2 and not 7.1.1?

comment:11 Changed 2 years ago by cluekenwinkels

I used this download link: git clone --single-branch --branch master https://www.flexpart.eu/gitmob/flex_extract
I could not find the version number within the files, but it should yield the most recent one, right?
I cannot upload data anymore to the bokubox link, so here are the first lines of my CONTROL file:

START_DATE 20180809
DTIME 1
TYPE AN AN AN AN AN AN AN AN AN AN AN AN AN AN AN AN AN AN AN AN AN AN AN AN
TIME 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23
STEP 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ACCTYPE FC
ACCTIME 06/18
ACCMAXSTEP 12
CLASS EA
STREAM OPER
GRID 0.28125  
LEFT -24.
LOWER 9.875
UPPER 74.
RIGHT 60.375
LEVELIST 1/to/137
RESOL 799
ETA 1
FORMAT GRIB1
PREFIX EA
CWC 1
RRINT 1
ECTRANS 1

This should be the first lines of the run_local.sh I used (I changed it since then but the parameters should be right):

QUEUE=' '
START_DATE='20170102'
DATE_CHUNK=None
JOB_CHUNK=None
BASETIME=None
STEP=None
LEVELIST=None
AREA=None
INPUTDIR='./Workspace/path_to_output'
OUTPUTDIR=None
PP_ID=None
JOB_TEMPLATE=' '
CONTROLFILE='/work/bb1170/RUN/b381737/software/flexpart_v10.4_3d7eebf/preprocess/flex_extract/Run/Control/CONTROL_EA5_grib1'
DEBUG=0
REQUEST=0
PUBLIC=1
Last edited 2 years ago by pesei (previous) (diff)

comment:12 Changed 2 years ago by anphi

Thank you! I need some time to figure out what the problem is. I am on it. I'll let you know as soon as it's solved and how you should proceed.

And by the way, you used RRINT=1. This is not yet supported in Flexpart10.4. I currently have a beta version under test so you need to either use RRINT=0 or wait for the release of my beta version.

Best,
Anne

comment:13 Changed 2 years ago by pesei

  • Description modified (diff)

comment:14 Changed 2 years ago by anphi

I solved the issue. The new version is currently in the dev branch. With a git pull followed by a git checkout dev. You should get the dev branch and you should be able to restart your extraction. The file MarsRetrieval.py in the Source/Python/Classes directory should have a new timestamp.

Best
Anne

comment:15 Changed 2 years ago by cluekenwinkels

Hi, thanks already! I will try it out!

Edit: It the simulations work now. Thanks alot!

(there are still a few warnings that occur however it seems that the correct output is produced)

 parId        3063 isec1(6)         143
 ***WARNING: undefined GRiB2 message found!           0           1         254           1
 parId           0 isec1(6)          -1
 ***WARNING: undefined GRiB2 message found!         192         128          27           1
 parId          27 isec1(6)          -1
 ***WARNING: undefined GRiB2 message found!         192         128          28           1
 parId          28 isec1(6)          -1
 ***WARNING: undefined GRiB2 message found!         192         128         244           1
 parId         244 isec1(6)          -1
Last edited 2 years ago by cluekenwinkels (previous) (diff)

comment:16 Changed 2 years ago by anphi

Thanks a lot for this feedback. I very much appreciate it.

A couple of these warnings are expected and we are aware that we need to adapt the output messages.
However, two warnings come unexpected to me.
First, the parameter with ID 254 shouldn't be there and I'd like to investigate this. Second, it seems that the convective precipitation is not recognized but Flexpart. I have a feeling this is due to GRIB1 vs GRIB2 differences, but I will check this out and report back later this week.


comment:17 follow-up: Changed 22 months ago by mqzhang

Hi,

I meet the same problem.The FLEXPART stop with error *ERROR: input file needs to contain GRiB1 formattedmessages.I have read the entire ticket, but I still don;t understant how to solve due to the ticket post may be not detailed. So, Could anyone explain how to solve the problem? I am grateful for any help.

(Version 10.4 use CONTROL_EA5 in flex_extract 7.1.2, installed in 05/26/2022 by follow the instruction in https://gmd.copernicus.org/articles/13/5277/2020/#section7)
---Mitch Zhang. (Write in 05/27/2022)

comment:18 in reply to: ↑ 17 ; follow-up: Changed 22 months ago by cluekenwinkels

Hi,
for me it worked to reinstall flexextract. In the instructions you referred to it is said that you should use Git clone --single-branch --branch master https://www.flexpart.eu/gitmob/flex_extract. This only downloads the last official release, which does not contain the fix which was done due to this ticket.
To get the fixed version just use git clone https://www.flexpart.eu/gitmob/flex_extract. To then change to the fixed version go into the downloaded folder and use git checkout dev as described comment 14.
Now you can continue the installation of flexextract as before, now with the new version. If you then use data which was downloaded with this new version in flexpart, it should work.

Replying to mqzhang:

Hi,

I meet the same problem.The FLEXPART stop with error *ERROR: input file needs to contain GRiB1 formattedmessages.I have read the entire ticket, but I still don;t understant how to solve due to the ticket post may be not detailed. So, Could anyone explain how to solve the problem? I am grateful for any help.

(Version 10.4 use CONTROL_EA5 in flex_extract 7.1.2, installed in 05/26/2022 by follow the instruction in https://gmd.copernicus.org/articles/13/5277/2020/#section7)
---Mitch Zhang. (Write in 05/27/2022)

comment:19 in reply to: ↑ 18 Changed 22 months ago by mqzhang

Thanks.It helps me and works.I appreciate it very much.
Replying to cluekenwinkels:

Hi,
for me it worked to reinstall flexextract. In the instructions you referred to it is said that you should use Git clone --single-branch --branch master https://www.flexpart.eu/gitmob/flex_extract. This only downloads the last official release, which does not contain the fix which was done due to this ticket.
To get the fixed version just use git clone https://www.flexpart.eu/gitmob/flex_extract. To then change to the fixed version go into the downloaded folder and use git checkout dev as described comment 14.
Now you can continue the installation of flexextract as before, now with the new version. If you then use data which was downloaded with this new version in flexpart, it should work.

Replying to mqzhang:

Hi,

I meet the same problem.The FLEXPART stop with error *ERROR: input file needs to contain GRiB1 formattedmessages.I have read the entire ticket, but I still don;t understant how to solve due to the ticket post may be not detailed. So, Could anyone explain how to solve the problem? I am grateful for any help.

(Version 10.4 use CONTROL_EA5 in flex_extract 7.1.2, installed in 05/26/2022 by follow the instruction in https://gmd.copernicus.org/articles/13/5277/2020/#section7)
---Mitch Zhang. (Write in 05/27/2022)

comment:20 follow-up: Changed 15 months ago by myoshimura

Dear all,
I'm meeting the same problem and reinstalling flex_extract. I installed the version with git checkout dev and tried to fetch met data with run_local.sh. However, I got the following error message and I couldn't download the data.

MARS request failed

  File "/flex_extract/Run/../Source/Python/submit.py", line 272, in <module>
    main()

  File "/flex_extract/Run/../Source/Python/submit.py", line 108, in main
    get_mars_data(c)

  File "/flex_extract/Source/Python/Mods/get_mars_data.py", line 161, in get_mars_data
    do_retrievement(c, server, start, end, datechunk, fluxes=True)

  File "/flex_extract/Source/Python/Mods/get_mars_data.py", line 393, in do_retrievement
    my_error('MARS request failed')

  File "/flex_extract/Source/Python/Mods/tools.py", line 344, in my_error
    trace = '\n'.join(traceback.format_stack())

Could anyone help me how to resolve this issue?
Thank you

comment:21 in reply to: ↑ 20 ; follow-up: Changed 14 months ago by jpianezze

Hi all,

@myoshimura : I had the same error when I tried to use flex_extract on my laptot to extract ERA5 data via CDSAPI. I fixed it by putting PUBLIC=1 in the script run_local.sh. Hope it will works for you too.

@anphi : Thank you for porting flex_extract to the new CEP servers and to fix some bugs ! It works well for me !

Cheers,
Joris

Replying to myoshimura:

Dear all,
I'm meeting the same problem and reinstalling flex_extract. I installed the version with git checkout dev and tried to fetch met data with run_local.sh. However, I got the following error message and I couldn't download the data.

MARS request failed

  File "/flex_extract/Run/../Source/Python/submit.py", line 272, in <module>
    main()

  File "/flex_extract/Run/../Source/Python/submit.py", line 108, in main
    get_mars_data(c)

  File "/flex_extract/Source/Python/Mods/get_mars_data.py", line 161, in get_mars_data
    do_retrievement(c, server, start, end, datechunk, fluxes=True)

  File "/flex_extract/Source/Python/Mods/get_mars_data.py", line 393, in do_retrievement
    my_error('MARS request failed')

  File "/flex_extract/Source/Python/Mods/tools.py", line 344, in my_error
    trace = '\n'.join(traceback.format_stack())

Could anyone help me how to resolve this issue?
Thank you

Version 1, edited 14 months ago by jpianezze (previous) (next) (diff)

comment:22 in reply to: ↑ 21 Changed 14 months ago by myoshimura

@jpianezze: Thank you so much for your advice! It helps me!

BTW, I sometimes encounter an error reported in #227 (I'm a public user and using local mode) when I change the data domain or start_date. And I sometimes get stuck during data retrieving. Curiously,
I cannot confirm how this happens (the latter issue may be due to my internet connection).

Replying to jpianezze:

Hi all,

@myoshimura : I had the same error when I tried to use flex_extract on my laptot to extract ERA5 data via CDSAPI. I fixed it by putting PUBLIC=1 in the script run_local.sh. Hope it will works for you too.

@anphi : Thank you for porting flex_extract to the new CEP servers and to fix some bugs ! It works well for me !

Cheers,
Joris

Replying to myoshimura:

Dear all,
I'm meeting the same problem and reinstalling flex_extract. I installed the version with git checkout dev and tried to fetch met data with run_local.sh. However, I got the following error message and I couldn't download the data.

MARS request failed

  File "/flex_extract/Run/../Source/Python/submit.py", line 272, in <module>
    main()

  File "/flex_extract/Run/../Source/Python/submit.py", line 108, in main
    get_mars_data(c)

  File "/flex_extract/Source/Python/Mods/get_mars_data.py", line 161, in get_mars_data
    do_retrievement(c, server, start, end, datechunk, fluxes=True)

  File "/flex_extract/Source/Python/Mods/get_mars_data.py", line 393, in do_retrievement
    my_error('MARS request failed')

  File "/flex_extract/Source/Python/Mods/tools.py", line 344, in my_error
    trace = '\n'.join(traceback.format_stack())

Could anyone help me how to resolve this issue?
Thank you

Note: See TracTickets for help on using tickets.
hosted by ZAMG