Opened 3 years ago

Closed 2 months ago

#130 closed New feature (fixed)

flex_extract (was: ECMWFDATA) 7.0 test version

Reported by: leohaim Owned by: leohaim
Priority: major Milestone:
Component: flex_extract Version: flex_extract_v7.0.2
Keywords: Cc:

Description (last modified by pesei)

Dear FLEXPART users,

a new version 7.0 of the ECMWFDATA software (also known as flex_extract, see FpInputMetEcmwf) used for retrieving ECMWF input data for FLEXPART is available for testing. Some preliminary documentation is attached as well.

Features:

  1. It uses WebMARS for data retrieval as default, although the conventional retrieval via ecgate or the HPC is still supported.
  2. Most parts are now written in python
  3. The scripts are prepared to be embedded in the FLEXPART file tree and support features in FLEXPART 10, e.g. retrieval of cloud water content or production of input files on height levels (.fp files)
  4. experimental support for retrieval of input files for WRF in addition to those for FLEXPART
  5. Support for retrieving long (>24h) range forecasts

This is a test version, so there are likely some bugs or gaps in the documentation. Any feedback is appreciated.

The tarball that includes a Software Installation Protocol and a Software User Tutorial can be downloaded as
ftp://srvx7.img.univie.ac.at/pub/ECMWFDATA7.0.tar.gz

With best regards,

Leo Haimberger

Change History (19)

comment:1 Changed 3 years ago by leohaim

  • Owner changed from Leo Haimberger to leohaim
  • Status changed from new to accepted

comment:2 Changed 3 years ago by pesei

Many thanks!

I have already adapted FpInputMetEcmwf, and I would propose to name it similar to the last version (flex_extract_ecgate_V6.0.tar.gz) - it is an extraction software, not data.

comment:3 Changed 3 years ago by pesei

  • Milestone set to FLEXPART 10
  • Version FLEXPART 9.0.2 deleted

comment:4 follow-ups: Changed 2 years ago by ahilboll

Thanks a lot for this, this gives a nice overview of the whole preprocessing!

I have two questions:

  1. It would help readability a lot if you would consistently use 4 spaces for indentation (currently, in some files there seems to be a mixed tab/space indentation). The standard Python install has a script reindent.py which can automatically fix this.
  1. If I understand the code in FlexpartTools.py correctly, you are currently downloading the time-invariant fields 129 and 172 again for each timestep, which seems to be a waste of space and bandwidth. Is this intentional (or, maybe Flexpart needs these fields to be there for each timestep)?

comment:5 follow-up: Changed 2 years ago by ahilboll

One further question:

  1. From what I can see, it should be possible to just use the prepareFLEXPART.py script to convert ERA-Interim .grb files so that FLEXPART can ingest them, i.e., if I use other tools/ways to download the data. Is this assumption correct?

comment:6 in reply to: ↑ 4 Changed 2 years ago by pesei

Replying to ahilboll:

  1. It would help readability a lot if you would consistently use 4 spaces for indentation (currently, in some files there seems to be a mixed tab/space indentation). The standard Python install has a script reindent.py which can automatically fix this.


Thanks for the hint! In Debian, reindent.py isn't a part of the standard python package, you need python2.7-examples.

comment:7 in reply to: ↑ 4 Changed 23 months ago by pesei

  1. If I understand the code in FlexpartTools.py correctly, you are currently downloading the time-invariant fields 129 and 172 again for each timestep, which seems to be a waste of space and bandwidth. Is this intentional (or, maybe Flexpart needs these fields to be there for each timestep)?

These are 2D fields, thus they are not very costly in terms of storage or bandwidth. On the other hand, if they are not present in each of the FLEXPART input files, they would need to be stored in some extra file and some changes are required in FLEXPART. To keep backward compatibility as well, some more changes in the code would be required. Considering our limited capacity, I would not consider this a priority to implement.

comment:8 in reply to: ↑ 5 Changed 23 months ago by pesei

Replying to ahilboll:

One further question:

  1. From what I can see, it should be possible to just use the prepareFLEXPART.py script to convert ERA-Interim .grb files so that FLEXPART can ingest them, i.e., if I use other tools/ways to download the data. Is this assumption correct?

Yes, it should be possible and an explicit feature for that may be included in a future version.

comment:9 Changed 23 months ago by pesei

changes still required

  • change some AN to FC in CONTROL files and elsewhere
  • clean up the tarball from unnecessary content, etc.

In a future version, enable using forecasts from 06 and 18 intermediate analyses (this is recommended to users)

comment:10 follow-up: Changed 14 months ago by ahilboll

ERA5

The forecasts starting at 06 and 18 seem to be mandatory for downloading ERA5 data. At least the download doesn't work for me using this CONTROL file:

DAY1
DAY2
DTIME 1
TYPE AN FC FC FC FC FC FC FC FC FC FC FC AN FC FC FC FC FC FC FC FC FC FC FC
TIME 00 18 18 18 18 18 18 18 18 18 18 18 12 06 06 06 06 06 06 06 06 06 06 06
STEP 00 07 08 09 10 11 12 13 14 15 16 17 00 07 08 09 10 11 12 13 14 15 16 17
CLASS EA
STREAM OPER
NUMBER OFF
EXPVER 1
GRID 300
LEFT -180000
LOWER -90000
UPPER 90000
RIGHT 179700
LEVEL 137
LEVELIST 40/to/137
RESOL 799
GAUSS 0
ACCURACY 16
OMEGA 0
OMEGADIFF 0
ETA 0
ETADIFF 0
DPDETA 1
SMOOTH 0
FORMAT GRIB1
DATASET ERA5
ADDPAR 27/28/173/186/187/188/235/139/39
PREFIX EA
ECSTORAGE 1
ECTRANS 0
ECFSDIR ectmp:/${USER}/econdemand/
MAILFAIL ${USER} 
MAILOPS ${USER}
GRIB2FLEXPART 0
EOF

The problem is that request which gets submitted to MARS,

RETRIEVE,
    DATASET    = era5,
    CLASS      = EA,
    TYPE       = FC,
    STREAM     = OPER,
    EXPVER     = 0001,
    REPRES     = SH,
    LEVTYPE    = SFC,
    PARAM      = 142/143/146/180/181/176,
    TIME       = 0000/1200,
    STEP       = 1/2/3/4/5/6/7/8/9/10/11/12,
    DOMAIN     = G,
    RESOL      = 799,
    ACCURACY   = 16,
    AREA       = 90.0/-180.0/-90.0/179.7,
    GRID       = 0.3/0.3,
    PADDING    = 0,
    EXPECT     = ANY,
    DATE       = 20160630

Raises the error

mars - INFO   - 20180124.103559 - Calling mars on 'marser', callback on 60228
mars - INFO   - 20180124.103559 - Server task is 621 [marser]
mars - INFO   - 20180124.103559 - Request cost: 0 field,  [marser]
mars - WARN   - 20180124.103559 - Data not found [marser]
mars - WARN   - 20180124.103559 - No data, but EXPECT was provided
mars - INFO   - 20180124.103559 - Memory used: 22.79 Mbyte(s)
mars - INFO   - 20180124.103559 - No errors reported
Process '['nice', 'mars', '/tmp/tmp-_marsn8HzbO.req']' finished
2018-01-24 11:36:07 Request is complete
2018-01-24 11:36:07 Transfering 0 byte into /mnt/beegfs/met/ecmwf/era5/TEST-flexpart/FCOG_acc_SL.20160630.19502.20365.grb
2018-01-24 11:36:07 From https://stream.ecmwf.int/data/atls02/data/data01/scratch/_mars-atls02-95e2cf679cd58ee9b4db4dd119a05a8d-WH2SsL.grib
2018-01-24 11:36:07 Transfer rate 0 byte/s
2018-01-24 11:36:07 Done.
MARS Request returned no data - please check request
MARS request failed
Last edited 14 months ago by pesei (previous) (diff)

comment:11 in reply to: ↑ 10 Changed 14 months ago by pesei

Replying to ahilboll:

TIME = 0000/1200,

yes, this does not work for ERA5. And the v7 code of the extraction software is not in final shape. As you discovered, analysis times are still hardcoded, e.g.

getMARSdata.py:    if c.basetime=='00' or c.basetime=='12':

A revision of this v7 is under way, but it won't be ready very soon. If you urgently need to use ERA5 data, it would probably be better if you just hack the code.

comment:12 Changed 12 months ago by ahilboll

In flexpartTools.py, around ~ line 1237, it should read

    fnout=c.outputdir+'/OMEGA'+suffix

instead of

    fnout=c.outputdir+'/OMEGA'

otherwise there will be only one OMEGA file (which contains only the last timestep which was retrieved).

One question: Is the OMEGA actually needed by FLEXPART, or is it only there for potential validation / interpretation of the model results?

Last edited 12 months ago by ahilboll (previous) (diff)

comment:13 Changed 12 months ago by ahilboll

Also, I noticed that when downloading data with GAUSS=1 and ETA=0 (which is, if I understand the comments in flexpartTools.py correctly, the recommended setting for ERA-Interim), the U and V parameters are downloaded in SH grid, not on GG - which means that it probably cannot be ingested by WRF's ungrib.exe. Or am I missing something here?

Furthermore, D is contained in the WRF files twice, once in GG and once in SH. That's because it explicitly gets added to the GG list in the

                if c.wrf=='1':                                                  

block even though it's already part of SH.

So maybe, the way would be to do

                if c.wrf=='1':
                    self.params['OG__ML'][0]+='/Z/VO'
                    for var_ in ['/D', '/U', '/V']:
                        if var_ not in self.params['OG__ML'][0]:
                            self.params['OG__ML'][0]+=var_

to add the GG U and V fields to the download list, and then (~line 1188)

                    if c.wrf=='1' and gridtype=='regular_ll':                   
                        if levtype=='hybrid':                                   
                            if paramId in [129,130,131,132,133,138,155]:        
                                grib_write(gid,fwrf)                            
                        else:                                                   
                            if paramId in wrfpars:                              
                                grib_write(gid,fwrf)                            

comment:14 Changed 12 months ago by pesei

  • Description modified (diff)

comment:15 Changed 3 months ago by pesei

  • Milestone changed from FLEXPART 10 to flex_extract_v7.1
  • Summary changed from ECMWFDATA 7.0 test version to flex_extract (was: ECMWFDATA) 7.0 test version

I am changing the milestone from FLEXPART to flex_extract (only v7.1 available - maybe ask for 7.3?)

comment:16 Changed 2 months ago by pesei

  • Component changed from FP input data to flex_extract
  • Version set to flex_extract_v7.0.2

comment:17 Changed 2 months ago by anphi

  • Milestone flex_extract_v7.1 deleted

comment:18 Changed 2 months ago by anphi

This version is no longer supported!
The issue with WRF data as mentioned in comment 13 is outsourced to ticket #226

Therefore I will close the ticket. Please use version 7.0.4.

comment:19 Changed 2 months ago by anphi

  • Resolution set to fixed
  • Status changed from accepted to closed
Note: See TracTickets for help on using tickets.
hosted by ZAMG