flex_extract (was: ECMWFDATA) 7.0 test version

[leohaim]

Dear FLEXPART users,

a new version 7.0 of the ECMWFDATA software (also known as flex_extract, see FpInputMetEcmwf) used for retrieving ECMWF input data for FLEXPART is available for testing. Some preliminary documentation is attached as well.

Features:

1. It uses WebMARS for data retrieval as default, although the conventional retrieval via ecgate or the HPC is still supported.
2. Most parts are now written in python
3. The scripts are prepared to be embedded in the FLEXPART file tree and support features in FLEXPART 10, e.g. retrieval of cloud water content or production of input files on height levels (.fp files)
4. experimental support for retrieval of input files for WRF in addition to those for FLEXPART
5. Support for retrieving long (>24h) range forecasts

This is a test version, so there are likely some bugs or gaps in the documentation. Any feedback is appreciated.

The tarball that includes a Software Installation Protocol and a Software User Tutorial can be downloaded as
​ftp://srvx7.img.univie.ac.at/pub/ECMWFDATA7.0.tar.gz

With best regards,

Leo Haimberger

[pesei]

Many thanks!

I have already adapted FpInputMetEcmwf, and I would propose to name it similar to the last version (flex_extract_ecgate_V6.0.tar.gz) - it is an extraction software, not data.

[ahilboll]

Thanks a lot for this, this gives a nice overview of the whole preprocessing!

I have two questions:

1. It would help readability a lot if you would consistently use 4 spaces for indentation (currently, in some files there seems to be a mixed tab/space indentation). The standard Python install has a script reindent.py which can automatically fix this.

2. If I understand the code in FlexpartTools.py correctly, you are currently downloading the time-invariant fields 129 and 172 again for each timestep, which seems to be a waste of space and bandwidth. Is this intentional (or, maybe Flexpart needs these fields to be there for each timestep)?

[ahilboll]

One further question:

3. From what I can see, it should be possible to just use the prepareFLEXPART.py script to convert ERA-Interim .grb files so that FLEXPART can ingest them, i.e., if I use other tools/ways to download the data. Is this assumption correct?

[pesei]

Replying to ahilboll:

1. It would help readability a lot if you would consistently use 4 spaces for indentation (currently, in some files there seems to be a mixed tab/space indentation). The standard Python install has a script reindent.py which can automatically fix this.

Thanks for the hint! In Debian, reindent.py isn't a part of the standard python package, you need python2.7-examples.

[pesei]

2. If I understand the code in FlexpartTools.py correctly, you are currently downloading the time-invariant fields 129 and 172 again for each timestep, which seems to be a waste of space and bandwidth. Is this intentional (or, maybe Flexpart needs these fields to be there for each timestep)?

These are 2D fields, thus they are not very costly in terms of storage or bandwidth. On the other hand, if they are not present in each of the FLEXPART input files, they would need to be stored in some extra file and some changes are required in FLEXPART. To keep backward compatibility as well, some more changes in the code would be required. Considering our limited capacity, I would not consider this a priority to implement.

[pesei]

Replying to ahilboll:

One further question:

3. From what I can see, it should be possible to just use the prepareFLEXPART.py script to convert ERA-Interim .grb files so that FLEXPART can ingest them, i.e., if I use other tools/ways to download the data. Is this assumption correct?

Yes, it should be possible and an explicit feature for that may be included in a future version.

[pesei]

changes still required

• change some AN to FC in CONTROL files and elsewhere
• clean up the tarball from unnecessary content, etc.

In a future version, enable using forecasts from 06 and 18 intermediate analyses (this is recommended to users)

[ahilboll]

ERA5

The forecasts starting at 06 and 18 seem to be mandatory for downloading ERA5 data. At least the download doesn't work for me using this CONTROL file:

DAY1
DAY2
DTIME 1
TYPE AN FC FC FC FC FC FC FC FC FC FC FC AN FC FC FC FC FC FC FC FC FC FC FC
TIME 00 18 18 18 18 18 18 18 18 18 18 18 12 06 06 06 06 06 06 06 06 06 06 06
STEP 00 07 08 09 10 11 12 13 14 15 16 17 00 07 08 09 10 11 12 13 14 15 16 17
CLASS EA
STREAM OPER
NUMBER OFF
EXPVER 1
GRID 300
LEFT -180000
LOWER -90000
UPPER 90000
RIGHT 179700
LEVEL 137
LEVELIST 40/to/137
RESOL 799
GAUSS 0
ACCURACY 16
OMEGA 0
OMEGADIFF 0
ETA 0
ETADIFF 0
DPDETA 1
SMOOTH 0
FORMAT GRIB1
DATASET ERA5
ADDPAR 27/28/173/186/187/188/235/139/39
PREFIX EA
ECSTORAGE 1
ECTRANS 0
ECFSDIR ectmp:/${USER}/econdemand/
MAILFAIL ${USER} 
MAILOPS ${USER}
GRIB2FLEXPART 0
EOF

The problem is that request which gets submitted to MARS,

RETRIEVE,
    DATASET    = era5,
    CLASS      = EA,
    TYPE       = FC,
    STREAM     = OPER,
    EXPVER     = 0001,
    REPRES     = SH,
    LEVTYPE    = SFC,
    PARAM      = 142/143/146/180/181/176,
    TIME       = 0000/1200,
    STEP       = 1/2/3/4/5/6/7/8/9/10/11/12,
    DOMAIN     = G,
    RESOL      = 799,
    ACCURACY   = 16,
    AREA       = 90.0/-180.0/-90.0/179.7,
    GRID       = 0.3/0.3,
    PADDING    = 0,
    EXPECT     = ANY,
    DATE       = 20160630

Raises the error

mars - INFO   - 20180124.103559 - Calling mars on 'marser', callback on 60228
mars - INFO   - 20180124.103559 - Server task is 621 [marser]
mars - INFO   - 20180124.103559 - Request cost: 0 field,  [marser]
mars - WARN   - 20180124.103559 - Data not found [marser]
mars - WARN   - 20180124.103559 - No data, but EXPECT was provided
mars - INFO   - 20180124.103559 - Memory used: 22.79 Mbyte(s)
mars - INFO   - 20180124.103559 - No errors reported
Process '['nice', 'mars', '/tmp/tmp-_marsn8HzbO.req']' finished
2018-01-24 11:36:07 Request is complete
2018-01-24 11:36:07 Transfering 0 byte into /mnt/beegfs/met/ecmwf/era5/TEST-flexpart/FCOG_acc_SL.20160630.19502.20365.grb
2018-01-24 11:36:07 From https://stream.ecmwf.int/data/atls02/data/data01/scratch/_mars-atls02-95e2cf679cd58ee9b4db4dd119a05a8d-WH2SsL.grib
2018-01-24 11:36:07 Transfer rate 0 byte/s
2018-01-24 11:36:07 Done.
MARS Request returned no data - please check request
MARS request failed

[pesei]

Replying to ahilboll:

TIME = 0000/1200,

yes, this does not work for ERA5. And the v7 code of the extraction software is not in final shape. As you discovered, analysis times are still hardcoded, e.g.

getMARSdata.py:    if c.basetime=='00' or c.basetime=='12':

A revision of this v7 is under way, but it won't be ready very soon. If you urgently need to use ERA5 data, it would probably be better if you just hack the code.

[ahilboll]

In prepareFLEXPART.py, around ~ line 1237, it should read

    fnout=c.outputdir+'/OMEGA'+suffix

instead of

    fnout=c.outputdir+'/OMEGA'

otherwise there will be only one OMEGA file (which contains only the last timestep which was retrieved).

One question: Is the OMEGA actually needed by FLEXPART, or is it only there for potential validation / interpretation of the model results?

[ahilboll]

Also, I noticed that when downloading data with GAUSS=1 and ETA=0 (which is, if I understand the comments in flexpartTools.py correctly, the recommended setting for ERA-Interim), the U and V parameters are downloaded in SH grid, not on GG - which means that it probably cannot be ingested by WRF's ungrib.exe. Or am I missing something here?

Furthermore, D is contained in the WRF files twice, once in GG and once in SH. That's because it explicitly gets added to the GG list in the

                if c.wrf=='1':                                                  

block even though it's already part of SH.

So maybe, the way would be to do

                if c.wrf=='1':
                    self.params['OG__ML'][0]+='/Z/VO'
                    for var_ in ['/D', '/U', '/V']:
                        if var_ not in self.params['OG__ML'][0]:
                            self.params['OG__ML'][0]+=var_

to add the GG U and V fields to the download list, and then (~line 1188)

                    if c.wrf=='1' and gridtype=='regular_ll':                   
                        if levtype=='hybrid':                                   
                            if paramId in [129,130,131,132,133,138,155]:        
                                grib_write(gid,fwrf)                            
                        else:                                                   
                            if paramId in wrfpars:                              
                                grib_write(gid,fwrf)                            

[pesei]

I am changing the milestone from FLEXPART to flex_extract (only v7.1 available - maybe ask for 7.3?)

[anphi]

This version is no longer supported!
The issue with WRF data as mentioned in comment 13 is outsourced to ticket #226

Therefore I will close the ticket. Please use version 7.0.4.