Opened 5 weeks ago

Last modified 5 weeks ago

#298 accepted Support

Running preprocessing with previously loaded grib files

Reported by: tcarion Owned by: anphi
Priority: minor Milestone:
Component: flex_extract Version: flex_extract_v7.1
Keywords: Cc:

Description

Hello,

When I'm running flex_extract, I sometimes get an error at etadot calculation :

Flex_extract error:

Command '['{fe_path}/flex_extract_v7.1.2/Source/Fortran/calc_etadot']' died with <Signals.SIGSEGV: 11>.

I don't know where it comes from, and for debugging purposes, I would like to rerun flex_extract without re-downloading the grib files through mars request, since these files are already available.

Unfortunately, reading the (very good) documentation, I didn't see any way of running only the pre-processing phase of the raw grib files given by mars.

So I wanted to know if this feature could be added (or if it already exists but I'm not aware).

Thank you very much in advance.

Tristan Carion

Attachments (2)

output_log.log (75.9 KB) - added by tcarion 5 weeks ago.
output_log
CONTROL_OD.OPER.FC.eta.highres.app (301 bytes) - added by tcarion 5 weeks ago.
control_file

Download all attachments as: .zip

Change History (12)

comment:1 Changed 5 weeks ago by anphi

  • Owner set to anphi
  • Status changed from new to accepted

comment:2 Changed 5 weeks ago by anphi

  • Type changed from Enhancement to Support

comment:3 Changed 5 weeks ago by anphi

Dear Tristan,

the feature you are looking for is already implemented. I just didn't put the instructions into the documentation yet. I'll have to do this soon.

However, to do the post-processing only, you can use the python script prepare_flexpart.py which is located in {fe_path}/Source/Python/Mods/prepare_flexpart.py.

This was modified so it can be used as a standalone script in addition. It can take the following parameters:

./prepare_flexpart.py --help
usage: prepare_flexpart.py [-h] [--start_date START_DATE]
                           [--end_date END_DATE] [--date_chunk DATE_CHUNK]
                           [--job_chunk JOB_CHUNK] [--controlfile CONTROLFILE]
                           [--basetime BASETIME] [--step STEP]
                           [--levelist LEVELIST] [--area AREA] [--debug DEBUG]
                           [--oper OPER] [--request REQUEST] [--public PUBLIC]
                           [--rrint RRINT] [--inputdir INPUTDIR]
                           [--outputdir OUTPUTDIR] [--ppid PPID]
                           [--job_template JOB_TEMPLATE] [--queue QUEUE]

Retrieve FLEXPART input from ECMWF MARS archive

optional arguments:
  -h, --help            show this help message and exit
  --start_date START_DATE
                        start date YYYYMMDD (default: None)
  --end_date END_DATE   end_date YYYYMMDD (default: None)
  --date_chunk DATE_CHUNK
                        # of days to be retrieved at once (default: None)
  --job_chunk JOB_CHUNK
                        # of days to be retrieved within a single job
                        (default: None)
  --controlfile CONTROLFILE
                        The file with all CONTROL parameters. (default:
                        CONTROL_EA5)
  --basetime BASETIME   base time such as 0 or 12 (for half day retrievals)
                        (default: None)
  --step STEP           Forecast steps such as 00/to/48 (default: None)
  --levelist LEVELIST   Vertical levels to be retrieved, e.g. 30/to/60
                        (default: None)
  --area AREA           area, defined by north/west/south/east (default: None)
  --debug DEBUG         debug mode - temporary files will be conserved
                        (default: None)
  --oper OPER           operational mode - prepares dates from environment
                        variables (default: None)
  --request REQUEST     list all MARS requests in file mars_requests.dat
                        (default: None)
  --public PUBLIC       public mode - retrieves public datasets (default:
                        None)
  --rrint RRINT         Selection of old or new interpolation method for
                        precipitation: 0 - old method 1 - new method
                        (additional subgrid points) (default: None)
  --inputdir INPUTDIR   Path to temporary directory for retrieved grib files
                        and other processing files. (default: None)
  --outputdir OUTPUTDIR
                        Path to final directory where FLEXPART input files
                        will be stored. (default: None)
  --ppid PPID           This is the specify the parent process id of a single
                        flex_extract run to identify the files. It is the
                        second number in the GRIB files. (default: None)
  --job_template JOB_TEMPLATE
                        Job template file. Will be used for submission to the
                        batch system on the ECMWF server after modification.
                        (default: job.temp)
  --queue QUEUE         The name of the ECMWF server name where thejob script
                        is to be submitted (e.g. ecgate | cca | ccb) (default:
                        None)

Use this script within the directory where you have the downloaded data files.
Copy the CONTROL file you used into this directory and then call the script with:

python3 {fe_path_to_source}/prepare_flexpart.py --controlfile={name_of_CONTROLfile} --inputdir=. --outputdir=. --ppid={second_number_in_downloaded_datafiles}

Does it work for you?

Best regards,
Anne

comment:4 Changed 5 weeks ago by tcarion

Dear Anne,

It worked, thank you! I got the same error as before, but at least I know how to run flex_extract with downloaded data files. Thank you very much for your quick support.

Kind regards,

Tristan

comment:5 Changed 5 weeks ago by anphi

You could send me your CONTROL file and the log output. Then, I'll have a quick look to see if I notice the problem right away. Which mode do you use? Local public? Or Remote (Gateway)?

Changed 5 weeks ago by tcarion

output_log

Changed 5 weeks ago by tcarion

control_file

comment:6 Changed 5 weeks ago by tcarion

The files are attached. I'm using the local mode with operational data.

comment:7 Changed 5 weeks ago by pesei

I see that you already use ETA 1, so calc_etadot is not used to calculate the vertical velocity, but it just combines the GRIB files into a single one.
It would be useful to know what still causes the program to crash. Maybe you don't have enough memory? Or you have limits in place that are too low? Does it happen immediately? If it's not memory, please recompile in debug mode and post the output.

Last edited 5 weeks ago by pesei (previous) (diff)

comment:8 Changed 5 weeks ago by anphi

Ok, my best guess after a quick_look is, that these lines in the CONTROL file cause the problem:

TYPE AN FC FC FC FC FC FC FC FC FC FC FC FC FC FC
TIME 00 00 00 00 00 00 00 00 00 00 00 00 12 12 12
STEP 00 01 02 03 04 05 06 07 08 09 10 11 00 01 02

Flex_extract is NOT trained to work with data in between a full day or half a day. Did you do this intentional or is it an accident that you tried to get data from 0am to 2pm?
I also think that this special definition of FC TIME and FC STEP without an AN field in between could cause a problem. I never used it this way.

You should try to use the full 24 hours like this:

TYPE AN FC FC FC FC FC FC FC FC FC FC FC AN FC FC FC FC FC FC FC FC FC FC FC
TIME 00 00 00 00 00 00 00 00 00 00 00 00 12 12 12 12 12 12 12 12 12 12 12 12
STEP 00 01 02 03 04 05 06 07 08 09 10 11 00 01 02 03 04 05 06 07 08 09 10 11

or
for half a day (you need to use the BASETIME parameter then):

TYPE AN FC FC FC FC FC FC FC FC FC FC FC AN FC FC FC FC FC FC FC FC FC FC FC
TIME 00 00 00 00 00 00 00 00 00 00 00 00 12 12 12 12 12 12 12 12 12 12 12 12
STEP 00 01 02 03 04 05 06 07 08 09 10 11 00 01 02 03 04 05 06 07 08 09 10 11
BASETIME 00

Let me know if this works. When this solves the problem, I will need to provide a fix to catch this.

comment:9 Changed 5 weeks ago by pesei

Oh yes, I think you can't use TYPE=FC with STEP=00 - there is no forecast at time zero! It is just strange that the error only happens in calc_etadot and not earlier.

comment:10 Changed 5 weeks ago by tcarion

I tried again with TYPE=AN at STEP=00 and it worked, even with getting data from 0am to 2pm. I didn't know flex_extract wasn't trained with data in the middle of the day, I will adapt the way I use it in the future.

Thank you for your support !

Note: See TracTickets for help on using tickets.
hosted by ZAMG