Opened 3 years ago
Last modified 13 months ago
#298 accepted Support
Running preprocessing with previously loaded grib files
| Reported by: | tcarion | Owned by: | anphi |
|---|---|---|---|
| Priority: | minor | Milestone: | flex_extract_v7.1.3 |
| Component: | flex_extract | Version: | flex_extract_v7.1 |
| Keywords: | Cc: |
Description
Hello,
When I'm running flex_extract, I sometimes get an error at etadot calculation :
Flex_extract error:
Command '['{fe_path}/flex_extract_v7.1.2/Source/Fortran/calc_etadot']' died with <Signals.SIGSEGV: 11>.
I don't know where it comes from, and for debugging purposes, I would like to rerun flex_extract without re-downloading the grib files through mars request, since these files are already available.
Unfortunately, reading the (very good) documentation, I didn't see any way of running only the pre-processing phase of the raw grib files given by mars.
So I wanted to know if this feature could be added (or if it already exists but I'm not aware).
Thank you very much in advance.
Tristan Carion
Attachments (2)
Change History (16)
comment:1 Changed 3 years ago by anphi
- Owner set to anphi
- Status changed from new to accepted
comment:2 Changed 3 years ago by anphi
- Type changed from Enhancement to Support
comment:3 Changed 3 years ago by anphi
comment:4 Changed 3 years ago by tcarion
Dear Anne,
It worked, thank you! I got the same error as before, but at least I know how to run flex_extract with downloaded data files. Thank you very much for your quick support.
Kind regards,
Tristan
comment:5 Changed 3 years ago by anphi
You could send me your CONTROL file and the log output. Then, I'll have a quick look to see if I notice the problem right away. Which mode do you use? Local public? Or Remote (Gateway)?
comment:6 Changed 3 years ago by tcarion
The files are attached. I'm using the local mode with operational data.
comment:7 Changed 3 years ago by pesei
If you use operational data, it is better to use the etadot from MARS rather than calculating it with calc_etadot!
Nevertheless, it would be useful to know what causes the program to crash. Maybe you don't have enough memory? Or you have limits in place that are too low?
comment:8 Changed 3 years ago by anphi
Ok, my best guess after a quick_look is, that these lines in the CONTROL file cause the problem:
TYPE AN FC FC FC FC FC FC FC FC FC FC FC FC FC FC TIME 00 00 00 00 00 00 00 00 00 00 00 00 12 12 12 STEP 00 01 02 03 04 05 06 07 08 09 10 11 00 01 02
Flex_extract is NOT trained to work with data in between a full day or half a day. Did you do this intentional or is it an accident that you tried to get data from 0am to 2pm?
I also think that this special definition of FC TIME and FC STEP without an AN field in between could cause a problem. I never used it this way.
You should try to use the full 24 hours like this:
TYPE AN FC FC FC FC FC FC FC FC FC FC FC AN FC FC FC FC FC FC FC FC FC FC FC TIME 00 00 00 00 00 00 00 00 00 00 00 00 12 12 12 12 12 12 12 12 12 12 12 12 STEP 00 01 02 03 04 05 06 07 08 09 10 11 00 01 02 03 04 05 06 07 08 09 10 11
or
for half a day (you need to use the BASETIME parameter then):
TYPE AN FC FC FC FC FC FC FC FC FC FC FC AN FC FC FC FC FC FC FC FC FC FC FC TIME 00 00 00 00 00 00 00 00 00 00 00 00 12 12 12 12 12 12 12 12 12 12 12 12 STEP 00 01 02 03 04 05 06 07 08 09 10 11 00 01 02 03 04 05 06 07 08 09 10 11 BASETIME 00
Let me know if this works. When this solves the problem, I will need to provide a fix to catch this.
comment:9 Changed 3 years ago by pesei
Oh yes, I think you can't use TYPE=FC with STEP=00 - there is no forecast at time zero! It is just strange that the error only happens in calc_etadot and not earlier.
comment:10 Changed 3 years ago by tcarion
I tried again with TYPE=AN at STEP=00 and it worked, even with getting data from 0am to 2pm. I didn't know flex_extract wasn't trained with data in the middle of the day, I will adapt the way I use it in the future.
Thank you for your support !
comment:11 Changed 3 years ago by anphi
- Milestone set to flex_extract_v7.1.3
comment:12 Changed 2 years ago by pesei
- Resolution set to fixed
- Status changed from accepted to closed
Looks like fixed.
comment:13 Changed 2 years ago by pesei
- Resolution fixed deleted
- Status changed from closed to reopened
Reopen, because it might be useful to catch this condition.
comment:14 Changed 13 months ago by anphi
- Status changed from reopened to accepted
Dear Tristan,
the feature you are looking for is already implemented. I just didn't put the instructions into the documentation yet. I'll have to do this soon.
However, to do the post-processing only, you can use the python script prepare_flexpart.py which is located in {fe_path}/Source/Python/Mods/prepare_flexpart.py.
This was modified so it can be used as a standalone script in addition. It can take the following parameters:
./prepare_flexpart.py --help usage: prepare_flexpart.py [-h] [--start_date START_DATE] [--end_date END_DATE] [--date_chunk DATE_CHUNK] [--job_chunk JOB_CHUNK] [--controlfile CONTROLFILE] [--basetime BASETIME] [--step STEP] [--levelist LEVELIST] [--area AREA] [--debug DEBUG] [--oper OPER] [--request REQUEST] [--public PUBLIC] [--rrint RRINT] [--inputdir INPUTDIR] [--outputdir OUTPUTDIR] [--ppid PPID] [--job_template JOB_TEMPLATE] [--queue QUEUE] Retrieve FLEXPART input from ECMWF MARS archive optional arguments: -h, --help show this help message and exit --start_date START_DATE start date YYYYMMDD (default: None) --end_date END_DATE end_date YYYYMMDD (default: None) --date_chunk DATE_CHUNK # of days to be retrieved at once (default: None) --job_chunk JOB_CHUNK # of days to be retrieved within a single job (default: None) --controlfile CONTROLFILE The file with all CONTROL parameters. (default: CONTROL_EA5) --basetime BASETIME base time such as 0 or 12 (for half day retrievals) (default: None) --step STEP Forecast steps such as 00/to/48 (default: None) --levelist LEVELIST Vertical levels to be retrieved, e.g. 30/to/60 (default: None) --area AREA area, defined by north/west/south/east (default: None) --debug DEBUG debug mode - temporary files will be conserved (default: None) --oper OPER operational mode - prepares dates from environment variables (default: None) --request REQUEST list all MARS requests in file mars_requests.dat (default: None) --public PUBLIC public mode - retrieves public datasets (default: None) --rrint RRINT Selection of old or new interpolation method for precipitation: 0 - old method 1 - new method (additional subgrid points) (default: None) --inputdir INPUTDIR Path to temporary directory for retrieved grib files and other processing files. (default: None) --outputdir OUTPUTDIR Path to final directory where FLEXPART input files will be stored. (default: None) --ppid PPID This is the specify the parent process id of a single flex_extract run to identify the files. It is the second number in the GRIB files. (default: None) --job_template JOB_TEMPLATE Job template file. Will be used for submission to the batch system on the ECMWF server after modification. (default: job.temp) --queue QUEUE The name of the ECMWF server name where thejob script is to be submitted (e.g. ecgate | cca | ccb) (default: None)Use this script within the directory where you have the downloaded data files.
Copy the CONTROL file you used into this directory and then call the script with:
python3 {fe_path_to_source}/prepare_flexpart.py --controlfile={name_of_CONTROLfile} --inputdir=. --outputdir=. --ppid={second_number_in_downloaded_datafiles}Does it work for you?
Best regards,
Anne