Flex_Extract MARS request failed

[nanditha]

Hi,

Im getting the following error, when I request data using submit.py.

2019-10-28 10:56:25 ECMWF API python library 1.5.4
2019-10-28 10:56:25 ECMWF API at https://api.ecmwf.int/v1
2019-10-28 10:56:27 Welcome nanditha js
Error:  <class 'ecmwfapi.api.APIException'>
MARS Request failed
Traceback (most recent call last):
  File "./submit.py", line 114, in <module>
    main()
  File "./submit.py", line 59, in main
    getMARSdata(args,c)
  File "/home/nanditha/lib/src/flexpart_9.0.3/flex_extract_v7.0.3/python/getMARSdata.py", line 116, in getMARSdata
    flexpart.retrieve(server, c.public, dates, times, c.inputdir)
  File "/home/nanditha/lib/src/flexpart_9.0.3/flex_extract_v7.0.3/python/FlexpartTools.py", line 979, in retrieve
    MR.dataRetrieve()
  File "/home/nanditha/lib/src/flexpart_9.0.3/flex_extract_v7.0.3/python/FlexpartTools.py", line 680, in dataRetrieve
    if not int(self.public) and os.stat(target).st_size==0:
OSError: [Errno 2] No such file or directory: '/home/nanditha/lib/src/flexpart_9.0.3/flex_extract_v7.0.3/python/../work/FCOG_acc_SL.20171104.3218.18276.grb'

I have requested data using the command
./submit.py --start_date 20171105 --controlfile CONTROL_EA5
I,m attaching the control file used for retrieval request

It would be highly appreciated if you could help me in this regard
Thanks in advance

[anphi]

Question answered in Ticket #142

[ebilgic]

I received similar error, too. I installed flex_extract 7.0.4 in ECMWF ssh server and I tried test submission in section 3.3.1. in flex_extract software installation plan:

./submit.py --start_date=20131107

and I received the following error message.

Retrieving EC data!
start date 20131107 
end date 20131107 
ecapi: True
removing content of /home/ms/tr/tm1e/Scripts/flex_extract/python/../work
retrieve 20131106/to/20131108 in dir /home/ms/tr/tm1e/Scripts/flex_extract/python/../work
accuracy: 24
stream: OPER
levelist: 1
area: 90.0/-179.0/-90.0/180.0
marsclass: EA
levtype: SFC
gaussian: 
number: OFF
expver: 1
repres: 
public: 0
step: 1/to/12/by/1
grid: 1.0/1.0
param: LSP/CP/SSHF/EWSS/NSSS/SSR
time: 06/18
date: 20131106/to/20131108
resol: 159
dataset: 
type: FC
target: /home/ms/tr/tm1e/Scripts/flex_extract/python/../work/FCOG_acc_SL.20131106.50805.37373.grb
target:  /home/ms/tr/tm1e/Scripts/flex_extract/python/../work/FCOG_acc_SL.20131106.50805.37373.grb
EXECUTE RETRIEVAL!
2020-02-13 10:24:23 ECMWF API python library 1.5.0
2020-02-13 10:24:23 ECMWF API at https://api.ecmwf.int/v1
Error:  <class 'ecmwfapi.api.APIException'>
MARS Request failed
Traceback (most recent call last):
  File "./submit.py", line 115, in <module>
    main()
  File "./submit.py", line 57, in main
    getMARSdata(args,c)
  File "/home/ms/tr/tm1e/Scripts/flex_extract/python/getMARSdata.py", line 133, in getMARSdata
    flexpart.retrieve(server, c.public, dates, c.request, times, c.inputdir)
  File "/home/ms/tr/tm1e/Scripts/flex_extract/python/FlexpartTools.py", line 1068, in retrieve
    MR.dataRetrieve()
  File "/home/ms/tr/tm1e/Scripts/flex_extract/python/FlexpartTools.py", line 764, in dataRetrieve
    if not int(self.public) and os.stat(target).st_size==0:
OSError: [Errno 2] No such file or directory: '/home/ms/tr/tm1e/Scripts/flex_extract/python/../work/FCOG_acc_SL.20131106.50805.37373.grb'

I read the comments in Ticket #142 but I don't think the reason of errors are same.

Regards.

[pesei]

Hi Efem,

I suggest that you activate the debugging option to keep the MARS request file, and then investigate the generated request to see what is going wrong.

[anphi]

Hi Efem,

please specify the CONTROL file you'd like to use AND tell flex_extract to use the ECMWF server, for example:

./submit.py --start_date=20131107 --queue=ecgate --controlfile=CONTROL_EA5

[ebilgic]

Replying to pesei:

Hi Efem,

I suggest that you activate the debugging option to keep the MARS request file, and then investigate the generated request to see what is going wrong.

You mean:

./submit.py --start_date=20131107 --debug=1

If yes, this command gave same things in command screen. Nothing happened related debugging.

Kind regards.

[ebilgic]

Replying to anphi:

Hi Efem,

please specify the CONTROL file you'd like to use AND tell flex_extract to use the ECMWF server, for example:

./submit.py --start_date=20131107 --queue=ecgate --controlfile=CONTROL_EA5

I tried it too. But, I received same error. Only difference: this error came via mail.

Retrieving EC data!
start date 20131107
end date 20131107
ecapi: True
removing content of /gpfs/scratch/ms/tr/tm1e/python40993/./work
retrieve 20131106/to/20131108 in dir
/gpfs/scratch/ms/tr/tm1e/python40993/./work
accuracy: 24
stream: OPER
levelist: 1
area: 90.0/-179.0/-90.0/180.0
marsclass: EA
levtype: SFC
gaussian:
number: OFF
expver: 1
repres:
public: 0
step: 1/to/12/by/1
grid: 1.0/1.0
param: LSP/CP/SSHF/EWSS/NSSS/SSR
time: 06/18
date: 20131106/to/20131108
resol: 159
dataset:
type: FC
target:
/gpfs/scratch/ms/tr/tm1e/python40993/./work/FCOG_acc_SL.20131106.40993.41014.grb
target:
/gpfs/scratch/ms/tr/tm1e/python40993/./work/FCOG_acc_SL.20131106.40993.41014.grb
EXECUTE RETRIEVAL!
2020-02-13 11:48:55 ECMWF API python library 1.5.0
2020-02-13 11:48:55 ECMWF API at https://api.ecmwf.int/v1
Error:  <class 'ecmwfapi.api.APIException'>
MARS Request failed
Traceback (most recent call last):
  File "/home/ms/tr/tm1e/flex_extract_v7.0.4/python/submit.py", line 115,
in <module>
    main()
  File "/home/ms/tr/tm1e/flex_extract_v7.0.4/python/submit.py", line 57,
in main
    getMARSdata(args,c)
  File "/home/ms/tr/tm1e/flex_extract_v7.0.4/python/getMARSdata.py", line
133, in getMARSdata
    flexpart.retrieve(server, c.public, dates, c.request, times, c.inputdir)
  File "/home/ms/tr/tm1e/flex_extract_v7.0.4/python/FlexpartTools.py",
line 1068, in retrieve
    MR.dataRetrieve()
  File "/home/ms/tr/tm1e/flex_extract_v7.0.4/python/FlexpartTools.py",
line 764, in dataRetrieve
    if not int(self.public) and os.stat(target).st_size==0:
OSError: [Errno 2] No such file or directory:
'/gpfs/scratch/ms/tr/tm1e/python40993/./work/FCOG_acc_SL.20131106.40993.41014.grb'

Best regards.

[pesei]

Replying to ebilgic:

Replying to pesei:

Hi Efem,

I suggest that you activate the debugging option to keep the MARS request file, and then investigate the generated request to see what is going wrong.

You mean:

./submit.py --start_date=20131107 --debug=1

If yes, this command gave same things in command screen. Nothing happened related debugging.

It should preserve the file with the MARS request in your working directory. You have to find it and then look at it to understand whether there is a mistake in the request or not.

[osandvik]

Hi all,

I have been having the same issue (and error messages) with Flex_Extract when downloading ERA5 files. My guess is that this error stems from that the ERA5 data has been moved to Copernicus Climate Change Service (C3S) Climate Data Store (CDS), which uses the CDS API instead of the ECMWF Web API. Could this be case?

Best regards.

[pesei]

Replying to osandvik:

I have been having the same issue (and error messages) with Flex_Extract when downloading ERA5 files. My guess is that this error stems from that the ERA5 data has been moved to Copernicus Climate Change Service (C3S) Climate Data Store (CDS), which uses the CDS API instead of the ECMWF Web API. Could this be case?

Efem is directly working with MARS on ecgate, no web API involved.

In any case, if you have trouble, please clearly post your access mode and a sample request that does not work.

[osandvik]

I have been having the same issue (and error messages) with Flex_Extract when downloading ERA5 files. My guess is that this error stems from that the ERA5 data has been moved to Copernicus Climate Change Service (C3S) Climate Data Store (CDS), which uses the CDS API instead of the ECMWF Web API. Could this be case?

Efem is directly working with MARS on ecgate, no web API involved.

In any case, if you have trouble, please clearly post your access mode and a sample request that does not work.

Thanks for the quick response. I am confused what you mean with "access mode" but a request that fails for me is using the standard CONTROL_EA5 file:

./submit.py --controlfile=CONTROL_EA5 --start_date=20090613 --debug=1

Which gives the following output:

Retrieving EC data!
start date 20090613
end date 20090613
ecapi: True
removing content of /home/oscar/flexpart_v10.4_3d7eebf/preprocess/flex_extract/python/../work
retrieve 20090612/to/20090614 in dir /home/oscar/flexpart_v10.4_3d7eebf/preprocess/flex_extract/python/../work
accuracy: 24
stream: OPER
levelist: 1
area: 90.0/-179.0/-90.0/180.0
marsclass: EA
levtype: SFC
gaussian:
number: OFF
expver: 1
repres:
public: 0
step: 1/to/12/by/1
grid: 1.0/1.0
param: LSP/CP/SSHF/EWSS/NSSS/SSR
time: 06/18
date: 20090612/to/20090614
resol: 159
dataset:
type: FC
target: /home/oscar/flexpart_v10.4_3d7eebf/preprocess/flex_extract/python/../work/FCOG_acc_SL.20090612.11.86.grb
target:  /home/oscar/flexpart_v10.4_3d7eebf/preprocess/flex_extract/python/../work/FCOG_acc_SL.20090612.11.86.grb
EXECUTE RETRIEVAL!
2020-02-14 13:46:07 ECMWF API python library 1.5.4
2020-02-14 13:46:07 ECMWF API at https://api.ecmwf.int/v1
2020-02-14 13:46:07 Welcome Oscar Sandvik
2020-02-14 13:46:07 In case of problems, please check https://confluence.ecmwf.int/display/WEBAPI/Web+API+FAQ or contact servicedesk@ecmwf.int
2020-02-14 13:46:08 Request submitted
2020-02-14 13:46:08 Request id: 5e4696e9e3242fa6ac3cf8fd
2020-02-14 13:46:08 Request is submitted
Error:  <class 'ecmwfapi.api.APIException'>
MARS Request failed
Traceback (most recent call last):
  File "./submit.py", line 115, in <module>
    main()
  File "./submit.py", line 57, in main
    getMARSdata(args,c)
  File "/home/oscar/flexpart_v10.4_3d7eebf/preprocess/flex_extract/python/getMARSdata.py", line 133, in getMARSdata
    flexpart.retrieve(server, c.public, dates, c.request, times, c.inputdir)
  File "/home/oscar/flexpart_v10.4_3d7eebf/preprocess/flex_extract/python/FlexpartTools.py", line 1068, in retrieve
    MR.dataRetrieve()
  File "/home/oscar/flexpart_v10.4_3d7eebf/preprocess/flex_extract/python/FlexpartTools.py", line 764, in dataRetrieve
    if not int(self.public) and os.stat(target).st_size==0:
OSError: [Errno 2] No such file or directory: '/home/oscar/flexpart_v10.4_3d7eebf/preprocess/flex_extract/python/../work/FCOG_acc_SL.20090612.11.86.grb'

Out of curiosity, why does area in the CONTROL file start from -179 deg E? Would that not miss the data between -180 and -179 deg E?

I am running Flex_Exctract under:

Ubuntu 18.04.3 LTS

Python 2.7.16 |Anaconda, Inc.| (default, Mar 14 2019, 21:00:58)
[GCC 7.3.0]

Best regards.

[pesei]

Replying to osandvik:

Thanks for the quick response. I am confused what you mean with "access mode"

It means either access directly through MARS with an authorised member-state account on ecgate, or as a member-state user through the API, or as a public user through the API.

Out of curiosity, why does area in the CONTROL file start from -179 deg E? Would that not miss the data between -180 and -179 deg E?

With 1 deg horizontal resolution, that is exactly what you want. No grid points between -180 and -179.

[osandvik]

Replying to pesei:

Replying to osandvik:

Thanks for the quick response. I am confused what you mean with "access mode"

It means either access directly through MARS with an authorised member-state account on ecgate, or as a member-state user through the API, or as a public user through the API.

Ah, okay. I am accessing as a member-state user through the API.

Out of curiosity, why does area in the CONTROL file start from -179 deg E? Would that not miss the data between -180 and -179 deg E?

With 1 deg horizontal resolution, that is exactly what you want. No grid points between -180 and -179.

Okay, thanks :)

[ebilgic]

Replying to pesei:

Replying to ebilgic:

Replying to pesei:

Hi Efem,

I suggest that you activate the debugging option to keep the MARS request file, and then investigate the generated request to see what is going wrong.

You mean:

./submit.py --start_date=20131107 --debug=1

If yes, this command gave same things in command screen. Nothing happened related debugging.

It should preserve the file with the MARS request in your working directory. You have to find it and then look at it to understand whether there is a mistake in the request or not.

Unfortunately no file was produced in working directory when I type "--debug=1". Is there any other options for debugging?

[ebilgic]

I realize that Era interim doesn't give this error. The error occurs only when using EA5 controlfiles.

CONTROLFILE_EIs also gave error. But it's about emoslib version. After I recompile the with emos/455-r64, this problem was resolved (#224).

[anphi]

Dear osandvik and nanditha,

as I already told ebilgic, please use flex_extract in version 7.1 from now on (details are listed ​here). Current problems in installation are documented in ticket #261.

Version 7.0.4 has some problems which will not be fixed anymore.

Kind regards,
anphi