on ecgate, ECCODES ERRORS : Maximum value out of range: inf

[bsteil]

Dear Flex_EXTRACT developers or users,

I'm trying to extract fields at ECMWF on ecgate.
flex_extract_v7.0.4 or flex_extract_v7.0.3 always produce the same error:

ECCODES ERROR   :  Maximum value out of range: inf.

This happens, when submit.py executes prepareFLEXPART.py. The downloading of the data
from MARS archive works without problems.
The error Maximum value out of range: inf also occurs, if I use the module grip_api instead of eccodes. What I'm doing wrong?

My installation process :
I have installed flex_extract_v7.0.4 directly on ecgate and I directly execute it there.
The gfortran I'm using is: /usr/local/apps/gcc/6.3.0/bin/gfortran

I have loaded the required modules in this order:
python 2.7.12-01 , jasper 1.900.1 , eccodes 2.9.2 , fftw 3.3.4 , emos 458

I have compiled the code in the subdirectory python with:
./install.py --target=local --makefile=Makefile.gfortran
Compilation is successful, no problems, no warnings here.

After inserting DAY1 20180711 and DAY2 20180711 in CONTROL.temp,
I run the model on ecgate with following command:

./submit.py --controlfile CONTROL.temp --inputdir /scratch/ms/datex/dw0/FLEXPART/INPUT --outputdir / scratch/ms/datex/dw0/FLEXPART/OUTPUT

After the successful download of the data from MARS-archive and the creation of a lot of
flux and fort files the process crashes with the following error message:

ECCODES ERROR   :  Maximum value out of range: inf
ECCODES ERROR   :  unable to compute packing parameters

ECCODES ERROR   :  unable to set double array codedValues (Encoding invalid)
ECCODES ERROR   :  set: values Encoding invalid
Traceback (most recent call last):
  File "./submit.py", line 115, in <module>
    main()
  File "./submit.py", line 59, in main
    prepareFLEXPART(args,c)
  File "/home/ms/datex/dw0/FLEXPART/flex_extract_v7.0.4/flex_extract/python/prepareFLEXPART.py", line  96, in prepareFLEXPART    flexpart.create(inputfiles, c) # produce FLEXPART-ready GRIB files
  File "/home/ms/datex/dw0/FLEXPART/flex_extract_v7.0.4/flex_extract/python/FlexpartTools.py", line    1468, in create
    p=subprocess.check_call([os.path.expandvars(os.path.expanduser(c.exedir))+'/CONVERT2'],shell=True)
  File "/usr/local/apps/python/2.7.12-01/lib/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/ms/datex/dw0/FLEXPART/flex_extract_v7.0.4/flex_extract/python/../src//CONVERT2']' returned non-zero exit status 242

The same happens using flex_extract_v7.0.3. I have also tested other control files. Always
the same error.

Thank you very much,
with kind regards,

Benedikt Steil

[anphi]

Hi Benedikt,

I know this error. We had this error with version 7.0.3 because a specific field in the CONVERT2 program wasn't initialized manually with 0. values and the gfortran compiler did not do the job either (as was expected).

After doing the initialization manually it worked well for us. So I need some time to find out whats going on now.

Anne

[bsteil]

Hi Anne,

thank you very much for your help. This error also occurs with version 7.0.3.
Actually, I have been running into this error with version 7.0.3 and I have switched to version 7.0.4 hoping this would help

Cheers,
Benedikt

[anphi]

Could you please check if there is feld = 0. on line 25 in file src/rwGRIB2.f90 in version 7.0.4?

I am not surprised that 7.0.3 does not work. But I am very surprised for version 7.0.4. I just did a test run on ecgate and it works well for me.

[bsteil]

Hi Anne,

yes this line is included in file src/rwGRIB2.f90 in version 7.0.4.
What gfortran version are you using?

Cheers,
Benedikt

[anphi]

Hey,
I use the standard gfortran compiler on ecgate: /usr/local/apps/gcc/6.3.0/bin/gfortran
Cheers,
Anne

[bsteil]

Hi Anne,

so do I. Could you please do a module list on ecgate?
These are the modules I have loaded:

1) version/3.2.10(default) 9) python/2.7.12-01(may18:new:default)
2) verbose/true(default) 10) perl/5.16.3-01(default)
3) mode/64(default) 11) ecaccess/4.0.2(default)
4) gnu/6.3.0(default) 12) sms/4.4.13(default)
5) slurm/17.02.2(default) 13) metview/5.2.3(nov18:default)
6) jasper/1.900.1(default) 14) ecmwf/1.0(default)
7) eccodes/2.9.2(nov18:default) 15) fftw/3.3.4(default)
8) ecfs/2.2.5(nov18:test:new:prodn:default) 16) emos/458(nov18:default)

Cheers,
Benedikt

[anphi]

Hi Benedikt,

Currently Loaded Modulefiles:

1) version/3.2.10(default)
2) verbose/true(default)
3) mode/64(default)
4) gnu/6.3.0(default)
5) slurm/17.02.2(default)
6) jasper/1.900.1(default)
7) eccodes/2.9.2(nov18:default)
8) ecfs/2.2.5(nov18:test:new:prodn:default)
9) python/2.7.12-01(may18:new:default)

10) perl/5.16.3-01(default)
11) ecaccess/4.0.2(default)
12) sms/4.4.13(default)
13) metview/5.2.3(nov18:default)
14) ecmwf/1.0(default)
15) fftw/3.3.4(default)
16) emos/458(nov18:default)

Cheers,
Anne

[anphi]

Now I was able to reproduce the error. I accidentally had an old CONVERT2 executable and recompiled it now.

From my intuition I switched emos for a test and CONVERT2 worked (except ectrans at the end which I investigate now).

Could you please try to use emos/455-r64 with eccodes instead of emos/458.
You have to reinstall and then try to use the prepareFlexpart (only postprocessing the data)

Call from the python directory (substitute the ppid):
e.g. the ppid in this file ANOG__ML.20180711.7470.34710.grb is: ppid=34710

./prepareFlexpart.py --controlfile=CONTROL.temp --ppid=???? --debug=1 --inputdir=/scratch/ms/datex/dw0/FLEXPART/INPUT --outputdir=/scratch/ms/datex/dw0/FLEXPART/OUTPUT

[pesei]

Hi all, could you please use proper trac syntax for code snippets? It will make the ticket much more readable and avoids misinterpretation of -rsomenumber and such stuff.. I mean backquotes for inline, and triple braces for blocks. Also for enumerations there is a proper syntac. Thanks!

[bsteil]

Dear Anne,

thank's a lot. Switching to emos/455-r64 has helped.
I have recompiled and resubmitted the job with:

 ./submit.py --controlfile CONTROL.temp --inputdir /scratch/ms/datex/dw0/FLEXPART/INPUT_temp05_org --outputdir /scratch/ms/datex/dw0/FLEXPART/OUTPUT_temp05_org

It ran nearly flawlessly. It has produced the output-files of the kind
EA18071108. The only remainig issue is, that the output-files are not moved to the outputdir.
Here is the err output of my sbatch job:

Currently Loaded Modulefiles:
  1) version/3.2.10(default)
  2) verbose/true(default)
  3) mode/64(default)
  4) slurm/17.02.2(default)
  5) ecfs/2.2.5(nov18:test:new:prodn:default)
  6) python/2.7.12-01(may18:new:default)
  7) perl/5.16.3-01(default)
  8) ecaccess/4.0.2(default)
  9) sms/4.4.13(default)
 10) metview/5.2.3(nov18:default)
 11) ecmwf/1.0(default)
 12) jasper/1.900.1(default)
 13) eccodes/2.9.2(nov18:default)
 14) gnu/6.3.0(default)
 15) fftw/3.3.4(default)
 16) emos/455-r64

STOP SUCCESSFULLY FINISHED CONVERT_PRE: CONGRATULATIONS
STOP SUCCESSFULLY FINISHED CONVERT_PRE: CONGRATULATIONS
STOP SUCCESSFULLY FINISHED CONVERT_PRE: CONGRATULATIONS
STOP SUCCESSFULLY FINISHED CONVERT_PRE: CONGRATULATIONS
STOP SUCCESSFULLY FINISHED CONVERT_PRE: CONGRATULATIONS
STOP SUCCESSFULLY FINISHED CONVERT_PRE: CONGRATULATIONS
STOP SUCCESSFULLY FINISHED CONVERT_PRE: CONGRATULATIONS
STOP SUCCESSFULLY FINISHED CONVERT_PRE: CONGRATULATIONS
STOP SUCCESSFULLY FINISHED CONVERT_PRE: CONGRATULATIONS
STOP SUCCESSFULLY FINISHED CONVERT_PRE: CONGRATULATIONS
STOP SUCCESSFULLY FINISHED CONVERT_PRE: CONGRATULATIONS
STOP SUCCESSFULLY FINISHED CONVERT_PRE: CONGRATULATIONS
STOP SUCCESSFULLY FINISHED CONVERT_PRE: CONGRATULATIONS
STOP SUCCESSFULLY FINISHED CONVERT_PRE: CONGRATULATIONS
STOP SUCCESSFULLY FINISHED CONVERT_PRE: CONGRATULATIONS
STOP SUCCESSFULLY FINISHED CONVERT_PRE: CONGRATULATIONS
STOP SUCCESSFULLY FINISHED CONVERT_PRE: CONGRATULATIONS
STOP SUCCESSFULLY FINISHED CONVERT_PRE: CONGRATULATIONS
STOP SUCCESSFULLY FINISHED CONVERT_PRE: CONGRATULATIONS
STOP SUCCESSFULLY FINISHED CONVERT_PRE: CONGRATULATIONS
STOP SUCCESSFULLY FINISHED CONVERT_PRE: CONGRATULATIONS
STOP SUCCESSFULLY FINISHED CONVERT_PRE: CONGRATULATIONS
STOP SUCCESSFULLY FINISHED CONVERT_PRE: CONGRATULATIONS
STOP SUCCESSFULLY FINISHED CONVERT_PRE: CONGRATULATIONS

Traceback (most recent call last):
  File "./submit.py", line 115, in <module>
    main()
  File "./submit.py", line 59, in main
    prepareFLEXPART(args,c)
  File "/home/ms/datex/dw0/FLEXPART/flex_extract_v7.0.4/flex_extract/python/prepareFLEXPART.py", line 97, in prepareFLEXPART
    flexpart.process_output(c) # process GRIB files - copy/transfer/interpolate them or make them GRIB2
  File "/home/ms/datex/dw0/FLEXPART/flex_extract_v7.0.4/flex_extract/python/FlexpartTools.py", line 1244, in process_output
    '-remote',c.destination,'-source',ofile])
  File "/usr/local/apps/python/2.7.12-01/lib/python2.7/subprocess.py", line 536, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/usr/local/apps/python/2.7.12-01/lib/python2.7/subprocess.py", line 523, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/local/apps/python/2.7.12-01/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/local/apps/python/2.7.12-01/lib/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
TypeError: execv() arg 2 must contain only strings

Cheers,
Benedikt

[anphi]

Hi Benedikt,

nice to hear that its working now.

The other problem is a minor thing and easy to solve:

The program crashes because it cannot find GATEWAY and DESTINATION information for the ectrans command. In the CONTROL.temp the ectrans is switched on, but if you work locally on ecgate flex_extract does not take the information from the ECMWF_ENV file (was not a requirement for this version, but will change in the next version). So you have to switch it off in the control file OR you have to add this information in the CONTROL.temp file. Keep in mind, that ectrans only works with local gateway servers!

Then it should work.

Cheers,
Anne

[bsteil]

Dear Anne,

thank you very much for your help.
That was great.

Cheers,
Benedikt

[anphi]

Dear Benedikt,

no problem. You're welcome.

So I close the ticket now.

Cheers,
Anne