Installation of flex_extract v7.1

[ebilgic]

Hi,

I am trying to install last version of flex_extract (v7.1). It seems easier than older versions. But, I failed in the beginning.

First, I changed related parameters in setup.sh:

 TARGET='ecgate'
    MAKEFILE='Makefile.gfortran'
    ECUID='tm1e'
    ECGID='tr'
    GATEWAY=None
    DESTINATION=None
    INSTALLDIR=None
    JOB_TEMPLATE='job.template'
    CONTROLFILE='CONTROL_EA5'

and I loaded pyhton3 module:

    module load python3

I get this message:

    load python3 3.6.8-01 (PATH, MANPATH)

I think everything is ok, but wen I run the setup.sh (./setup.sh), I received following message:

Please enter your ECMWF user id and group id as well as the
    name of the local gateway and the ectrans destination
    with command line options --ecuid --ecgid                    --gateway --destination
    Try "install.py -h" to print usage information
    Please consult ecaccess documentation or ECMWF user                    support for further details

Regards,
Efem

[anphi]

First of all, you are doing the installation directly on ECMWF's ecgate server. Secondly, you don't have a local gateway server for to apply file transfer. This is important to know.

There are two problems here:

1. I did not think about a case where one uses the ECMWF server without having a local gateway server to send the resulting files to. Therefore, if you use the ecgate or cca/b version of flex_extract you have to set the GATEWAY and DESTINATION parameters. I the case you don't have a local gateway server, it is enough to set dummy values. E.g.

   GATEWAY=dummy@gateway
   DESTINATION=dummy@genericSftp
   

2. the makefile nameing has changed recently and is not documented yet. You can look in the directory Source/Fortran for a list of available makefiles. But in the case of installing it on ecgate take this one : makefile_ecgate

Anyway, I will create 2 tickets to deal with this behavior properly.

[ebilgic]

Replying to anphi:

1. I did not think about a case where one uses the ECMWF server without having a local gateway server to send the resulting files to. Therefore, if you use the ecgate or cca/b version of flex_extract you have to set the GATEWAY and DESTINATION parameters. I the case you don't have a local gateway server, it is enough to set dummy values. E.g.

   GATEWAY=dummy@gateway
   DESTINATION=dummy@genericSftp
   

2. the makefile nameing has changed recently and is not documented yet. You can look in the directory Source/Fortran for a list of available makefiles. But in the case of installing it on ecgate take this one : makefile_ecgate

I applied both of them and it worked. But I received an e mail that compilation is not completed:

make: *** No rule to make target `phgrreal.f', needed by `phgrreal.o'. Stop.

Actually there is a file called phgrreal.f90 in Fortran directory. Should I update extentions as 'f90' instead of 'f' in makefile?

[pesei]

Replying to ebilgic:

Actually there is a file called phgrreal.f90 in Fortran directory. Should I update extensions as 'f90' instead of 'f' in makefile?

Oh, I am afraid that the makefile_ecgate is not up-to-date. Please compare with makefile_fast. I'll take care of this as soon as possible.

[ebilgic]

Replying to pesei:

Replying to ebilgic:

Actually there is a file called phgrreal.f90 in Fortran directory. Should I update extensions as 'f90' instead of 'f' in makefile?

Oh, I am afraid that the makefile_ecgate is not up-to-date. Please compare with makefile_fast. I'll take care of this as soon as possible.

Before I read your reply, I change file extension as I mentioned. I received a success message:

 gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore
-ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp
phgrreal.f90
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore
-ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp
grphreal.f90
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore
-ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -g -O3 -fopenmp
ftrafo.f90
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore
-ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I.
-I/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/include -g rwgrib2.f90
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore
-ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I.
-I/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/include -g posnam.f90
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore
-ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -c -O3 -I.
-I/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/include -g calc_etadot.f90
gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore
-ffixed-line-length-132 -fopenmp  -fconvert=big-endian  -g -O3 -o
./calc_etadot ftrafo.o phgrreal.o grphreal.o rwgrib2.o posnam.o
calc_etadot.o -L/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib
-Wl,-rpath,/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -leccodes_f90
-leccodes -ljasper -lpthread -L/usr/local/apps/jasper/1.900.1/LP64/lib
-ljasper -lm -L/usr/local/apps/libemos/000455/GNU/6.3.0/lib
-Wl,-rpath,/usr/local/apps/libemos/000455/GNU/6.3.0/lib
-lemos.R64.D64.I32 -L/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib
-Wl,-rpath,/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -lfftw3
-rwxr-x---. 1 tm1e tr 358369 Feb 27 17:16 calc_etadot
SUCCESS!

However, while I expected CONVERT2 file, this makefile produced a file called 'calc_etadot'. Exe of makefile_fast is also 'calc_etadot'. Is it ok? Should I update also it as CONVERT2?

[pesei]

No, we changed the name to be more meaningful, and we don't use all-upercase. However, we have not yet tested your ad-hoc makefile and the executable produced by it. So use at your own risk.

[ebilgic]

Replying to pesei:

No, we changed the name to be more meaningful, and we don't use all-upercase. However, we have not yet tested your ad-hoc makefile and the executable produced by it. So use at your own risk.

So, I didn't change anything else. And I conducted a test regarding online documentation (​https://www.flexpart.eu/flex_extract/quick_start.html#remote-and-gateway-modes). It worked well, and I get CERA data.

STOP SUCCESSFULLY FINISHED calc_etadot: CONGRATULATIONS
STATISTICS:          98144.5219  98059.7628   4078.0015
124164554
124164556
124164557
124164559
124164562
124164565
124164568
124164569
outputfile = /gpfs/scratch/ms/tr/tm1e/python48667/./work/CE00090821


Postprocessing:
 Format: GRIB1

ecstorage: 0
 ecfsdir: ectmp:/tm1e/econdemand/

ectrans: 1
 gateway: dummy@gateway
 destination: dummy@genericSftp

Output filelist:
['CE00090800', 'CE00090803', 'CE00090806', 'CE00090809', 'CE00090812',
'CE00090815', 'CE00090818', 'CE00090821']
... clean inputdir!
... done!
FLEX_EXTRACT IS DONE!

However, I think there is a one minor problem. Actually I expected the output files in a 'work' directory in my workspace (/home/ms/tr/tm1e/flex_extract_v7.1/work or /home/ms/tr/tm1e/flex_extract_v7.1/Run/work). But, these were produced under /gpfs/stratch/....

It's ok for me, I can access data, but it's also unfamiliar situation.

[anphi]

On ECMWF servers that is the normal behavior as should be described in the documentation (isn't it?). Since the home directory doesn't have lots of memory it does contain the software only. All retrieval jobs are done in the $SCRATCH directory of yours! Naming convention for each retrieval job is pythonXXXX where the X is an individual process id.

[anphi]

I created the tickets #262, #263, #264 and #265 from the above findings as tasks.

Since the installation worked, the issue is solved and I will close the ticket.