source: flex_extract.git/Documentation/html/_sources/Installation/gateway.rst.txt @ 08c9091

ctbtodev
Last change on this file since 08c9091 was 08c9091, checked in by Anne Philipp <anne.philipp@…>, 4 years ago

updated online docu (rm tabs)
  • Property mode set to 100644
File size: 11.5 KB
Line 
1*************************
2Gateway mode installation
3*************************
4
5.. role:: underline
6    :class: underline
7   
8.. toctree::
9    :hidden:
10    :maxdepth: 2
11   
12.. _Python 3: https://docs.python.org/3/
13.. _Python3: https://www.python.org/downloads/
14.. _Anaconda Python3: https://www.anaconda.com/distribution/#download-section
15
16.. _numpy: http://www.numpy.org/
17.. _ecmwf-api-client: https://confluence.ecmwf.int/display/WEBAPI/ECMWF+Web+API+Home
18.. _cdsapi: https://cds.climate.copernicus.eu/api-how-to
19.. _genshi: https://genshi.edgewall.org/
20.. _eccodes for python: https://packages.debian.org/sid/python3-eccodes
21.. _eccodes for conda: https://anaconda.org/conda-forge/eccodes
22.. _gfortran: https://gcc.gnu.org/wiki/GFortran
23.. _fftw3: http://www.fftw.org
24.. _eccodes: https://software.ecmwf.int/wiki/display/ECC
25.. _emoslib: https://software.ecmwf.int/wiki/display/EMOS/Emoslib
26.. _member state: https://www.ecmwf.int/en/about/who-we-are/member-states
27.. _registration form: https://apps.ecmwf.int/registration/
28.. _CDS API registration: https://cds.climate.copernicus.eu/user/register
29.. _ECMWF ectrans site: https://confluence.ecmwf.int/display/ECAC/Unattended+file+transfer+-+ectrans
30.. _ECaccess Presentation: https://confluence.ecmwf.int/download/attachments/45759146/ECaccess.pdf
31.. _ECMWF's instructions on gateway server: https://confluence.ecmwf.int/display/ECAC/ECaccess+Home
32.. _Computing Representative: https://www.ecmwf.int/en/about/contact-us/computing-representatives
33.. _MARS access: https://confluence.ecmwf.int//display/WEBAPI/Access+MARS
34
35.. _download section: https://www.flexpart.eu/downloads
36
37
38
39.. _ref-gateway-mode:
40
41
42
43.. _ref-req-gateway:
44 
45Gateway mode - dependencies
46===========================
47
48* `Python3`_
49* `numpy`_
50* `genshi`_
51 
52
53.. _ref-prep-gateway:
54
55Prepare gateway environment
56===========================
57
58In this mode, access to the ECMWF computing and archiving facilities is enabled through an ECaccess gateway server on a local member state server. The ECaccess framework is necessary to interactively submit jobs to the ECMWF batch system and to transfer files between ECMWF and local gateway server.
59
60Please see `ECMWF's instructions on gateway server`_ to establish the gateway server if not already in place. Additionally, to be able to use the ECaccess file transfer service **ectrans** it is necessary to create a so-called association. The easiest way is to visit the ECaccess Member State Gateway website (e.g. https://msgatway.ecmwf.int) and follow the instructions in the short `ECaccess Presentation`_ (page 17 ff.). Additional documentation can be found on the `ECMWF ectrans site`_.
61
62The next step is to create an *ecaccess certificate* to be able to send and receive files from and to the ECMWF server. You can do this by using the ``ecaccess-certificate-create`` command on the gateway server. You will be prompted for your ECMWF member state user name and a password (generated by a token). This certificate has to be renewed periodically (every 7 days).
63
64
65.. code-block:: bash
66   
67   $ ecaccess-certificate-create
68   Please enter your user-id: <username>
69   Your passcode: ***
70   
71   
72   
73The easiest way to install all required packages is to use the package management system of your Linux distribution which requires admin rights.
74The installation was tested under *GNU/Linux Debian buster* and *Ubuntu 18.04 Bionic Beaver*.
75
76.. code-block:: sh
77
78
79   # On a Debian or Debian-derived sytem (e. g. Ubuntu) system you may use the following commands (or equivalent commands of your preferred package manager):
80   # (if not already available):
81   apt-get install python3
82   apt-get install python3-genshi
83   apt-get install python3-numpy
84   
85
86.. _ref-test-gateway:
87
88Test gateway environment
89========================
90
91Check the availability of the python packages by typing ``python3`` in
92a terminal window and run the ``import`` commands in the python shell.
93If there are no error messages, you succeeded in setting up the environment.
94 
95.. code-block:: python
96   
97   # check in python3 console
98   import genshi
99   import numpy
100 
101
102
103.. _ref-install-gateway:
104
105Gateway installation
106====================
107
108``Flex_extract`` will be run on an ECMWF server which makes the setup the same as for the **remote mode**. In the ``setup.sh`` script :doc:`../Documentation/Input/setup`, select the ``makefile_ecgate`` or ``makefile_cray`` for the ``calc_etadot`` Fortran program and the ECMWF server (*target*) you would like to use.
109The job script, send to the job queue via the ECaccess software, loads the required modules from the module system. For enabling the file transfer via the gateway, you have to set the *ECUID*, *ECGID*, *GATEWAY* and *DESTINATION* parameter values.
110
111Open ``setup.sh`` with your preferred editor (e.g., nano) and adapt the values: 
112   
113+----------------------------------------------+----------------------------------------------+   
114|   Use this for target = **ectrans**          |   Use this for target = **cca** or **ccb**   |
115+----------------------------------------------+----------------------------------------------+
116| .. code-block:: bash                         | .. code-block:: bash                         |
117|                                              |                                              |
118|   ...                                        |   ...                                        |   
119|   # -----------------------------------------|   # -----------------------------------------|
120|   # AVAILABLE COMMANDLINE ARGUMENTS TO SET   |   # AVAILABLE COMMANDLINE ARGUMENTS TO SET   |
121|   #                                          |   #                                          | 
122|   # THE USER HAS TO SPECIFY THESE PARAMETER  |   # THE USER HAS TO SPECIFY THESE PARAMETER  |
123|   #                                          |   #                                          |
124|   TARGET='ecgate'                            |   TARGET='cca'                               |
125|   MAKEFILE='makefile_ecgate'                 |   MAKEFILE='makefile_cray'                   | 
126|   ECUID='<username>'                         |   ECUID='<username>'                         | 
127|   ECGID='<groupID>'                          |   ECGID='<groupID>'                          |
128|   GATEWAY='<gatewayname>'                    |   GATEWAY='<gatewayname>'                    |
129|   DESTINATION='<assoc_name>@genericSftp'     |   DESTINATION='<assoc_name>@genericSftp'     |
130|   INSTALLDIR=None                            |   INSTALLDIR=None                            |
131|   JOB_TEMPLATE='installscript.template'      |   JOB_TEMPLATE='installscript.template'      |
132|   CONTROLFILE='CONTROL_EA5'                  |   CONTROLFILE='CONTROL_EA5'                  |
133|   ...                                        |   ...                                        |   
134+----------------------------------------------+----------------------------------------------+
135
136
137
138
139Afterwards, type:
140
141.. code-block:: bash
142
143   $ ./setup.sh
144   
145to start the installation. You should see the following on standard output.
146   
147   
148.. code-block:: bash
149
150   # Output of setup.sh
151   Create tarball ...
152   Job compilation script has been submitted to ecgate for installation in ${HOME}/flex_extract_vX.X
153   You should get an email with subject "flexcompile" within the next few minutes!
154
155``Flex_extract`` automatically uses the email address connected to the user account on ECMWF servers. The email content should look like this with a "SUCCESS" statement in the last line:
156
157.. code-block:: bash
158
159    gfortran    -O3 -march=native -L/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -Wl,-rpath,/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -leccodes_f90 -leccodes -ljasper -lpthread -L/usr/local/apps/jasper/1.900.1/LP64/lib -ljasper -lm -L/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -Wl,-rpath,/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -lemos.R64.D64.I32 -L/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -lfftw3    -I. -I/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c ./rwgrib2.f90
160    gfortran    -O3 -march=native -L/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -Wl,-rpath,/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -leccodes_f90 -leccodes -ljasper -lpthread -L/usr/local/apps/jasper/1.900.1/LP64/lib -ljasper -lm -L/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -Wl,-rpath,/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -lemos.R64.D64.I32 -L/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -lfftw3    -I. -I/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c ./phgrreal.f90
161    gfortran    -O3 -march=native -L/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -Wl,-rpath,/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -leccodes_f90 -leccodes -ljasper -lpthread -L/usr/local/apps/jasper/1.900.1/LP64/lib -ljasper -lm -L/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -Wl,-rpath,/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -lemos.R64.D64.I32 -L/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -lfftw3    -I. -I/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c ./grphreal.f90
162    gfortran    -O3 -march=native -L/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -Wl,-rpath,/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -leccodes_f90 -leccodes -ljasper -lpthread -L/usr/local/apps/jasper/1.900.1/LP64/lib -ljasper -lm -L/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -Wl,-rpath,/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -lemos.R64.D64.I32 -L/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -lfftw3    -I. -I/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c ./ftrafo.f90
163    gfortran    -O3 -march=native -L/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -Wl,-rpath,/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -leccodes_f90 -leccodes -ljasper -lpthread -L/usr/local/apps/jasper/1.900.1/LP64/lib -ljasper -lm -L/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -Wl,-rpath,/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -lemos.R64.D64.I32 -L/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -lfftw3    -I. -I/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c ./calc_etadot.f90
164    gfortran    -O3 -march=native -L/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -Wl,-rpath,/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -leccodes_f90 -leccodes -ljasper -lpthread -L/usr/local/apps/jasper/1.900.1/LP64/lib -ljasper -lm -L/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -Wl,-rpath,/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -lemos.R64.D64.I32 -L/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -lfftw3    -I. -I/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/include -fdefault-real-8 -fopenmp -fconvert=big-endian   -c ./posnam.f90
165    gfortran   rwgrib2.o calc_etadot.o ftrafo.o grphreal.o posnam.o phgrreal.o -o calc_etadot_fast.out  -O3 -march=native -L/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -Wl,-rpath,/usr/local/apps/eccodes/2.13.0/GNU/7.3.0/lib -leccodes_f90 -leccodes -ljasper -lpthread -L/usr/local/apps/jasper/1.900.1/LP64/lib -ljasper -lm -L/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -Wl,-rpath,/usr/local/apps/libemos/000455/GNU/6.3.0/lib  -lemos.R64.D64.I32 -L/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -Wl,-rpath,/usr/local/apps/fftw/3.3.4/GNU/6.3.0/lib -lfftw3    -fopenmp
166    ln -sf calc_etadot_fast.out calc_etadot
167    lrwxrwxrwx. 1 <username> at 20 Mar  8 14:11 calc_etadot -> calc_etadot_fast.out
168    SUCCESS!   
Note: See TracBrowser for help on using the repository browser.
hosted by ZAMG