Changeset d720895 in flex_extract.git for Documentation/html/Installation/gateway.html


Ignore:
Timestamp:
Dec 19, 2019, 8:12:11 PM (4 years ago)
Author:
Anne Philipp <anne.philipp@…>
Branches:
master, ctbto, dev
Children:
e70c85f
Parents:
76c37a9
Message:

updated online docu

File:
1 edited

Legend:

Unmodified
Added
Removed

  • Documentation/html/Installation/gateway.html

    r30f7911 rd720895  
    9494<li class="toctree-l2"><a class="reference internal" href="../installation.html#licence-agreement-for-public-datasets">Licence agreement for public datasets</a></li>
    9595<li class="toctree-l2"><a class="reference internal" href="../installation.html#download-flex-extract">Download <code class="docutils literal notranslate"><span class="pre">flex_extract</span></code></a></li>
    96 <li class="toctree-l2"><a class="reference internal" href="../installation.html#environment-requirements">Environment requirements</a></li>
     96<li class="toctree-l2"><a class="reference internal" href="../installation.html#dependencies">Dependencies</a></li>
    9797<li class="toctree-l2 current"><a class="reference internal" href="../installation.html#installation-of-flex-extract">Installation of <code class="docutils literal notranslate"><span class="pre">flex_extract</span></code></a><ul class="current">
    9898<li class="toctree-l3"><a class="reference internal" href="remote.html">Remote mode installation</a></li>
    9999<li class="toctree-l3 current"><a class="current reference internal" href="#">Gateway mode installation</a><ul>
    100 <li class="toctree-l4"><a class="reference internal" href="#gateway-environment-requirements">Gateway environment requirements</a></li>
     100<li class="toctree-l4"><a class="reference internal" href="#gateway-mode-dependencies">Gateway mode - dependencies</a></li>
    101101<li class="toctree-l4"><a class="reference internal" href="#prepare-gateway-environment">Prepare gateway environment</a></li>
    102102<li class="toctree-l4"><a class="reference internal" href="#test-gateway-environment">Test gateway environment</a></li>
    … …  
    190190<div class="toctree-wrapper compound">
    191191</div>
    192 <div class="section" id="gateway-environment-requirements">
    193 <span id="ref-req-gateway"></span><span id="ref-gateway-mode"></span><h2>Gateway environment requirements<a class="headerlink" href="#gateway-environment-requirements" title="Permalink to this headline">¶</a></h2>
    194 <p>The environment on your local system has to provide these software packages
    195 and libraries, since <code class="docutils literal notranslate"><span class="pre">flex_extract</span></code> does only prepare the job script and send
    196 it to the ECMWF servers:</p>
     192<div class="section" id="gateway-mode-dependencies">
     193<span id="ref-req-gateway"></span><span id="ref-gateway-mode"></span><h2>Gateway mode - dependencies<a class="headerlink" href="#gateway-mode-dependencies" title="Permalink to this headline">¶</a></h2>
    197194<ul class="simple">
    198 <li><a class="reference external" href="https://www.python.org/downloads/">Python3</a> or <a class="reference external" href="https://www.anaconda.com/distribution/#download-section">Anaconda Python3</a></li>
     195<li><a class="reference external" href="https://www.python.org/downloads/">Python3</a></li>
    199196<li><a class="reference external" href="http://www.numpy.org/">numpy</a></li>
    200197<li><a class="reference external" href="https://genshi.edgewall.org/">genshi</a></li>
    … …  
    203200<div class="section" id="prepare-gateway-environment">
    204201<span id="ref-prep-gateway"></span><h2>Prepare gateway environment<a class="headerlink" href="#prepare-gateway-environment" title="Permalink to this headline">¶</a></h2>
    205 <p>The easiest way to install all required packages is to use the package management system of your Linux distribution. To do so, it is necessary to use a user with admin rights.
    206 The installation was tested on a <em>GNU/Linux Debian buster</em> and an <em>Ubuntu 18.04 Bionic Beaver</em> system.</p>
    207 <div class="highlight-sh notranslate"><div class="highlight"><pre><span></span><span class="c1"># On a Linux Debian or Ubuntu system do</span>
     202<p>In this mode, access to the ECMWF computing and archiving facilities is enabled through an ECaccess gateway server on a local member state server. The ECaccess framework is necessary to interactively submit jobs to the ECMWF batch system and to transfer files between ECMWF and local gateway server.</p>
     203<p>Please see <a class="reference external" href="https://confluence.ecmwf.int/display/ECAC/ECaccess+Home">ECMWF’s instructions on gateway server</a> to establish the gateway server if not already in place. Additionally, to be able to use the ECaccess file transfer service <strong>ectrans</strong> it is necessary to create a so-called association. The easiest way is to visit the ECaccess Member State Gateway website (e.g. <a class="reference external" href="https://msgatway.ecmwf.int">https://msgatway.ecmwf.int</a>) and follow the instructions in the short <a class="reference external" href="https://confluence.ecmwf.int/download/attachments/45759146/ECaccess.pdf">ECaccess Presentation</a> (page 17 ff.). Additional documentation can be found on the <a class="reference external" href="https://confluence.ecmwf.int/display/ECAC/Unattended+file+transfer+-+ectrans">ECMWF ectrans site</a>.</p>
     204<p>The next step is to create an <em>ecaccess certificate</em> to be able to send and receive files from and to the ECMWF server. You can do this by using the <code class="docutils literal notranslate"><span class="pre">ecaccess-certificate-create</span></code> command on the gateway server. You will be prompted for your ECMWF member state user name and a password (generated by a token). This certificate has to be renewed periodically (every 7 days).</p>
     205<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ ecaccess-certificate-create
     206Please enter your user-id: example_username
     207Your passcode: ***
     208</pre></div>
     209</div>
     210<p>The easiest way to install all required packages is to use the package management system of your Linux distribution which required admin rights.
     211The installation was tested under <em>GNU/Linux Debian buster</em> and <em>Ubuntu 18.04 Bionic Beaver</em>.</p>
     212<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span><span class="c1"># On a Debian or Debian-derived sytem (e. g. Ubuntu) system you may use the following commands (or equivalent commands of your preferred package manager):</span>
    208213<span class="c1"># (if not already available):</span>
    209214apt-get install python3
    210 apt-get install pip
    211 apt-get install genshi
    212 apt-get install numpy
     215apt-get install python3-genshi
     216apt-get install python3-numpy
    213217</pre></div>
    214218</div>
    … …  
    227231<div class="section" id="gateway-installation">
    228232<span id="ref-install-gateway"></span><h2>Gateway installation<a class="headerlink" href="#gateway-installation" title="Permalink to this headline">¶</a></h2>
    229 <p>In this mode, access to the ECMWF computing and archiving facilities is enabled through an ECaccess gateway server on a local member state server. The ECaccess framework is necessary to interactively submit jobs to the ECMWF batch system and to transfer files between ECMWF and local gateway server.</p>
    230 <p>Please see <a class="reference external" href="https://confluence.ecmwf.int/display/ECAC/ECaccess+Home">ECMWF’s instructions on gateway server</a> to establish the gateway server if not already in place.
    231 Additionally, to be able to use the Ecaccess file transfer service <strong>ectrans</strong> please also create an association.
    232 The easiest way is to visit the ECaccess Member State Gateway website (e.g. msgatway.ecmwf.int) and follow the instructions in the short <a class="reference external" href="https://confluence.ecmwf.int/download/attachments/45759146/ECaccess.pdf">ECaccess Presentation</a> (page 17 ff.). Additional documentation can be found on the <a class="reference external" href="https://confluence.ecmwf.int/display/ECAC/Unattended+file+transfer+-+ectrans">ECMWF ectrans site</a>.</p>
    233 <p>After everything is set up you have to create an <em>ecaccess certificate</em> to be able to send and receive files from and to the ECMWF server. You can do this by using the <code class="docutils literal notranslate"><span class="pre">ecaccess-certificate-create</span></code> command on the gateway server. You will be prompted for your ECMWF member state user name and a password (which will be usually generated by a Token). This certificate has to be re-newed periodically (every 7 days).</p>
    234 <div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>$ ecaccess-certificate-create
    235 Please enter your user-id: example_username
    236 Your passcode: ***
    237 </pre></div>
    238 </div>
    239 <p><code class="docutils literal notranslate"><span class="pre">Flex_extract</span></code> will be run on an ECMWF server which makes the setup the same as for the <strong>remote mode</strong>. In the <code class="docutils literal notranslate"><span class="pre">setup.sh</span></code> script <a class="reference external" href="Documentation/Input/setup.html">[ref]</a>, select the <code class="docutils literal notranslate"><span class="pre">Makefile.gfortran</span></code> for the <code class="docutils literal notranslate"><span class="pre">CONVERT2</span></code> Fortran program and the ECMWF server (<em>target</em>) you would like to use.
    240 The job script, send to the job queue via the ECaccess software, selects again automatically the correct libraries from the module system. For enableing the file transfer you have to set the <em>ECUID</em>, <em>ECGID</em>, <em>GATEWAY</em> and <em>DESTINATION</em> parameter values.</p>
     233<p><code class="docutils literal notranslate"><span class="pre">Flex_extract</span></code> will be run on an ECMWF server which makes the setup the same as for the <strong>remote mode</strong>. In the <code class="docutils literal notranslate"><span class="pre">setup.sh</span></code> script <a class="reference external" href="Documentation/Input/setup.html">[ref]</a>, select the <code class="docutils literal notranslate"><span class="pre">makefile.gfortran</span></code> for the <code class="docutils literal notranslate"><span class="pre">CONVERT2</span></code> Fortran program and the ECMWF server (<em>target</em>) you would like to use.
     234The job script, send to the job queue via the ECaccess software, loads the required modules from the module system. For enabling the file transfer via the gateway, you have to set the <em>ECUID</em>, <em>ECGID</em>, <em>GATEWAY</em> and <em>DESTINATION</em> parameter values.</p>
    241235<div class="literal-block-wrapper docutils container" id="setup-sh">
    242236<div class="code-block-caption"><span class="caption-text">‘Example settings for a gateway installation.’</span><a class="headerlink" href="#setup-sh" title="Permalink to this code">¶</a></div>
    … …  
    247241<span class="c1">#</span>
    248242<span class="nv">TARGET</span><span class="o">=</span><span class="s1">&#39;ecgate&#39;</span>
    249 <span class="nv">MAKEFILE</span><span class="o">=</span><span class="s1">&#39;Makefile.gfortran&#39;</span>
     243<span class="nv">MAKEFILE</span><span class="o">=</span><span class="s1">&#39;makefile.gfortran&#39;</span>
    250244<span class="nv">ECUID</span><span class="o">=</span><span class="s1">&#39;uid&#39;</span>
    251245<span class="nv">ECGID</span><span class="o">=</span><span class="s1">&#39;gid&#39;</span>
    252 <span class="nv">GATEWAY</span><span class="o">=</span><span class="s1">&#39;server.example.edu&#39;</span>
    253 <span class="nv">DESTINATION</span><span class="o">=</span><span class="s1">&#39;example@genericSftp&#39;</span>
     246<span class="nv">GATEWAY</span><span class="o">=</span><span class="s1">&#39;localserver.tld&#39;</span>
     247<span class="nv">DESTINATION</span><span class="o">=</span><span class="s1">&#39;association_name@genericSftp&#39;</span>
    254248<span class="nv">INSTALLDIR</span><span class="o">=</span>None
    255249<span class="nv">JOB_TEMPLATE</span><span class="o">=</span><span class="s1">&#39;job.template&#39;</span>
    … …  
    262256</pre></div>
    263257</div>
    264 <p>to start the installation. You should see the following output at the command line.</p>
     258<p>to start the installation. You should see the following on standard output.</p>
    265259<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># Output of setup.sh</span>
    266260Create tarball ...
    … …  
    269263</pre></div>
    270264</div>
    271 <p>The email content should look like this with a “SUCCESS” statement in the last line:</p>
     265<p><code class="docutils literal notranslate"><span class="pre">Flex_extract</span></code> uses the email address connectd to the user account on ECMWF servers. The email content should look like this with a “SUCCESS” statement in the last line:</p>
    272266<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert<span class="o">=</span>big-endian  -c -g -O3 -fopenmp phgrreal.f
    273267gfortran   -m64 -fdefault-real-8 -fcray-pointer -fno-second-underscore  -ffixed-line-length-132 -fopenmp  -fconvert<span class="o">=</span>big-endian  -c -g -O3 -fopenmp grphreal.f
Note: See TracChangeset for help on using the changeset viewer.
hosted by ZAMG