Changeset d720895 in flex_extract.git for Documentation/html/Installation/local.html


Ignore:
Timestamp:
Dec 19, 2019, 8:12:11 PM (4 years ago)
Author:
Anne Philipp <anne.philipp@…>
Branches:
master, ctbto, dev
Children:
e70c85f
Parents:
76c37a9
Message:

updated online docu

File:
1 edited

Legend:

Unmodified
Added
Removed

  • Documentation/html/Installation/local.html

    r30f7911 rd720895  
    9494<li class="toctree-l2"><a class="reference internal" href="../installation.html#licence-agreement-for-public-datasets">Licence agreement for public datasets</a></li>
    9595<li class="toctree-l2"><a class="reference internal" href="../installation.html#download-flex-extract">Download <code class="docutils literal notranslate"><span class="pre">flex_extract</span></code></a></li>
    96 <li class="toctree-l2"><a class="reference internal" href="../installation.html#environment-requirements">Environment requirements</a></li>
     96<li class="toctree-l2"><a class="reference internal" href="../installation.html#dependencies">Dependencies</a></li>
    9797<li class="toctree-l2 current"><a class="reference internal" href="../installation.html#installation-of-flex-extract">Installation of <code class="docutils literal notranslate"><span class="pre">flex_extract</span></code></a><ul class="current">
    9898<li class="toctree-l3"><a class="reference internal" href="remote.html">Remote mode installation</a></li>
    9999<li class="toctree-l3"><a class="reference internal" href="gateway.html">Gateway mode installation</a></li>
    100100<li class="toctree-l3 current"><a class="current reference internal" href="#">Local mode installation</a><ul>
    101 <li class="toctree-l4"><a class="reference internal" href="#local-environment-requirements">Local environment requirements</a></li>
     101<li class="toctree-l4"><a class="reference internal" href="#local-mode-dependencies">Local mode - dependencies</a></li>
    102102<li class="toctree-l4"><a class="reference internal" href="#prepare-local-environment">Prepare local environment</a></li>
    103103<li class="toctree-l4"><a class="reference internal" href="#test-local-environment">Test local environment</a></li>
    … …  
    190190<div class="toctree-wrapper compound">
    191191</div>
    192 <div class="section" id="local-environment-requirements">
    193 <span id="ref-req-local"></span><span id="ref-local-mode"></span><h2>Local environment requirements<a class="headerlink" href="#local-environment-requirements" title="Permalink to this headline">¶</a></h2>
    194 <p>For the local access modes <strong>member</strong> and <strong>public</strong> there is no difference in
    195 the installation process.</p>
     192<div class="section" id="local-mode-dependencies">
     193<span id="ref-req-local"></span><span id="ref-local-mode"></span><h2>Local mode - dependencies<a class="headerlink" href="#local-mode-dependencies" title="Permalink to this headline">¶</a></h2>
     194<p>The installation is the same for the access modes <strong>member</strong> and <strong>public</strong>.</p>
    196195<p>The environment on your local system has to provide these software packages
    197 and libraries, since all extraction and preparation is done at the local system:</p>
     196and libraries, since the preparation of the extraction and the post-processing is done on the local machine:</p>
    198197<table border="1" class="docutils">
    199198<colgroup>
    … …  
    227226<div class="section" id="prepare-local-environment">
    228227<span id="ref-prep-local"></span><h2>Prepare local environment<a class="headerlink" href="#prepare-local-environment" title="Permalink to this headline">¶</a></h2>
    229 <p>The easiest way to install all required packages is to use the package management system of your Linux distribution. To do so, it is necessary to use a user with admin rights.
    230 The installation was tested on a <em>Debian GNU/Linux buster/sid</em> and an <em>Ubuntu 18.04 Bionic Beaver</em> system.</p>
    231 <div class="highlight-sh notranslate"><div class="highlight"><pre><span></span><span class="c1"># On a Linux Debian or Ubuntu system do</span>
     228<p>The easiest way to install all required packages is to use the package management system of your Linux distribution  which requires admin rights.
     229The installation was tested on a <em>Debian GNU/Linux buster</em> and an <em>Ubuntu 18.04 Bionic Beaver</em> system.</p>
     230<div class="highlight-sh notranslate"><div class="highlight"><pre><span></span><span class="c1"># On a Debian or Debian-derived sytem (e. g. Ubuntu) system you may use the following commands (or equivalent commands of your preferred package manager):</span>
    232231<span class="c1"># (if not already available):</span>
    233 apt-get install python3 <span class="o">(</span>usually available on normal Linux systems<span class="o">)</span>
    234 apt-get install pip
    235 apt-get install gfortran
    236 apt-get install fftw3-dev
    237 apt-get install libeccodes-dev
    238 apt-get install libemos-dev
    239 apt-get install python3-eccodes
    240 apt-get install genshi
    241 apt-get install numpy
    242 pip install cdsapi
    243 pip install ecmwf-api-client
     232 apt-get install python3 <span class="o">(</span>usually already available on GNU/Linux systems<span class="o">)</span>
     233 apt-get install python3-eccodes
     234 apt-get install python3-genshi
     235 apt-get install python3-numpy
     236 apt-get install gfortran
     237 apt-get install fftw3-dev
     238 apt-get install libeccodes-dev
     239 apt-get install libemos-dev
     240<span class="c1"># Some of these packages will pull in further packages as dependencies. This is fine, and some are even needed by ``flex_extract&#39;&#39;.</span>
     241
     242
     243<span class="c1"># As currently the CDS and ECMWF API packages are not available as Debian packages, they need to be installed outside of the Debian (Ubuntu etc.) package management system. The recommended way is:</span>
     244 apt-get install pip
     245 pip install cdsapi
     246 pip install ecmwf-api-client
    244247</pre></div>
    245248</div>
    … …  
    253256</div>
    254257</div>
    255 <p>The CDS API (cdsapi) and the ECMWF Web API (ecmwf-api-client) have both to be installed since ERA5 can only be retrieved with the <code class="docutils literal notranslate"><span class="pre">CDS</span> <span class="pre">API</span></code> and all other datasets with the <code class="docutils literal notranslate"><span class="pre">ECMWF</span> <span class="pre">Web</span> <span class="pre">API</span></code>.</p>
     258<p>The CDS API (cdsapi) is required for ERA5 data and the ECMWF Web API (ecmwf-api-client) for all other public datasets.</p>
    256259<div class="admonition note">
    257260<p class="first admonition-title">Note</p>
    258261<p class="last">Since <strong>public users</strong> currently don’t have access to the full <em>ERA5</em> dataset they can skip the installation of the <code class="docutils literal notranslate"><span class="pre">CDS</span> <span class="pre">API</span></code>.</p>
    259262</div>
    260 <p>Both user groups have to provide key’s with their credentials for the Web API’s in their home directory. Therefore, follow these instructions:</p>
     263<p>Both user groups have to provide keys with their credentials for the Web API’s in their home directory. Therefore, follow these instructions:</p>
    261264<dl class="docutils">
    262265<dt>ECMWF Web API:</dt>
    263 <dd>Go to <a class="reference external" href="https://confluence.ecmwf.int//display/WEBAPI/Access+MARS">MARS access</a> website and log in with your credentials. Afterwards, on this site in section “Install ECMWF KEY” the key for the ECMWF Web API should be listed. Please follow the instructions in this section under point 1 (save the key in a file <cite>.ecmwfapirc</cite> in your home directory).</dd>
     266<dd>Go to <a class="reference external" href="https://confluence.ecmwf.int//display/WEBAPI/Access+MARS">MARS access</a> website and log in with your credentials. Afterwards, on this site in section “Install ECMWF KEY” the key for the ECMWF Web API should be listed. Please follow the instructions in this section under 1 (save the key in a file <cite>.ecmwfapirc</cite> in your home directory).</dd>
    264267<dt>CDS API:</dt>
    265268<dd>Go to ‘CDS API registration’_ and register there too. Log in at the <a class="reference external" href="https://cds.climate.copernicus.eu/api-how-to">cdsapi</a> website and follow the instructions at section “Install the CDS API key” to save your credentials in a <cite>.cdsapirc</cite> file.</dd>
    … …  
    268271<div class="section" id="test-local-environment">
    269272<span id="ref-test-local"></span><h2>Test local environment<a class="headerlink" href="#test-local-environment" title="Permalink to this headline">¶</a></h2>
    270 <p>Check the availability of the system packages with <code class="docutils literal notranslate"><span class="pre">dpkg</span> <span class="pre">-s</span> <span class="pre">&lt;package-name&gt;</span> <span class="pre">|</span>&#160; <span class="pre">grep</span> <span class="pre">Status</span></code> or <code class="docutils literal notranslate"><span class="pre">rpm</span> <span class="pre">-q</span> <span class="pre">&lt;package_name&gt;</span></code>, depending on your system. For example:</p>
    271 <div class="highlight-sh notranslate"><div class="highlight"><pre><span></span>$ dpkg -s libeccodes-dev <span class="p">|</span>  grep Status
    272 <span class="c1"># or</span>
    273 $ rpm -q libeccodes-dev
    274 </pre></div>
    275 </div>
    276 <p>Afterwards, check the availability of the python packages by typing <code class="docutils literal notranslate"><span class="pre">python3</span></code> in
    277 a terminal window and run the <code class="docutils literal notranslate"><span class="pre">import</span></code> commands in the python shell. If there are no error messages, you succeeded in setting up the environment.</p>
     273<p>Check the availability of the python packages by typing <code class="docutils literal notranslate"><span class="pre">python3</span></code> in a terminal window and run the <code class="docutils literal notranslate"><span class="pre">import</span></code> commands in the python shell. If there are no error messages, you succeeded in setting up the environment.</p>
    278274<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="c1"># check in python3 console</span>
    279275<span class="kn">import</span> <span class="nn">eccodes</span>
    … …  
    286282<div class="section" id="test-the-web-api-s">
    287283<h3>Test the Web API’s<a class="headerlink" href="#test-the-web-api-s" title="Permalink to this headline">¶</a></h3>
    288 <p>You can start very simple test retrievals for both Web API’s to be sure that everything works. This is recommended to minimize the range of possible errors using <code class="docutils literal notranslate"><span class="pre">flex_extract</span></code> later on.</p>
     284<p>You can start very simple test retrievals for both Web APIs to be sure that everything works. This is recommended to minimise the range of possible errors using <code class="docutils literal notranslate"><span class="pre">flex_extract</span></code> later on.</p>
    289285<div class="section" id="ecmwf-web-api">
    290286<h4>ECMWF Web API<a class="headerlink" href="#ecmwf-web-api" title="Permalink to this headline">¶</a></h4>
    … …  
    295291</colgroup>
    296292<tbody valign="top">
    297 <tr class="row-odd"><td>Please use this piece of python code for <strong>Member user</strong>:</td>
    298 <td>Please use this piece of python code for <strong>Public user</strong>:</td>
     293<tr class="row-odd"><td>Please use this piece of Python code for <strong>Member user</strong>:</td>
     294<td>Please use this piece of Python code for <strong>Public user</strong>:</td>
    299295</tr>
    300296<tr class="row-even"><td><div class="first last highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">ecmwfapi</span> <span class="kn">import</span> <span class="n">ECMWFService</span>
    … …  
    344340<div class="section" id="cds-api">
    345341<h4>CDS API<a class="headerlink" href="#cds-api" title="Permalink to this headline">¶</a></h4>
    346 <p>Since ERA5 extraction with CDS API might take some time due to the very high number of requests, you can start by retrieving some online stored pressure levels (not from MARS). This is usually much faster and gives a quick result to find out if the web API works:</p>
    347 <p>Please use this piece of python code to retrieve a small sample of <em>ERA5</em> pressure levels:</p>
     342<p>Extraction of ERA5 data via CDS API might take time as currently there is a high demand for ERA5 data. Therefore, as a simple test for the API just retrieve pressure-level data (even if that is NOT what we need for FLEXPART), as they are stored on disk and don’t need to be retrieved from MARS (which is the time-consuming action):</p>
     343<p>Please use this piece of Python code to retrieve a small sample of <em>ERA5</em> pressure levels:</p>
    348344<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">cdsapi</span>
    349345
    … …  
    364360</pre></div>
    365361</div>
    366 <p>Afterwards, when you know that the CDS API generally works, you can try to extract some
    367 data from the MARS archive. From the latest experience we know that this can take a while.</p>
    368 <p>Please use this piece of python code to retrieve a small <em>ERA5</em> data sample as a <strong>member user</strong>! The <strong>public user</strong> doesn’t have access to the full <em>ERA5</em> dataset!</p>
     362<p>If you know that your CDS API works, you can try to extract some data from MARS.</p>
     363<p>Please use this piece of Python code to retrieve a small <em>ERA5</em> data sample as a <strong>member-state user</strong>! The <strong>Public user</strong> do not have access to the full <em>ERA5</em> dataset!</p>
    369364<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">cdsapi</span>
    370365
    … …  
    394389<div class="section" id="local-installation">
    395390<span id="ref-install-local"></span><h2>Local installation<a class="headerlink" href="#local-installation" title="Permalink to this headline">¶</a></h2>
    396 <p>First prepare the Fortran <code class="docutils literal notranslate"><span class="pre">Makefile</span></code> for your environment and set it
    397 in the <code class="docutils literal notranslate"><span class="pre">setup.sh</span></code> script. (See section <span class="xref std std-ref">ref-convert</span> for information on
    398 the Fortran program.)
    399 <code class="docutils literal notranslate"><span class="pre">flex_extract</span></code> has already two <code class="docutils literal notranslate"><span class="pre">Makefiles</span></code> prepared for te <code class="docutils literal notranslate"><span class="pre">gfortran</span></code> and
    400 the <code class="docutils literal notranslate"><span class="pre">ifort</span></code> compiler:</p>
     391<p>First prepare the Fortran <code class="docutils literal notranslate"><span class="pre">makefile</span></code> for your environment and set it in the <code class="docutils literal notranslate"><span class="pre">setup.sh</span></code> script. (See section <span class="xref std std-ref">ref-convert</span> for information on the Fortran program.)
     392<code class="docutils literal notranslate"><span class="pre">flex_extract</span></code> comes with two <code class="docutils literal notranslate"><span class="pre">makefiles</span></code> prepared for the <code class="docutils literal notranslate"><span class="pre">gfortran</span></code> and
     393the <code class="docutils literal notranslate"><span class="pre">ifort</span></code> compiler. The <code class="docutils literal notranslate"><span class="pre">gfortran</span></code> version assumes that <code class="docutils literal notranslate"><span class="pre">eccodes</span></code> and <code class="docutils literal notranslate"><span class="pre">emoslib</span></code> are installed as distribution packages.</p>
    401394<blockquote>
    402395<div><ul class="simple">
    403 <li>Makefile.local.gfortran</li>
    404 <li>Makefile.local.ifort</li>
     396<li>makefile.local.gfortran</li>
     397<li>makefile.local.ifort</li>
    405398</ul>
    406399</div></blockquote>
    407 <p>They can be found in the path <code class="docutils literal notranslate"><span class="pre">flex_extract_vX.X/source/fortran</span></code>, where
     400<p>They can be found at <code class="docutils literal notranslate"><span class="pre">flex_extract_vX.X/Source/Fortran</span></code>, where
    408401<code class="docutils literal notranslate"><span class="pre">vX.X</span></code> should be substituted with the current version number.</p>
    409402<div class="admonition caution">
    410403<p class="first admonition-title">Caution</p>
    411404<p class="last">It is necessary to adapt <strong>ECCODES_INCLUDE_DIR</strong> and <strong>ECCODES_LIB</strong> in these
    412 <code class="docutils literal notranslate"><span class="pre">Makefiles</span></code>.</p>
     405<code class="docutils literal notranslate"><span class="pre">makefiles</span></code> if other than standard paths are used.</p>
    413406</div>
    414407<p>So starting from the root directory of <code class="docutils literal notranslate"><span class="pre">flex_extract</span></code>,
    415 go to the <code class="docutils literal notranslate"><span class="pre">Fortran</span></code> source directory and open the <code class="docutils literal notranslate"><span class="pre">Makefile</span></code> of your
     408go to the <code class="docutils literal notranslate"><span class="pre">Fortran</span></code> source directory and open the <code class="docutils literal notranslate"><span class="pre">makefile</span></code> of your
    416409choice to modify with an editor of your choice. We use the <code class="docutils literal notranslate"><span class="pre">nedit</span></code> in this case.</p>
    417 <div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="nb">cd</span> flex_extract_vX.X/source/fortran
    418 nedit Makefile.local.gfortran
    419 </pre></div>
    420 </div>
    421 <p>Edit the pathes to the <code class="docutils literal notranslate"><span class="pre">eccodes</span></code> library on your local machine.</p>
     410<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="nb">cd</span> flex_extract_vX.X/Source/Fortran
     411nedit makefile.local.gfortran
     412</pre></div>
     413</div>
     414<p>Edit the paths to the <code class="docutils literal notranslate"><span class="pre">eccodes</span></code> library on your local machine.</p>
    422415<div class="admonition caution">
    423416<p class="first admonition-title">Caution</p>
    … …  
    432425<p class="last">to find out the path to the <code class="docutils literal notranslate"><span class="pre">eccodes</span></code> library.</p>
    433426</div>
    434 <p>Substitute these paths in the <code class="docutils literal notranslate"><span class="pre">Makefile</span></code> for parameters <strong>ECCODES_INCLUDE_DIR</strong>
     427<p>Substitute these paths in the <code class="docutils literal notranslate"><span class="pre">makefile</span></code> for parameters <strong>ECCODES_INCLUDE_DIR</strong>
    435428and <strong>ECCODES_LIB</strong> and save it.</p>
    436429<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># these are the paths on a current Debian 10 Testing system (May 2019)</span>
    … …  
    440433</div>
    441434<p>The Fortran program called <code class="docutils literal notranslate"><span class="pre">CONVERT2</span></code> will be compiled during the
    442 installation process to get an executable. Therefore the <code class="docutils literal notranslate"><span class="pre">Makefile</span></code>
    443 has to be set in the <code class="docutils literal notranslate"><span class="pre">setup.sh</span></code> script.</p>
     435installation process.Therefore the name of the <code class="docutils literal notranslate"><span class="pre">makefile</span></code> to be used needs to be given in  <code class="docutils literal notranslate"><span class="pre">setup.sh</span></code>.</p>
    444436<p>In the root directory of <code class="docutils literal notranslate"><span class="pre">flex_extract</span></code>, open the <code class="docutils literal notranslate"><span class="pre">setup.sh</span></code> script
    445437and adapt the installation parameters in the section labelled with
    … …  
    454446<span class="c1">#</span>
    455447<span class="nv">TARGET</span><span class="o">=</span><span class="s1">&#39;local&#39;</span>
    456 <span class="nv">MAKEFILE</span><span class="o">=</span><span class="s1">&#39;Makefile.local.gfortran&#39;</span>
     448<span class="nv">MAKEFILE</span><span class="o">=</span><span class="s1">&#39;makefile.local.gfortran&#39;</span>
    457449<span class="nv">ECUID</span><span class="o">=</span>None
    458450<span class="nv">ECGID</span><span class="o">=</span>None
    … …  
    470462</pre></div>
    471463</div>
    472 <p>to start the installation. You should see the following output at the command line.</p>
     464<p>to start the installation. You should see the following standard output.</p>
    473465<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># Output of setup.sh</span>
    474466WARNING: installdir has not been specified
Note: See TracChangeset for help on using the changeset viewer.
hosted by ZAMG