- Timestamp:
- Oct 4, 2018, 7:18:23 AM (6 years ago)
- Branches:
- master, ctbto, dev
- Children:
- aa1af8c
- Parents:
- 27d9d3c
- File:
-
- 1 edited
Legend:
- Unmodified
- Added
- Removed
-
python/README.md
r70166b9 r5d3d09b 1 #README # 2 3 This documentation shows how to use these python scripts to extract ECMWF data and generate the ***FLEXPART*** specific input files. 4 5 ### Introduction ### 6 7 To run ***FLEXPART*** with ECMWF data, you first need to retrieve ECMWF GRIB fields and generate ***FLEXPART*** specific input files afterwards. ***FLEXPART*** needs the GRIB data seperated by time so that flex_extract will combine all data in output files seperated for each time step. The format of these output filenames looks like: `prefixYYMMDDHH`, where prefix can be defined in ***flex_extract*** (usually it should be limited to "2" characters because of further processing reasons). 1 #README 2 3 This documentation shows how to install and use the ***flex_extract*** software package to extract ECMWF data and generate the ***FLEXPART*** specific input files. 4 5 Introduction 6 ================= 7 8 To run ***FLEXPART*** with ECMWF data, you first need to retrieve ECMWF GRIB fields and generate ***FLEXPART*** specific input files afterwards. ***FLEXPART*** needs the GRIB data seperated by time so that ***flex_extract*** will combine all data in output files seperated for each time step. The format of these output filenames looks like: `prefixYYMMDDHH`, where prefix can be defined in ***flex_extract*** (usually it should be limited to "2" characters because of further processing reasons). 9 10 There are 4 different modes of running ***flex_extract*** : 11 1. Remote (**): ***flex_extract*** is installed and executed directly on the ECMWF Linux member state server ecgate. Users have to be a registered member-state user. 12 2. Gateway (**): ***flex_extract*** is installed on a local machine and the scripts are submitted to ECMWF 13 computer facilities via a member-state gateway server. Users have to be a registered member-state user. 14 3. Local member state (**): ***flex_extract*** is installed and executed on a local machine. Users have to be a registered member-state user. 15 4. Local public (***): ***flex_extract*** is installed and executed on a local machine. Users do not have to be a member-state user. A normal registration at the ECMWF website is enough. See also the instruction to install the ECMWF Web API 16 17 *(\*\*) To get a ms (member-state) user access, users have to be a resident of a [member state](https://www.ecmwf.int/en/about/who-we-are/member-states) and have to contact their national meteorological center for granting access.* 18 19 *(\*\*\*) To get a public user account, the user can just register at the ECMWF website following the instructions [here](https://software.ecmwf.int/wiki/display/WEBAPI/ECMWF+Web+API+Home)!* 8 20 9 21 For setting up a complete ***FLEXPART*** environment it is recommended to build a directory structure such as: 10 22 11 ![FLEXPART directory structure](../docs/Fp_extract_strukt -pngnew.png)23 ![FLEXPART directory structure](../docs/Fp_extract_strukt.png) 12 24 13 25 For installation instructions and usage of ***FLEXPART*** please see ***FLEXPART***-Documentation! 14 26 15 ### Overview ### 16 17 %-- list all files in python dir and shortly explain structure 18 19 ##### CONTROL file ##### 20 For specifying which ECMWF dataset you would like to retrieve and which time and spatial resolution you would like to retrieve you have to prepare a `CONTROL` file. In the directory you can find a couple of example `CONTROL` files: 27 Data sets archived at ECMWF 28 =========================== 29 30 There are two modes for accessing ECMWF data sets: the **_full access_** and the **_public access_** mode. 31 A description of these two modes and links to the available data sets is available here: https://www.ecmwf.int/en/forecasts/datasets/archive-datasets 32 33 #### Important note for public users 34 35 If you want to extract public data sets from the ECMWF servers you first have to register at the ECMWF. Do this by following the instructions at: https://software.ecmwf.int/wiki/display/WEBAPI/Access+MARS 36 37 Afterwards you **_need to accept the licenses_** of the data sets you want to extract: 38 39 https://software.ecmwf.int/wiki/display/WEBAPI/Available+ECMWF+Public+Datasets 40 41 Now your able to use ***flex_extract*** in public mode (command line parameter "*--public=1*"). 42 43 Installation 44 =================== 45 46 ### Some important notes for the installation process: 47 - Use the same compiler and compiler version for all libraries and the fortran program. 48 - Don't forget to set all library paths to the LD_LIBRARY_PATH variable. 49 - Don't forget to adapt the makefile for the Fortran program. 50 51 ### Requirements ### 52 53 | Python Support | Link | Notes | 54 | --------------------- |:----------------------:|:-------------------------| 55 | python | [http://www.python.org](http://www.python.org) | We have used [Anaconda python](https://store.continuum.io/cshop/anaconda/) for our testing | 56 | python-numpy | [http://www.numpy.org/](http://www.numpy.org/) | Not necessary if you have installed python Anaconda 57 | ecmwfapi | [https://software.ecmwf.int/wiki/display/WEBAPI/ECMWF+Web+API+Home](https://software.ecmwf.int/wiki/display/WEBAPI/ECMWF+Web+API+Home) | You also need to install your API key (as explained in the documentation) 58 59 | Utilities | Link | Notes | 60 | --------------------- |:----------------------:|:-------------------------| 61 |gfortran|[https://gcc.gnu.org/wiki/GFortran](https://gcc.gnu.org/wiki/GFortran)| Make sure that you are using the same compiler version for all libraries in connection with flex_extract, GRIB_API and FLEXPART, otherwise you will have problems.| 62 | grib-api or eccodes | [https://software.ecmwf.int/wiki/display/GRIB/Home](https://software.ecmwf.int/wiki/display/GRIB/Home) / [https://software.ecmwf.int/wiki/display/ECC](https://software.ecmwf.int/wiki/display/ECC) | Make sure you install grib_api/eccodes with JPEG support and python.| 63 |FFTW|[http://www.fftw.org](http://www.fftw.org)|This is needed by the Emos library, as is mentioned in the installation instructions for Emos| 64 | Emos | [https://software.ecmwf.int/wiki/display/EMOS/Emoslib](https://software.ecmwf.int/wiki/display/EMOS/Emoslib) | Make sure you carefully read the instructions and have installed the FFTW package before installing Emos.| 65 | CONVERT2 | |to run prepareFLEXPART.py, you need to compile the Fortran program CONVERT2 (located in the src directory). See separate README file in this directory to get instructions on how to compile this code.| 66 67 ### Prepare the environment for using ***flex_extract*** on a local machine 68 69 Please follow the instructions on the websites of the libraries. The installation of these libraries can be done by package installation or from scratch, for either case it is useful to consider the official documentation (and also the hints from the ECMWF) rather than trying to explain every detail in here. Nevertheless, we recommend doing the installation in the following order (see links to the libraries in the table above): 70 71 1. Read Emoslib installation instructions. 72 2. Read ECMWF blog about gfortran (https://software.ecmwf.int/wiki/display/SUP/2015/05/11/Building+ECMWF+software+with+gfortran). 73 3. Install FFTW. 74 4. Install EMOSLIB (Hint: Apply make 2 times! One time without any options and one time with single precision option.). 75 5. Install ECCODES OR GRIB_API (for python and fortran). ECCODES is downward compatible with GRIB_API and GRIB_API support will expire at the end of 2018. 76 6. Register at Mars (see link to ecmwfapi). 77 7. Install ECMWF_WEBAPI. 78 8. Check LD_LIBRARY_PATH environment variable if it contains all paths to the libs. 79 9. Check available python packages (e.g. check in python console the commands: import eccodes / import grib_api / import ecmwfapi) 80 10. Start a simple test retrieval (following the instructions at ECMWF Web API website). 81 11. Install ***flex_extract*** (see next section). 82 83 ### Installation of ***flex_extract*** 84 85 To install ***flex_extract*** you have to go to the python directory and call the `install.py` script. It has the following options: 86 87 ``` 88 usage: install.py [-h] [--target INSTALL_TARGET] [--makefile MAKEFILE] 89 [--ecuid ECUID] [--ecgid ECGID] [--gateway GATEWAY] 90 [--destination DESTINATION] 91 [--flexpart_root_scripts FLEXPART_ROOT_SCRIPTS] 92 [--job_template JOB_TEMPLATE] [--controlfile CONTROLFILE] 93 94 Install flex_extract software locally or on ECMWF machines 95 96 optional arguments: 97 -h, --help show this help message and exit 98 --target INSTALL_TARGET 99 Valid targets: local | ecgate | cca , the latter two 100 are at ECMWF (default: None) 101 --makefile MAKEFILE Name of Makefile to use for compiling CONVERT2 102 (default: None) 103 --ecuid ECUID user id at ECMWF (default: None) 104 --ecgid ECGID group id at ECMWF (default: None) 105 --gateway GATEWAY name of local gateway server (default: None) 106 --destination DESTINATION 107 ecaccess destination, e.g. leo@genericSftp (default: 108 None) 109 --flexpart_root_scripts FLEXPART_ROOT_SCRIPTS 110 FLEXPART root directory on ECMWF servers (to find 111 grib2flexpart and COMMAND file) Normally flex_extract 112 resides in the scripts directory of the FLEXPART 113 distribution, thus the: (default: None) 114 --job_template JOB_TEMPLATE 115 job template file for submission to ECMWF (default: 116 job.temp.o) 117 --controlfile CONTROLFILE 118 file with control parameters (default: CONTROL.temp) 119 ``` 120 121 122 #### Local installation of ***flex_extract*** (***flex_extract*** modes 1 (on ECMWF servers), 3 and 4) 123 124 --- 125 126 For a local installation of ***flex_extract*** the user has to adapt the makefile for the Fortran program first. Set the paths to the grib_api or eccodes library and emoslib. Then the installation script should be called as follows: 127 128 ``` 129 ./install.py --target=local --makefile=Makefile.local.gfortan 130 ``` 131 132 where the makefile name can differ if the user prepared another makefile for his environment. 133 On ECMWF servers a makefile named "Makefile.gfortran" is already provided in the "src"-directory. It is also the default makefile, so that the parameter does not have to be set when calling "install.py". 134 135 If the ***flex_extract*** is to be installed within a ***FLEXPART*** environment it should be called with the parameter `flexpart_root_scripts` to let ***flex_extract*** know about the path to ***FLEXPART***. 136 137 The installation was successful if the compilation of the Fortran program ``CONVERT2`` didn't fail and is displayed in the terminal. 138 139 #### Installation using a gateway server (***flex_extract*** mode 2 only) 140 141 --- 142 143 If the user has a gateway server and the Ecaccess toolkit available, ***flex_extract*** can be installed such that it is executed on the local machine and a job script is submitted to the ECMWF server. The final ***FLEXPART*** input files are then send back via the Ecaccess command line tool "ectrans". To use this tool, the user has to create an association directly on the local gateway server via the localhost ecaccess.ecmwf.int and provide the association name as the "destination"-parameter when calling "install.py". 144 145 The gateway server has to be registered at ECMWF. For information about a gateway server see: 146 - https://software.ecmwf.int/wiki/display/ECAC/Releases+-+Gateway+package 147 - and the Software Installation Plan (SIP.pdf) in the docs/ directory. 148 149 Afterwards ***flex_extract*** can be installed with the following command: 150 ``` 151 ./install.py --target=local --makefile=Makefile.gfortan --ecuid=<id> --ecgid=<gid> --gateway=<servername> --destination=<destinationname> 152 ``` 153 154 If the installation was successful the user gets an email with a success note! 155 156 157 Using ***flex_extract*** 158 =================== 159 160 This section provides the absolut basics on how ***flex_extract*** is to be used. For more information about possible settings, performance, parameter combinations and so one, the user is referred to the SIP.pdf and SUT.pdf documents and the www.flexpart.eu community website. It is also possible to ask for help by writing to the ***FLEXPART*** user email list (registration needed) or by creating a ticket on the community website. 161 162 #### CONTROL file 163 164 ***flex_extract*** is controlled by providing "CONTROL"-files with a list of parameter settings. These parameter are described in detail in the *Software User Tutorial (SUT.pdf)*. 165 The "CONTROL"-files specifies which ECMWF data set is to be retrieved, the time and spatial resolution, the format of the grib file and many more. In the `python` directory are some example `CONTROL` files for the different data sets and access modes which can be retrieved with ***flex_extract***: 21 166 22 167 ``` … … 38 183 ``` 39 184 40 #### Python scripts #### 41 42 The main program for doing both, retrieving ECMWF data and generate the FLEXPART input files, is `submit.py`. It combines the two sub-programs `getMARSdata.py` and `prepareFLEXPART.py` which can also be run by themselves for example for debugging purposes. 43 To get the usage of the program, use the `-h` option: 185 For information about all the possible parameter settings and explanation, please see the SUT.pdf document in the `docs/` directory. 186 187 `CONTROL`-files with a `.public` ending is usable for the public access mode. The main difference is the parameter ``dataset`` which explicitly specifies the public data sets. Additionally, not all meteorological fields and times were archived in the public data sets and are considered in the public `CONTROL`-files. 188 189 #### Python scripts 190 191 * `submit.py` 192 193 This is the main program for doing both, retrieving ECMWF data and generating the ***FLEXPART*** input files. It combines the two sub-programs `getMARSdata.py` and `prepareFLEXPART.py` which can also be run by themselves for debugging purposes. 194 Use the `-h` option to get the parameter options: 44 195 45 196 ``` … … 88 239 --debug DEBUG Debug mode - leave temporary files intact (default: 0) 89 240 --public PUBLIC Public mode - retrieves the public datasets (default: 90 False) 91 ``` 92 93 94 Optional arguments are listed in squared brackets. 95 96 The sub-programs `getMARSdata.py` and `prepareFLEXPART.py` are the two main parts of this software. 241 0) 242 ``` 243 244 *Optional arguments are listed in squared brackets.* 245 97 246 98 247 * `getMARSdata.py` … … 103 252 * `prepareFLEXPART.py` 104 253 105 This program generates FLEXPART input files. It requires python interface to grib_api and the Fortran program `CONVERT2` (located in `src` directory with instruction on how to compile it). `CONVERT2` needs a namelist which is generated from the python program. 106 107 108 ### Requirements ### 109 110 | Python Support | Link | Notes | 111 | --------------------- |:----------------------:|:-------------------------| 112 | python | [http://www.python.org](http://www.python.org) | We have used [Anaconda python](https://store.continuum.io/cshop/anaconda/) for our testing | 113 | python-numpy | [http://www.numpy.org/](http://www.numpy.org/) | Not necessary if you have installed python Anaconda 114 | ecmwfapi | [https://software.ecmwf.int/wiki/display/WEBAPI/ECMWF+Web+API+Home](https://software.ecmwf.int/wiki/display/WEBAPI/ECMWF+Web+API+Home) | You also need to install your API key (as explained in the documentation) 115 116 | Utilities | Link | Notes | 117 | --------------------- |:----------------------:|:-------------------------| 118 |gfortran|[https://gcc.gnu.org/wiki/GFortran](https://gcc.gnu.org/wiki/GFortran)| Make sure that you are using the same compiler version for all libraries in connection with flex_extract, GRIB_API and FLEXPART, otherwise you will have problems.| 119 | grib-api | [https://software.ecmwf.int/wiki/display/GRIB/Home](https://software.ecmwf.int/wiki/display/GRIB/Home) | Make sure you install GRIB-API with JPEG support and python GRIB-API.| 120 |FFTW|[http://www.fftw.org](http://www.fftw.org)|This is needed by the Emos library, as is mentioned in the installation instructions for Emos| 121 | Emos | [https://software.ecmwf.int/wiki/display/EMOS/Emoslib](https://software.ecmwf.int/wiki/display/EMOS/Emoslib) | Make sure you carefully read the instructions and have installed the FFTW package before installing Emos.| 122 | CONVERT2 | |to run prepareFLEXPART.py, you need to compile the Fortragn program CONVERT2 (located in the src directory). See separate README file in this directory to get instructions on how to compile this code.| 123 124 ### Installation ### 125 126 * Environment 127 128 At UIO, Red Hat 6 Linux systems (64 bits) were used for testing. We use the [Module package](http://modules.sourceforge.net/) to set-up user environment. 129 130 * Getting the source code 131 In the directory of your choice: 132 133 ` 134 git clone git@bitbucket.org:flexpart/flexpart.git 135 ` 136 This command will create a subdirectory called flexpart: it contains the latest FLEXPART version. 137 138 Then set the environment variable `FLEXPART_ROOT`: 139 140 ** Korn-shell or Bash users: 141 ` 142 cd flexpart 143 export FLEXPART_ROOT=$PWD 144 ` 145 146 ** C-shell users: 147 148 ` 149 cd flexpart 150 setenv FLEXPART_ROOT=$PWD 151 ` 152 153 * Installation 154 155 Make sure you first generate the `CONVERT2` program (see separate instructions in `preproc/src`). 156 157 Users need to be able to execute prepareFLEXPART.py and getMARSdata.py so make sure they have the correct unix permissions: 158 159 ` 160 cd preproc/python 161 chmod uog+rx getMARSdata.py prepareFLEXPART.py 162 ` 163 164 These two programs must be in the user PATH. At UIO this is done automatically when loading flexpart. If not, you would need to do the following: 165 166 ** Korn-shell or Bash users: 167 ` 168 export PATH=$FLEXPART_HOME/preproc/python:$PATH 169 ` 170 171 ** C-shell users: 172 173 ` 174 setenv PATH $FLEXPART_HOME/preproc/python:$PATH 175 ` 176 177 Where `$FLEXPART_HOME` is the directory where FLEXPART 178 179 * Testing your installation 180 181 First check that grib-api python interface is correctly installed on your platform: 182 ` 183 python 184 >>> from gribapi import * 185 >>> 186 ` 187 Use `CTRL-D` to quit python. 188 189 Then check that `ecmwfapi` is properly installed: 190 ` 191 python 192 >>> from ecmwfapi import * 193 >>> 194 ` 195 196 If the two previous tests were successful, you can run `tests/preproc` (See separate instructions in `tests/preproc`). 197 198 If any of these two tests fail, this probably means that either `ecmwfapi` or `grib-api` have not been installed properly. 199 200 Please report any problems. 201 202 ### Installation FAQ ### 203 204 205 206 254 This program generates FLEXPART input files from the retrieved meteorological fields. It requires python interface to grib_api or eccodes and the Fortran program `CONVERT2` (located in `src` directory). `CONVERT2` needs a namelist which is generated from the python program. 255 256 #### execute flex_extract locally (member-state user) 257 258 Almost every command line parameter for the `submit.py` script call has a default value or is read from the specified `CONTROL` file. 259 Therefore, for a local extraction of ERA-Interim data as a member-state user the script can be called simply by passing a `CONTROL` file and a starting date with the following command: 260 ``` 261 ./submit.py --controlfile=CONTROL_EI.global --start_date=20101201 262 ``` 263 If there was just a starting date without an end date, ***flex_extract*** retrieves just this one day of meteorological data. The `CONTROL_EI.global` file has a set of default values which needs to be adjusted regarding users needs. It is also possible to set the dates within the `CONTROL`-file instead of passing it via command line parameter. 264 265 266 #### execute flex_extract locally (public user) 267 268 As a public user one has to use the `CONTROL` file with the `.public` extension and the parameter `public`: 269 ``` 270 ./submit.py --controlfile=CONTROL_EI.public --start_date=20101201 --public=1 271 ``` 272 Otherwise, except of the different data available, the usage of ***flex_extract*** is the same as for member-state users. 273 274 #### execute flex_extract via gateway server 275 276 If ***flex_extract*** was installed via a gateway server and a file "*ECMWF_ENV*" is available in the ``python``-directory, the ``submit.py`` script can be called with the `--queue` option. A job script is then prepared and submitted to the ECMWF server (in this case, "ecgate") . 277 278 ``` 279 ./submit.py --queue=ecgate --controlfile=CONTROL_EI.global --start_date=20101201 280 ``` 281 282 The job status can then be checked by typing: 283 ```` 284 ecaccess-job-list 285 ```` 286 287 If the parameters *ecuid*, *ecgid*, *gateway* and *destination* were set correctly during installation and the gateway server is running, the resulting ***FLEXPART*** input files can be directly transferred to the local gateway by setting the paramter ``ectrans`` in the ``CONTROL``-file to ``1``. 288 289 Regardless of the job being successful or not, the user will receive an email with the log messages. 290 291 # Installation FAQ 292 293 294 295
Note: See TracChangeset
for help on using the changeset viewer.