Changeset 0b00607 in flex_extract.git for for_developers/Sphinx/source/Documentation/output.rst
- Timestamp:
- Jul 29, 2019, 8:23:57 AM (5 years ago)
- Branches:
- master, ctbto, dev
- Children:
- bc27d19
- Parents:
- 41c9dbc
- File:
-
- 1 edited
Legend:
- Unmodified
- Added
- Removed
-
for_developers/Sphinx/source/Documentation/output.rst
r41c9dbc r0b00607 1 =========== 1 *********** 2 2 Output Data 3 =========== 4 5 UNDER CONSTRUCTION 3 *********** 4 5 The output data of ``flex_extract`` are separated mainly into temporary files and the final ``FLEXPART`` input files: 6 7 +-----------------------------------------------+----------------------------------------------+ 8 | ``FLEXPART`` input files | Temporary files (saved in debug mode) | 9 +-----------------------------------------------+----------------------------------------------+ 10 | - Standard output filenames | - MARS request file (opt) | 11 | - Output for pure forecast | - flux files | 12 | - Output for ensemble members | - VERTICAL.EC | 13 | - Output for new precip. disaggregation | - index file | 14 | | - fort files | 15 | | - MARS grib files | 16 +-----------------------------------------------+----------------------------------------------+ 17 18 19 20 ``FLEXPART`` input files 21 ======================== 22 23 The final output files of ``flex_extract`` are also the meteorological ``FLEXPART`` input files. 24 The naming of these files depend on the kind of data extracted by ``flex_extract``. 25 26 Standard output files 27 --------------------- 28 29 In general, there is a file for each time step with the filename format: 30 31 .. code-block:: bash 32 33 <prefix>YYMMDDHH 34 35 The ``prefix`` is by default defined as ``EN`` and can be re-defined in the ``CONTROL`` file. 36 Each file contains all meteorological fields needed by ``FLEXPART`` for all selected model levels for a specific time step. 37 38 Here is an example output which lists the meteorological fields in a single file called ``CE00010800`` where we extracted only the lowest model level for demonstration reasons: 39 40 .. code-block:: bash 41 42 $ grib_ls CE00010800 43 44 edition centre date dataType gridType stepRange typeOfLevel level shortName packingType 45 2 ecmf 20000108 an regular_ll 0 hybrid 91 u grid_simple 46 2 ecmf 20000108 an regular_ll 0 hybrid 91 v grid_simple 47 2 ecmf 20000108 an regular_ll 0 hybrid 91 etadot grid_simple 48 2 ecmf 20000108 an regular_ll 0 hybrid 91 t grid_simple 49 2 ecmf 20000108 an regular_ll 0 surface 1 sp grid_simple 50 2 ecmf 20000108 an regular_ll 0 hybrid 91 q grid_simple 51 2 ecmf 20000108 an regular_ll 0 hybrid 91 qc grid_simple 52 1 ecmf 20000108 fc regular_ll 0 surface 0 sshf grid_simple 53 1 ecmf 20000108 fc regular_ll 0 surface 0 ewss grid_simple 54 1 ecmf 20000108 fc regular_ll 0 surface 0 nsss grid_simple 55 1 ecmf 20000108 fc regular_ll 0 surface 0 ssr grid_simple 56 1 ecmf 20000108 fc regular_ll 0 surface 0 lsp grid_simple 57 1 ecmf 20000108 fc regular_ll 0 surface 0 cp grid_simple 58 1 ecmf 20000108 an regular_ll 0 surface 0 sd grid_simple 59 1 ecmf 20000108 an regular_ll 0 surface 0 msl grid_simple 60 1 ecmf 20000108 an regular_ll 0 surface 0 tcc grid_simple 61 1 ecmf 20000108 an regular_ll 0 surface 0 10u grid_simple 62 1 ecmf 20000108 an regular_ll 0 surface 0 10v grid_simple 63 1 ecmf 20000108 an regular_ll 0 surface 0 2t grid_simple 64 1 ecmf 20000108 an regular_ll 0 surface 0 2d grid_simple 65 1 ecmf 20000108 an regular_ll 0 surface 0 z grid_simple 66 1 ecmf 20000108 an regular_ll 0 surface 0 lsm grid_simple 67 1 ecmf 20000108 an regular_ll 0 surface 0 cvl grid_simple 68 1 ecmf 20000108 an regular_ll 0 surface 0 cvh grid_simple 69 1 ecmf 20000108 an regular_ll 0 surface 0 lcc grid_simple 70 1 ecmf 20000108 an regular_ll 0 surface 0 mcc grid_simple 71 1 ecmf 20000108 an regular_ll 0 surface 0 hcc grid_simple 72 1 ecmf 20000108 an regular_ll 0 surface 0 skt grid_simple 73 1 ecmf 20000108 an regular_ll 0 depthBelowLandLayer 0 stl1 grid_simple 74 1 ecmf 20000108 an regular_ll 0 depthBelowLandLayer 0 swvl1 grid_simple 75 1 ecmf 20000108 an regular_ll 0 surface 0 sr grid_simple 76 1 ecmf 20000108 an regular_ll 0 surface 0 sdor grid_simple 77 1 ecmf 20000108 an regular_ll 0 surface 0 cvl grid_simple 78 1 ecmf 20000108 an regular_ll 0 surface 0 cvh grid_simple 79 1 ecmf 20000108 an regular_ll 0 surface 0 fsr grid_simple 80 35 of 35 messages in CE00010800 81 82 83 Output files for pure forecast 84 ------------------------------ 85 86 ``Flex_extract`` can retrieve forecasts which can be longer than 23 hours. To avoid collisions of time steps for forecasts of more than one day a new scheme for filenames in pure forecast mode is introduced: 87 88 .. code-block:: bash 89 90 <prefix>YYMMDD.HH.<FORECAST_STEP> 91 92 The ``<prefix>`` is, as in the standard output, by default ``EN`` and can be re-defined in the ``CONTROL`` file. ``YYMMDD`` is the date format and ``HH`` the forecast time which is the starting time for the forecasts. The ``FORECAST_STEP`` is a 3 digit number which represents the forecast step in hours. 93 94 95 Output files for ensemble predictions 96 ------------------------------------- 97 98 Ensembles can be retrieved and are addressed by the grib message parameter ``number``. The ensembles are saved per file and standard filenames are supplemented by the letter ``N`` and the ensemble member number in a 3 digit format. 99 100 .. code-block:: bash 101 102 <prefix>YYMMDDHH.N<ENSEMBLE_MEMBER> 103 104 105 Additional fields with new precipitation disaggregation 106 ------------------------------------------------------- 107 108 The new disaggregation method for precipitation fields produces two additional precipitation fields for each time step and precipitation type. They serve as sub-grid points in the original time interval. For details of the method see :doc:`disagg` ??????????????????. 109 The two additional fields are marked with the ``step`` parameter in the Grib messages and are set to "1" and "2" for sub-grid point 1 and 2 respectively. 110 The output filenames do not change in this case. 111 Below is an example list of precipitation fields in an output file generated with the new disaggregation method: 112 113 .. code-block:: bash 114 115 $ grib_ls 116 117 edition centre date dataType gridType stepRange typeOfLevel level shortName packingType 118 1 ecmf 20000108 fc regular_ll 0 surface 0 lsp grid_simple 119 1 ecmf 20000108 fc regular_ll 1 surface 0 lsp grid_simple 120 1 ecmf 20000108 fc regular_ll 2 surface 0 lsp grid_simple 121 1 ecmf 20000108 fc regular_ll 0 surface 0 cp grid_simple 122 1 ecmf 20000108 fc regular_ll 1 surface 0 cp grid_simple 123 1 ecmf 20000108 fc regular_ll 2 surface 0 cp grid_simple 124 125 126 127 128 Temporary files 129 =============== 130 131 ``Flex_extract`` works with a number of temporary data files which are usually deleted after a successful data extraction. They are only stored if the ``DEBUG`` mode is switched on (see :doc:`Input/control_params`. 132 133 MARS grib files 134 --------------- 135 136 ``Flex_extract`` retrieves all meteorological fields from MARS and stores them in files ending with ``.grb``. 137 Since the request times and data transfer of MARS access are limited and ECMWF asks for efficiency in requesting data from MARS, ``flex_extract`` splits the overall data request in several smaller requests. Each request is stored in an extra ``.grb`` file and the file names are put together by several pieces of information: 138 139 .. code-block:: bash 140 141 <field_type><grid_type><temporal_property><level_type>.<date>.<ppid>.<pid>.grb 142 143 Description: 144 145 Field type: 146 ``AN`` - Analysis, ``FC`` - Forecast, ``4V`` - 4d variational analysis, ``CV`` - Validation forecast, ``CF`` - Control forecast, ``PF`` - Perturbed forecast 147 Grid type: 148 ``SH`` - Spherical Harmonics, ``GG`` - Gaussian Grid, ``OG`` - Output Grid (typically lat/lon), ``_OROLSM`` - Orography parameter 149 Temporal property: 150 ``__`` - instantaneous fields, ``_acc`` - accumulated fields 151 Level type: 152 ``ML`` - Model Level, ``SL`` - Surface Level 153 ppid: 154 The process number of the parent process of submitted script. 155 pid: 156 The process number of the submitted script. 157 158 The process ids should avoid mixing of fields if several ``flex_extract`` jobs are performed in parallel (which is, however, not recommended). The date format is YYYYMMDDHH. 159 160 Example ``.grb`` files for a day of CERA-20C data: 161 162 .. code-block:: bash 163 164 ANOG__ML.20000908.71851.71852.grb 165 FCOG_acc_SL.20000907.71851.71852.grb 166 ANOG__SL.20000908.71851.71852.grb 167 OG_OROLSM__SL.20000908.71851.71852.grb 168 ANSH__SL.20000908.71851.71852.grb 169 170 171 MARS request file 172 ----------------- 173 174 This file is a ``csv`` file called ``mars_requests.csv`` with a list of the actual settings of MARS request parameters (one request per line) in a flex_extract job. It is used for documenting the data which were retrieved and for testing reasons. 175 176 Each request consist of the following parameters, whose meaning mainly can be taken from :doc:`Input/control_params` or :doc:`Input/run`: 177 request_number, accuracy, area, dataset, date, expver, gaussian, grid, levelist, levtype, marsclass, number, param, repres, resol, step, stream, target, time, type 178 179 Example output of a one day retrieval of CERA-20c data: 180 181 .. code-block:: bash 182 183 request_number, accuracy, area, dataset, date, expver, gaussian, grid, levelist, levtype, marsclass, number, param, repres, resol, step, stream, target, time, type 184 1, 24, 40.0/-5.0/30.0/5.0, None, 20000107/to/20000109, 1, , 1.0/1.0, 1, SFC, EP, 000, 142.128/143.128/146.128/180.128/181.128/176.128, , 159, 3/to/24/by/3, ENDA, /mnt/nas/Anne/Interpolation/flexextract/flex_extract_v7.1/run/./workspace/CERA_testgrid_local_cds/FCOG_acc_SL.20000107.23903.23904.grb, 18, FC 185 1, 24, 40.0/-5.0/30.0/5.0, None, 20000108/to/20000108, 1, , 1.0/1.0, 85/to/91, ML, EP, 000, 130.128/133.128/131.128/132.128/077.128/246.128/247.128, , 159, 00, ENDA, /mnt/nas/Anne/Interpolation/flexextract/flex_extract_v7.1/run/./workspace/CERA_testgrid_local_cds/ANOG__ML.20000108.23903.23904.grb, 00/03/06/09/12/15/18/21, AN 186 2, 24, 40.0/-5.0/30.0/5.0, None, 20000108/to/20000108, 1, , OFF, 1, ML, EP, 000, 152.128, , 159, 00, ENDA, /mnt/nas/Anne/Interpolation/flexextract/flex_extract_v7.1/run/./workspace/CERA_testgrid_local_cds/ANSH__SL.20000108.23903.23904.grb, 00/03/06/09/12/15/18/21, AN 187 3, 24, 40.0/-5.0/30.0/5.0, None, 20000108, 1, , 1.0/1.0, 1, SFC, EP, 000, 160.128/027.128/028.128/244.128, , 159, 000, ENDA, /mnt/nas/Anne/Interpolation/flexextract/flex_extract_v7.1/run/./workspace/CERA_testgrid_local_cds/OG_OROLSM__SL.20000108.23903.23904.grb, 00, AN 188 4, 24, 40.0/-5.0/30.0/5.0, None, 20000108/to/20000108, 1, , 1.0/1.0, 1, SFC, EP, 000, 141.128/151.128/164.128/165.128/166.128/167.128/168.128/129.128/172.128/027.128/028.128/186.128/187.128/188.128/235.128/139.128/039.128/173.128, , 159, 00, ENDA, /mnt/nas/Anne/Interpolation/flexextract/flex_extract_v7.1/run/./workspace/CERA_testgrid_local_cds/ANOG__SL.20000108.23903.23904.grb, 00/03/06/09/12/15/18/21, AN 189 190 191 VERTICAL.EC 192 ----------- 193 194 The vertical discretization of model levels. This file contains the ``A`` and ``B`` parameters to calculate the model level height in meters. 195 196 197 Index file 198 ---------- 199 200 This file is usually called ``date_time_stepRange.idx``. It contains indices pointing to specific grib messages from one or more grib files. The messages are selected with a composition of grib message keywords. 201 202 203 flux files 204 ---------- 205 206 The flux files contain the de-accumulated and dis-aggregated flux fields of large scale and convective precipitation, eastward turbulent surface stress, northward turbulent surface stress, surface sensible heat flux and the surface net solar radiation. 207 208 .. code-block:: bash 209 210 flux<date>[.N<xxx>][.<xxx>] 211 212 The date format is YYYYMMDDHH. The optional block ``[.N<xxx>]`` marks the ensemble forecast number, where ``<xxx>`` is the ensemble member number. The optional block ``[.<xxx>]`` marks a pure forecast with ``<xxx>`` being the forecast step. 213 214 .. note:: 215 216 In the case of the new dis-aggregation method for precipitation, two new sub-intervals are added in between each time interval. They are identified by the forecast step parameter which is ``0`` for the original time interval and ``1`` or ``2`` for the two new intervals respectively. 217 218 219 fort files 220 ---------- 221 222 There are a number of input files for the ``CONVERT2`` Fortran program named 223 224 .. code-block:: bash 225 226 fort.xx 227 228 where ``xx`` is the number which defines the meteorological fields stored in these files. 229 They are generated by the Python part of ``flex_extract`` by just splitting the meteorological fields for a unique time stamp from the ``*.grb`` files into the ``fort`` files. 230 The following table defines the numbers with their corresponding content. 231 232 .. csv-table:: Content of fort - files 233 :header: "Number", "Content" 234 :widths: 5, 20 235 236 "10", "U and V wind components" 237 "11", "temperature" 238 "12", "logarithm of surface pressure" 239 "13", "divergence (optional)" 240 "16", "surface fields" 241 "17", "specific humidity" 242 "18", "surface specific humidity (reduced gaussian)" 243 "19", "vertical velocity (pressure) (optional)" 244 "21", "eta-coordinate vertical velocity (optional)" 245 "22", "total cloud water content (optional)" 246 247 Some of the fields are solely retrieved with specific settings, e.g. the eta-coordinate vertical velocity is not available in ERA-Interim datasets and the total cloud water content is an optional field for ``FLEXPART v10`` and newer. Please see section ????????? for more information. 248 249 The ``CONVERT2`` program saves its results in file ``fort.15`` which typically contains: 250 251 .. csv-table:: Output file of the Fortran program ``CONVERT2`` 252 :header: "Number", "Content" 253 :widths: 5, 20 254 255 "15", "U and V wind components, eta-coordinate vertical velocity, temperature, surface pressure, specific humidity " 256 257 More details about the content of ``CONVERT2`` can be found in :doc:`vertco`. 258 259 .. note:: 260 261 The ``fort.4`` file is the namelist file to drive the Fortran program ``CONVERT2``. It is therefore also an input file and is described in ??????????????? 262 263 Example of a namelist: 264 265 .. code-block:: bash 266 267 &NAMGEN 268 maxl = 11, 269 maxb = 11, 270 mlevel = 91, 271 mlevelist = "85/to/91", 272 mnauf = 159, 273 metapar = 77, 274 rlo0 = -5.0, 275 rlo1 = 5.0, 276 rla0 = 30.0, 277 rla1 = 40.0, 278 momega = 0, 279 momegadiff = 0, 280 mgauss = 0, 281 msmooth = 0, 282 meta = 1, 283 metadiff = 0, 284 mdpdeta = 1 285 / 286 6 287 7 288 .. toctree::
Note: See TracChangeset
for help on using the changeset viewer.
