[5920b7f] | 1 | ************************** |
---|
[f20342a] | 2 | The job script ``job.ksh`` |
---|
[5920b7f] | 3 | ************************** |
---|
[0b00607] | 4 | |
---|
[f20342a] | 5 | The job script is a Korn-shell script which will be created at runtime for each ``flex_extract`` execution in the application modes **remote** and **gateway**. |
---|
[0b00607] | 6 | |
---|
[f20342a] | 7 | It is based on the ``job.temp`` template file stored in the ``Templates`` directory. |
---|
| 8 | This template is generated in the installation process from a ``job.template`` template file. |
---|
[0b00607] | 9 | |
---|
[f20342a] | 10 | ``Flex_extract`` uses the Python package `genshi <https://genshi.edgewall.org/>`_ to generate |
---|
[0b00607] | 11 | the Korn-shell script from the template files by substituting the individual parameters. |
---|
[f20342a] | 12 | These individual parameters are marked by ``$$`` in ``job.temp``. |
---|
[0b00607] | 13 | |
---|
[f20342a] | 14 | The job script has a number of settings for the batch system which are fixed, and differentiates between the *ecgate* and the *cca/ccb* |
---|
[0b00607] | 15 | server system to load the necessary modules for the environment when submitted to the batch queue. |
---|
| 16 | |
---|
| 17 | The submission is done by the ``ECaccess`` tool from within ``flex_extract`` with the command ``ecaccess-job-submit``. |
---|
| 18 | |
---|
| 19 | |
---|
| 20 | |
---|
[f20342a] | 21 | What does the job script do? |
---|
[5920b7f] | 22 | ---------------------------- |
---|
[0b00607] | 23 | |
---|
[f20342a] | 24 | #. It sets necessary batch system parameters. |
---|
| 25 | #. It prepares the job environment at the ECMWF servers by loading the necessary library modules. |
---|
| 26 | #. It sets some environment variables for the single session. |
---|
| 27 | #. It creates the directory structure in the user's ``$SCRATCH`` file system. |
---|
| 28 | #. It creates a CONTROL file on the ECMWF servers whith the parameters set before creating the ``jobscript.ksh``. ``Flex_extract`` has a set of parameters which are passed to the job script with their default or the user-defined values. It also sets ``CONTROL`` as an environment variable. |
---|
| 29 | #. ``Flex_extract`` is started from within the ``work`` directory of the new directory structure by calling the ``submit.py`` script. It sets new paths for input and output directories and the recently generated ``CONTROL`` file. |
---|
| 30 | #. At the end, it checks whether the script has returned an error or not, and emails the log file to the user. |
---|
[0b00607] | 31 | |
---|
| 32 | |
---|
| 33 | |
---|
| 34 | |
---|
| 35 | Example ``jobscript.ksh`` |
---|
| 36 | ------------------------- |
---|
| 37 | |
---|
| 38 | .. code-block:: bash |
---|
| 39 | |
---|
| 40 | #!/bin/ksh |
---|
| 41 | |
---|
| 42 | # ON ECGB: |
---|
| 43 | # start with ecaccess-job-submit -queueName ecgb NAME_OF_THIS_FILE on gateway server |
---|
| 44 | # start with sbatch NAME_OF_THIS_FILE directly on machine |
---|
| 45 | |
---|
| 46 | #SBATCH --workdir=/scratch/ms/at/km4a |
---|
| 47 | #SBATCH --qos=normal |
---|
| 48 | #SBATCH --job-name=flex_ecmwf |
---|
| 49 | #SBATCH --output=flex_ecmwf.%j.out |
---|
| 50 | #SBATCH --error=flex_ecmwf.%j.out |
---|
| 51 | #SBATCH --mail-type=FAIL |
---|
| 52 | #SBATCH --time=12:00:00 |
---|
| 53 | |
---|
| 54 | ## CRAY specific batch requests |
---|
| 55 | ##PBS -N flex_ecmwf |
---|
| 56 | ##PBS -q np |
---|
| 57 | ##PBS -S /usr/bin/ksh |
---|
| 58 | ## -o /scratch/ms/at/km4a/flex_ecmwf.${PBS_JOBID}.out |
---|
| 59 | ## job output is in .ecaccess_DO_NOT_REMOVE |
---|
| 60 | ##PBS -j oe |
---|
| 61 | ##PBS -V |
---|
| 62 | ##PBS -l EC_threads_per_task=24 |
---|
| 63 | ##PBS -l EC_memory_per_task=32000MB |
---|
| 64 | |
---|
| 65 | set -x |
---|
| 66 | export VERSION=7.1 |
---|
| 67 | case ${HOST} in |
---|
| 68 | *ecg*) |
---|
| 69 | module unload grib_api |
---|
| 70 | module unload eccodes |
---|
| 71 | module unload python |
---|
| 72 | module unload emos |
---|
| 73 | module load python3 |
---|
| 74 | module load eccodes/2.12.0 |
---|
| 75 | module load emos/455-r64 |
---|
[b1674ed] | 76 | export PATH=${PATH}:${HOME}/flex_extract_v7.1/Source/Python |
---|
[0b00607] | 77 | ;; |
---|
| 78 | *cca*) |
---|
| 79 | module unload python |
---|
| 80 | module switch PrgEnv-cray PrgEnv-intel |
---|
| 81 | module load python3 |
---|
| 82 | module load eccodes/2.12.0 |
---|
| 83 | module load emos |
---|
| 84 | export SCRATCH=${TMPDIR} |
---|
[b1674ed] | 85 | export PATH=${PATH}:${HOME}/flex_extract_v7.1/Source/Python |
---|
[0b00607] | 86 | ;; |
---|
| 87 | esac |
---|
| 88 | |
---|
| 89 | cd ${SCRATCH} |
---|
| 90 | mkdir -p python$$ |
---|
| 91 | cd python$$ |
---|
| 92 | |
---|
| 93 | export CONTROL=CONTROL |
---|
| 94 | |
---|
| 95 | cat >${CONTROL}<<EOF |
---|
| 96 | accmaxstep 24 |
---|
| 97 | acctime 18 |
---|
| 98 | acctype FC |
---|
| 99 | accuracy 24 |
---|
| 100 | addpar None |
---|
| 101 | area 74.0/-24.0/10.0/60.0 |
---|
| 102 | basetime None |
---|
| 103 | cds_api None |
---|
| 104 | controlfile CONTROL_CERA |
---|
| 105 | cwc 1 |
---|
| 106 | dataset None |
---|
| 107 | date_chunk 3 |
---|
| 108 | debug 1 |
---|
| 109 | destination annep@genericSftp |
---|
| 110 | doubleelda 0 |
---|
| 111 | dpdeta 1 |
---|
| 112 | dtime 3 |
---|
| 113 | ec_api None |
---|
| 114 | ecfsdir ectmp:/${USER}/econdemand/ |
---|
| 115 | ecgid at |
---|
| 116 | ecstorage 0 |
---|
| 117 | ectrans 1 |
---|
| 118 | ecuid km4a |
---|
| 119 | end_date 20000809 |
---|
| 120 | eta 1 |
---|
| 121 | etadiff 0 |
---|
| 122 | etapar 77 |
---|
| 123 | expver 1 |
---|
| 124 | format GRIB1 |
---|
| 125 | gateway srvx8.img.univie.ac.at |
---|
| 126 | gauss 0 |
---|
| 127 | gaussian |
---|
| 128 | grib2flexpart 0 |
---|
| 129 | grid 1.0/1.0 |
---|
| 130 | inputdir /raid60/nas/tmc/Anne/Interpolation/flexextract/flex_extract_v7.1/run/workspace |
---|
| 131 | install_target None |
---|
| 132 | job_chunk 1 |
---|
| 133 | job_template job.temp |
---|
| 134 | left -24. |
---|
| 135 | level 91 |
---|
| 136 | levelist 1/to/91 |
---|
| 137 | logicals gauss omega omegadiff eta etadiff dpdeta cwc wrf grib2flexpart ecstorage ectrans debug oper request public purefc rrint doubleelda |
---|
| 138 | lower 10. |
---|
| 139 | mailfail ${USER} |
---|
| 140 | mailops ${USER} |
---|
| 141 | makefile Makefile.gfortran |
---|
| 142 | marsclass EP |
---|
| 143 | maxstep 0 |
---|
| 144 | number 000 |
---|
| 145 | omega 0 |
---|
| 146 | omegadiff 0 |
---|
| 147 | oper 0 |
---|
| 148 | outputdir /raid60/nas/tmc/Anne/Interpolation/flexextract/flex_extract_v7.1/run/workspace |
---|
| 149 | prefix CE |
---|
| 150 | public 0 |
---|
| 151 | purefc 0 |
---|
| 152 | queue ecgate |
---|
| 153 | request 2 |
---|
| 154 | resol 159 |
---|
| 155 | right 60. |
---|
| 156 | rrint 0 |
---|
| 157 | smooth 0 |
---|
| 158 | start_date 20000809 |
---|
| 159 | step 00 00 00 00 00 00 00 00 |
---|
| 160 | stream ENDA |
---|
| 161 | time 00 03 06 09 12 15 18 21 |
---|
| 162 | type AN AN AN AN AN AN AN AN |
---|
| 163 | upper 74. |
---|
| 164 | wrf 0 |
---|
| 165 | |
---|
| 166 | EOF |
---|
| 167 | |
---|
| 168 | |
---|
| 169 | submit.py --controlfile=${CONTROL} --inputdir=./work --outputdir=./work 1> prot 2>&1 |
---|
| 170 | |
---|
| 171 | if [ $? -eq 0 ] ; then |
---|
| 172 | l=0 |
---|
| 173 | for muser in `grep -i MAILOPS ${CONTROL}`; do |
---|
| 174 | if [ ${l} -gt 0 ] ; then |
---|
| 175 | mail -s flex.${HOST}.$$ ${muser} <prot |
---|
| 176 | fi |
---|
| 177 | l=$((${l}+1)) |
---|
| 178 | done |
---|
| 179 | else |
---|
| 180 | l=0 |
---|
| 181 | for muser in `grep -i MAILFAIL ${CONTROL}`; do |
---|
| 182 | if [ ${l} -gt 0 ] ; then |
---|
| 183 | mail -s "ERROR! flex.${HOST}.$$" ${muser} <prot |
---|
| 184 | fi |
---|
| 185 | l=$((${l}+1)) |
---|
| 186 | done |
---|
| 187 | fi |
---|
| 188 | |
---|
| 189 | |
---|
| 190 | |
---|
| 191 | |
---|
| 192 | .. toctree:: |
---|
| 193 | :hidden: |
---|
| 194 | :maxdepth: 2 |
---|