1 | ************************************************* |
---|
2 | The compilation job script ``compilejob.ksh`` |
---|
3 | ************************************************* |
---|
4 | |
---|
5 | The compile job is a Korn-shell script which will be created during the installation process for the application modes **remote** and **gateway** from a template called ``installscript.template`` in the template directory. |
---|
6 | |
---|
7 | ``Flex_extract`` uses the Python package `genshi <https://genshi.edgewall.org/>`_ to generate |
---|
8 | the Korn-shell script from the template files by substituting the individual parameters. |
---|
9 | These individual parameters are marked by a doubled ``$`` sign in ``installscript.template``. |
---|
10 | |
---|
11 | The compilation script has a number of settings for the batch system which are fixed, and it differentiates between the *ecgate* and the *cca/ccb* |
---|
12 | server system to load the necessary modules for the environment when submitted to the batch queue. |
---|
13 | |
---|
14 | The submission is done by the ``ECaccess`` tool from within ``flex_extract`` with the command ``ecaccess-job-submit``. |
---|
15 | |
---|
16 | |
---|
17 | |
---|
18 | What does the compilation script do? |
---|
19 | ------------------------------------ |
---|
20 | |
---|
21 | #. It sets the necessary batch-system parameters |
---|
22 | #. It prepares the job environment at the ECMWF servers by loading the necessary library modules |
---|
23 | #. It sets some environment variables for the single session |
---|
24 | #. It creates the ``flex_extract`` root directory in the ``$HOME`` path of the user |
---|
25 | #. It untars the tarball into the root directory. |
---|
26 | #. It compiles the Fortran program using ``makefile``. |
---|
27 | #. At the end, it checks whether the script has returned an error or not, and emails the log file to the user. |
---|
28 | |
---|
29 | |
---|
30 | |
---|
31 | |
---|
32 | |
---|
33 | Example ``compilejob.ksh`` |
---|
34 | -------------------------- |
---|
35 | |
---|
36 | .. code-block:: bash |
---|
37 | |
---|
38 | |
---|
39 | #!/bin/ksh |
---|
40 | |
---|
41 | # ON ECGB: |
---|
42 | # start with ecaccess-job-submit -queueName ecgb NAME_OF_THIS_FILE on gateway server |
---|
43 | # start with sbatch NAME_OF_THIS_FILE directly on machine |
---|
44 | |
---|
45 | #SBATCH --workdir=/scratch/ms/at/km4a |
---|
46 | #SBATCH --qos=normal |
---|
47 | #SBATCH --job-name=flex_ecmwf |
---|
48 | #SBATCH --output=flex_ecmwf.%j.out |
---|
49 | #SBATCH --error=flex_ecmwf.%j.out |
---|
50 | #SBATCH --mail-type=FAIL |
---|
51 | #SBATCH --time=12:00:00 |
---|
52 | |
---|
53 | ## CRAY specific batch requests |
---|
54 | ##PBS -N flex_ecmwf |
---|
55 | ##PBS -q ns |
---|
56 | ##PBS -S /usr/bin/ksh |
---|
57 | ##PBS -o /scratch/ms/at/km4a/flex_ecmwf.${Jobname}.${Job_ID}.out |
---|
58 | # job output is in .ecaccess_DO_NOT_REMOVE |
---|
59 | ##PBS -j oe |
---|
60 | ##PBS -V |
---|
61 | ##PBS -l EC_threads_per_task=1 |
---|
62 | ##PBS -l EC_memory_per_task=3200MB |
---|
63 | |
---|
64 | set -x |
---|
65 | export VERSION=7.1 |
---|
66 | case ${HOST} in |
---|
67 | *ecg*) |
---|
68 | module unload grib_api |
---|
69 | module unload emos |
---|
70 | module load python3 |
---|
71 | module load eccodes |
---|
72 | module load emos/455-r64 |
---|
73 | export FLEXPART_ROOT_SCRIPTS=${HOME} |
---|
74 | export MAKEFILE=makefile_ecgate |
---|
75 | ;; |
---|
76 | *cca*) |
---|
77 | module switch PrgEnv-cray PrgEnv-intel |
---|
78 | module load python3 |
---|
79 | module load eccodes |
---|
80 | module load emos/455-r64 |
---|
81 | echo ${GROUP} |
---|
82 | echo ${HOME} |
---|
83 | echo ${HOME} | awk -F / '{print $1, $2, $3, $4}' |
---|
84 | export GROUP=`echo ${HOME} | awk -F / '{print $4}'` |
---|
85 | export SCRATCH=/scratch/ms/${GROUP}/${USER} |
---|
86 | export FLEXPART_ROOT_SCRIPTS=${HOME} |
---|
87 | export MAKEFILE=makefile_ecgate |
---|
88 | ;; |
---|
89 | esac |
---|
90 | |
---|
91 | mkdir -p ${FLEXPART_ROOT_SCRIPTS}/flex_extract_v${VERSION} |
---|
92 | cd ${FLEXPART_ROOT_SCRIPTS}/flex_extract_v${VERSION} # if FLEXPART_ROOT is not set this means cd to the home directory |
---|
93 | tar -xvf ${HOME}/flex_extract_v${VERSION}.tar |
---|
94 | cd Source/Fortran |
---|
95 | \rm *.o *.mod calc_etadot |
---|
96 | make -f ${MAKEFILE} >flexcompile 2>flexcompile |
---|
97 | |
---|
98 | ls -l calc_etadot >>flexcompile |
---|
99 | if [ $? -eq 0 ]; then |
---|
100 | echo 'SUCCESS!' >>flexcompile |
---|
101 | mail -s flexcompile.${HOST}.$$ ${USER} <flexcompile |
---|
102 | else |
---|
103 | echo Environment: >>flexcompile |
---|
104 | env >> flexcompile |
---|
105 | mail -s "ERROR! flexcompile.${HOST}.$$" ${USER} <flexcompile |
---|
106 | fi |
---|
107 | |
---|
108 | |
---|
109 | |
---|
110 | |
---|
111 | .. toctree:: |
---|
112 | :hidden: |
---|
113 | :maxdepth: 2 |
---|