source: flexpart.git/src/timemanager_mpi.f90 @ 41d8574

10.4.1_peseiGFS_025bugfixes+enhancementsdevrelease-10release-10.4.1scaling-bugunivie
Last change on this file since 41d8574 was 41d8574, checked in by Espen Sollum ATMOS <eso@…>, 8 years ago

bugfix: MPI version gave wrong wet deposition when using ECMWF cloud water fields. Cloud water in ECMWF fields now uses parameter qc, or reads clwc+ciwc. Added minmass variable as limit for terminating particles.

  • Property mode set to 100644
File size: 32.6 KB
Line 
1!**********************************************************************
2! Copyright 1998,1999,2000,2001,2002,2005,2007,2008,2009,2010         *
3! Andreas Stohl, Petra Seibert, A. Frank, Gerhard Wotawa,             *
4! Caroline Forster, Sabine Eckhardt, John Burkhart, Harald Sodemann   *
5!                                                                     *
6! This file is part of FLEXPART.                                      *
7!                                                                     *
8! FLEXPART is free software: you can redistribute it and/or modify    *
9! it under the terms of the GNU General Public License as published by*
10! the Free Software Foundation, either version 3 of the License, or   *
11! (at your option) any later version.                                 *
12!                                                                     *
13! FLEXPART is distributed in the hope that it will be useful,         *
14! but WITHOUT ANY WARRANTY; without even the implied warranty of      *
15! MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the       *
16! GNU General Public License for more details.                        *
17!                                                                     *
18! You should have received a copy of the GNU General Public License   *
19! along with FLEXPART.  If not, see <http://www.gnu.org/licenses/>.   *
20!**********************************************************************
21
22subroutine timemanager
23
24!*****************************************************************************
25!                                                                            *
26! Handles the computation of trajectories, i.e. determines which             *
27! trajectories have to be computed at what time.                             *
28! Manages dry+wet deposition routines, radioactive decay and the computation *
29! of concentrations.                                                         *
30!                                                                            *
31!     Author: A. Stohl                                                       *
32!                                                                            *
33!     20 May 1996                                                            *
34!                                                                            *
35!*****************************************************************************
36!  Changes, Bernd C. Krueger, Feb. 2001:                                     *
37!        Call of convmix when new windfield is read                          *
38!------------------------------------                                        *
39!  Changes Petra Seibert, Sept 2002                                          *
40!     fix wet scavenging problem                                             *
41!     Code may not be correct for decay of deposition!                       *
42!  Changes Petra Seibert, Nov 2002                                           *
43!     call convection BEFORE new fields are read in BWD mode                 *
44!  Changes Caroline Forster, Feb 2005                                        *
45!   new interface between flexpart and convection scheme                     *
46!   Emanuel's latest subroutine convect43c.f is used                         *
47!  Changes Stefan Henne, Harald Sodemann, 2013-2014                          *
48!   added netcdf output code                                                 *
49!  Changes Espen Sollum 2014                                                 *
50!   MPI version                                                              *
51!   Variables uap,ucp,uzp,us,vs,ws,cbt now in module com_mod                 *
52!*****************************************************************************
53!                                                                            *
54! Variables:                                                                 *
55! dep                .true. if either wet or dry deposition is switched on   *
56! decay(maxspec) [1/s] decay constant for radioactive decay                  *
57! drydep             .true. if dry deposition is switched on                 *
58! ideltas [s]        modelling period                                        *
59! itime [s]          actual temporal position of calculation                 *
60! ldeltat [s]        time since computation of radioact. decay of depositions*
61! loutaver [s]       averaging period for concentration calculations         *
62! loutend [s]        end of averaging for concentration calculations         *
63! loutnext [s]       next time at which output fields shall be centered      *
64! loutsample [s]     sampling interval for averaging of concentrations       *
65! loutstart [s]      start of averaging for concentration calculations       *
66! loutstep [s]       time interval for which concentrations shall be         *
67!                    calculated                                              *
68! npoint(maxpart)    index, which starting point the trajectory has          *
69!                    starting positions of trajectories                      *
70! nstop              serves as indicator for fate of particles               *
71!                    in the particle loop                                    *
72! nstop1             serves as indicator for wind fields (see getfields)     *
73! memstat            additional indicator for wind fields (see getfields)    *
74! outnum             number of samples for each concentration calculation    *
75! prob               probability of absorption at ground due to dry          *
76!                    deposition                                              *
77! wetdep             .true. if wet deposition is switched on                 *
78! weight             weight for each concentration sample (1/2 or 1)         *
79! uap(maxpart),ucp(maxpart),uzp(maxpart) = random velocities due to          *
80!                    turbulence                                              *
81! us(maxpart),vs(maxpart),ws(maxpart) = random velocities due to inter-      *
82!                    polation                                                *
83! xtra1(maxpart), ytra1(maxpart), ztra1(maxpart) =                           *
84!                    spatial positions of trajectories                       *
85!                                                                            *
86! Constants:                                                                 *
87! maxpart            maximum number of trajectories                          *
88!                                                                            *
89!*****************************************************************************
90
91  use unc_mod
92  use point_mod
93  use xmass_mod
94  use flux_mod
95  use outg_mod
96  use oh_mod
97  use par_mod
98  use com_mod
99  use mpi_mod
100  use netcdf_output_mod, only: concoutput_netcdf,concoutput_nest_netcdf,&
101       &concoutput_surf_netcdf,concoutput_surf_nest_netcdf
102
103  implicit none
104
105  logical :: reqv_state=.false. ! .true. if waiting for a MPI_Irecv to complete
106  integer :: j,ks,kp,l,n,itime=0,nstop,nstop1,memstat=0,mind
107! integer :: ksp
108  integer :: ip
109  integer :: loutnext,loutstart,loutend
110  integer :: ix,jy,ldeltat,itage,nage
111  integer :: i_nan=0,ii_nan,total_nan_intl=0  !added by mc to check instability in CBL scheme
112  integer :: numpart_tot_mpi ! for summing particles on all processes
113  real :: outnum,weight,prob(maxspec)
114  real :: decfact
115
116  real :: drydeposit(maxspec),gridtotalunc,wetgridtotalunc
117  real :: drygridtotalunc,xold,yold,zold,xmassfract
118!double precision xm(maxspec,maxpointspec_act),
119!    +                 xm_depw(maxspec,maxpointspec_act),
120!    +                 xm_depd(maxspec,maxpointspec_act)
121
122!open(88,file='TEST.dat')
123
124! Measure time spent in timemanager
125  if (mp_measure_time) call mpif_mtime('timemanager',0)
126
127! First output for time 0
128!************************
129
130  loutnext=loutstep/2
131  outnum=0.
132  loutstart=loutnext-loutaver/2
133  loutend=loutnext+loutaver/2
134
135
136!**********************************************************************
137! Loop over the whole modelling period in time steps of mintime seconds
138!**********************************************************************
139
140
141!  itime=0
142  if (lroot) then
143  !  write(*,45) itime,numpart*mp_partgroup_np,gridtotalunc,wetgridtotalunc,drygridtotalunc
144    write(*,46) float(itime)/3600,itime,numpart*mp_partgroup_np
145   
146    if (verbosity.gt.0) then
147      write (*,*) 'timemanager> starting simulation'
148    end if
149  end if ! (lroot)
150
151!CGZ-lifetime: set lifetime to 0
152  ! checklifetime(:,:)=0
153  ! species_lifetime(:,:)=0
154  ! print*, 'Initialized lifetime'
155!CGZ-lifetime: set lifetime to 0
156
157
158
159  do itime=0,ideltas,lsynctime
160
161! Computation of wet deposition, OH reaction and mass transfer
162! between two species every lsynctime seconds
163! maybe wet depo frequency can be relaxed later but better be on safe side
164! wetdepo must be called BEFORE new fields are read in but should not
165! be called in the very beginning before any fields are loaded, or
166! before particles are in the system
167! Code may not be correct for decay of deposition
168! changed by Petra Seibert 9/02
169!********************************************************************
170
171    if (mp_dev_mode) write(*,*) 'myid, itime: ',mp_pid,itime
172   
173    if (wetdep .and. itime .ne. 0 .and. numpart .gt. 0) then
174      if (verbosity.gt.0) then
175        write (*,*) 'timemanager> call wetdepo'
176      endif
177
178      if (mp_measure_time) call mpif_mtime('wetdepo',0)
179       
180! readwind process skips this step
181      if (.not.(lmpreader.and.lmp_use_reader)) then
182        call wetdepo(itime,lsynctime,loutnext)
183      end if
184
185      if (mp_measure_time) call mpif_mtime('wetdepo',1)
186    end if
187
188    if (OHREA .and. itime .ne. 0 .and. numpart .gt. 0) then
189! readwind process skips this step
190      if (.not.(lmpreader.and.lmp_use_reader)) then
191        call ohreaction(itime,lsynctime,loutnext)
192      endif
193    end if
194
195    if (assspec .and. itime .ne. 0 .and. numpart .gt. 0) then
196      stop 'associated species not yet implemented!'
197!     call transferspec(itime,lsynctime,loutnext)
198    endif
199
200
201! compute convection for backward runs
202!*************************************
203
204    if ((ldirect.eq.-1).and.(lconvection.eq.1).and.(itime.lt.0)) then
205      if (verbosity.gt.0) then
206        write (*,*) 'timemanager> call convmix -- backward'
207      endif
208! readwind process skips this step
209      if (.not.(lmpreader.and.lmp_use_reader)) call convmix(itime)
210    endif
211
212! Get necessary wind fields if not available
213!*******************************************
214    if (verbosity.gt.0 .and. lmpreader) then
215      write (*,*) 'timemanager> call getfields'
216    endif
217
218! This time measure includes reading/MPI communication (for the reader process),
219! or MPI communication time only (for other processes)
220    if (mp_measure_time) call mpif_mtime('getfields',0)
221
222    call getfields(itime,nstop1,memstat)
223
224    if (mp_measure_time) call mpif_mtime('getfields',1)
225
226
227! Broadcast fields to all MPI processes
228! Skip if all processes have called getfields or if no new fields
229!*****************************************************************
230
231    if (mp_measure_time.and..not.(lmpreader.and.lmp_use_reader)) call mpif_mtime('getfields',0)
232
233! Version 1 (lmp_sync=.true.) uses a read-ahead process where send/recv is done
234! in sync at start of each new field time interval
235    if (lmp_sync.and.lmp_use_reader.and.memstat.gt.0) then
236      call mpif_gf_send_vars(memstat)
237      call mpif_gf_send_vars_nest(memstat)
238! Version 2  (lmp_sync=.false., see below) is also used whenever 2 new fields are
239! read (as at first time step), in which case async send/recv is impossible.
240    else if (.not.lmp_sync.and.lmp_use_reader.and.memstat.ge.32) then
241      call mpif_gf_send_vars(memstat)
242      call mpif_gf_send_vars_nest(memstat)
243    end if
244
245! Version 2 (lmp_sync=.false.) is for holding three fields in memory. Uses a
246! read-ahead process where sending/receiving of the 3rd fields is done in
247! the background in parallel with performing computations with fields 1&2
248!********************************************************************************
249    if (.not.lmp_sync) then
250   
251! READER PROCESS:
252      if (memstat.gt.0..and.memstat.lt.32.and.lmp_use_reader.and.lmpreader) then
253        if (mp_dev_mode) write(*,*) 'Reader process: calling mpif_gf_send_vars_async'
254        call mpif_gf_send_vars_async(memstat)
255      end if
256
257! COMPLETION CHECK:
258! Issued at start of each new field period.
259      if (memstat.ne.0.and.memstat.lt.32.and.lmp_use_reader) then
260! TODO: z0(7) changes with time, so should be dimension (numclass,2) to
261! allow transfer of the future value in the background
262        call MPI_Bcast(z0,numclass,mp_pp,id_read,MPI_COMM_WORLD,mp_ierr)
263        call mpif_gf_request
264      end if
265
266! RECVEIVING PROCESS(ES):
267      ! eso TODO: at this point we do not know if clwc/ciwc will be available
268      ! at next time step. Issue receive request anyway, cancel at mpif_gf_request
269      if (memstat.gt.0.and.lmp_use_reader.and..not.lmpreader) then
270        if (mp_dev_mode) write(*,*) 'Receiving process: calling mpif_gf_send_vars_async. PID: ', mp_pid
271        call mpif_gf_recv_vars_async(memstat)
272      end if
273
274    end if
275
276    if (mp_measure_time.and..not.(lmpreader.and.lmp_use_reader)) call mpif_mtime('getfields',1)
277
278! For validation and tests: call the function below to set all fields to simple
279! homogeneous values
280    if (mp_dev_mode) call set_fields_synthetic
281
282!*******************************************************************************
283
284    if (lmpreader.and.nstop1.gt.1) stop 'NO METEO FIELDS AVAILABLE'
285
286! Reader process goes back to top of time loop (unless simulation end)
287!*********************************************************************
288
289    if (lmpreader.and.lmp_use_reader) then
290      if (itime.lt.ideltas) then
291        cycle
292      else
293        goto 999
294      end if
295    end if
296
297
298! Get hourly OH fields if not available
299!****************************************************
300    if (OHREA) then
301      if (verbosity.gt.0) then
302        write (*,*) 'timemanager> call gethourlyOH'
303      endif
304      call gethourlyOH(itime)
305    endif
306
307
308! Release particles
309!******************
310
311    if (verbosity.gt.0.and.lroot) then
312      write (*,*) 'timemanager>  Release particles'
313    endif
314
315    if (mdomainfill.ge.1) then
316      if (itime.eq.0) then
317        if (verbosity.gt.0) then
318          write (*,*) 'timemanager>  call init_domainfill'
319        endif
320        call init_domainfill
321      else
322        if (verbosity.gt.0.and.lroot) then
323          write (*,*) 'timemanager>  call boundcond_domainfill'
324        endif
325        call boundcond_domainfill(itime,loutend)
326      endif
327    else
328      if (verbosity.gt.0.and.lroot) then
329        print*,'call releaseparticles' 
330      endif
331      call releaseparticles(itime)
332    endif
333
334
335! Compute convective mixing for forward runs
336! for backward runs it is done before next windfield is read in
337!**************************************************************
338
339    if ((ldirect.eq.1).and.(lconvection.eq.1)) then
340      if (verbosity.gt.0) then
341        write (*,*) 'timemanager> call convmix -- forward'
342      endif
343      call convmix(itime)
344    endif
345
346! If middle of averaging period of output fields is reached, accumulated
347! deposited mass radioactively decays
348!***********************************************************************
349
350    if (DEP.and.(itime.eq.loutnext).and.(ldirect.gt.0)) then
351      do ks=1,nspec
352        do kp=1,maxpointspec_act
353          if (decay(ks).gt.0.) then ! TODO move this statement up 2 levels
354            do nage=1,nageclass
355              do l=1,nclassunc
356! Mother output grid
357                do jy=0,numygrid-1
358                  do ix=0,numxgrid-1
359                    wetgridunc(ix,jy,ks,kp,l,nage)= &
360                         wetgridunc(ix,jy,ks,kp,l,nage)* &
361                         exp(-1.*outstep*decay(ks))
362                    drygridunc(ix,jy,ks,kp,l,nage)= &
363                         drygridunc(ix,jy,ks,kp,l,nage)* &
364                         exp(-1.*outstep*decay(ks))
365                  end do
366                end do
367! Nested output grid
368                if (nested_output.eq.1) then
369                  do jy=0,numygridn-1
370                    do ix=0,numxgridn-1
371                      wetgriduncn(ix,jy,ks,kp,l,nage)= &
372                           wetgriduncn(ix,jy,ks,kp,l,nage)* &
373                           exp(-1.*outstep*decay(ks))
374                      drygriduncn(ix,jy,ks,kp,l,nage)= &
375                           drygriduncn(ix,jy,ks,kp,l,nage)* &
376                           exp(-1.*outstep*decay(ks))
377                    end do
378                  end do
379                endif
380              end do
381            end do
382          endif
383        end do
384      end do
385    endif
386
387
388!!! CHANGE: These lines may be switched on to check the conservation
389!!! of mass within FLEXPART
390!   if (itime.eq.loutnext) then
391!   do 247 ksp=1, nspec
392!   do 247 kp=1, maxpointspec_act
393!47         xm(ksp,kp)=0.
394
395!   do 249 ksp=1, nspec
396!     do 249 j=1,numpart
397!          if (ioutputforeachrelease.eq.1) then
398!            kp=npoint(j)
399!          else
400!            kp=1
401!          endif
402!       if (itra1(j).eq.itime) then
403!          xm(ksp,kp)=xm(ksp,kp)+xmass1(j,ksp)
404!         write(*,*) 'xmass: ',xmass1(j,ksp),j,ksp,nspec
405!       endif
406!49     continue
407!  do 248 ksp=1,nspec
408!  do 248 kp=1,maxpointspec_act
409!  xm_depw(ksp,kp)=0.
410!  xm_depd(ksp,kp)=0.
411!     do 248 nage=1,nageclass
412!       do 248 ix=0,numxgrid-1
413!         do 248 jy=0,numygrid-1
414!           do 248 l=1,nclassunc
415!              xm_depw(ksp,kp)=xm_depw(ksp,kp)
416!    +                  +wetgridunc(ix,jy,ksp,kp,l,nage)
417!48                 xm_depd(ksp,kp)=xm_depd(ksp,kp)
418!    +                  +drygridunc(ix,jy,ksp,kp,l,nage)
419!             do 246 ksp=1,nspec
420!46                    write(88,'(2i10,3e12.3)')
421!    +              itime,ksp,(xm(ksp,kp),kp=1,maxpointspec_act),
422!    +                (xm_depw(ksp,kp),kp=1,maxpointspec_act),
423!    +                (xm_depd(ksp,kp),kp=1,maxpointspec_act)
424!  endif
425!!! CHANGE
426
427
428
429
430! Check whether concentrations are to be calculated
431!**************************************************
432
433    if ((ldirect*itime.ge.ldirect*loutstart).and. &
434         (ldirect*itime.le.ldirect*loutend)) then ! add to grid
435      if (mod(itime-loutstart,loutsample).eq.0) then
436
437! If we are exactly at the start or end of the concentration averaging interval,
438! give only half the weight to this sample
439!*****************************************************************************
440
441        if ((itime.eq.loutstart).or.(itime.eq.loutend)) then
442          weight=0.5
443        else
444          weight=1.0
445        endif
446        outnum=outnum+weight
447
448        call conccalc(itime,weight)
449
450      endif
451
452! :TODO: MPI output of particle positions;  each process sequentially
453!   access the same file
454      if ((mquasilag.eq.1).and.(itime.eq.(loutstart+loutend)/2)) &
455           call partoutput_short(itime)    ! dump particle positions in extremely compressed format
456
457
458! Output and reinitialization of grid
459! If necessary, first sample of new grid is also taken
460!*****************************************************
461
462      if ((itime.eq.loutend).and.(outnum.gt.0.)) then
463        if ((iout.le.3.).or.(iout.eq.5)) then
464
465! MPI: Root process collects/sums grids
466!**************************************
467          call mpif_tm_reduce_grid
468
469          if (mp_measure_time) call mpif_mtime('iotime',0)
470          if (surf_only.ne.1) then
471            if (lroot) then
472              if (lnetcdfout.eq.1) then
473                call concoutput_netcdf(itime,outnum,gridtotalunc,wetgridtotalunc,&
474                     &drygridtotalunc)
475              else
476                call concoutput(itime,outnum,gridtotalunc,wetgridtotalunc,drygridtotalunc)
477              endif
478            else
479! zero arrays on non-root processes
480              gridunc(:,:,:,:,:,:,:)=0.
481              creceptor(:,:)=0.
482            end if
483          else
484            if (lroot) then
485              if (lnetcdfout.eq.1) then
486                call concoutput_surf_netcdf(itime,outnum,gridtotalunc,wetgridtotalunc,&
487                     &drygridtotalunc)
488              else
489                call concoutput_surf(itime,outnum,gridtotalunc,wetgridtotalunc,drygridtotalunc)
490              end if
491            else
492! zero arrays on non-root processes
493              gridunc(:,:,:,:,:,:,:)=0.
494              creceptor(:,:)=0.
495            endif
496          endif
497          if (mp_measure_time) call mpif_mtime('iotime',1)
498
499! :TODO: Correct calling of conc_surf above?
500
501!   call concoutput_surf(itime,outnum,gridtotalunc,wetgridtotalunc,drygridtotalunc)
502! endif
503
504          if (nested_output.eq.1) then
505
506! MPI: Root process collects/sums nested grids
507!*********************************************
508            call mpif_tm_reduce_grid_nest
509 
510           if (mp_measure_time) call mpif_mtime('iotime',0)
511
512            if (lnetcdfout.eq.0) then
513              if (surf_only.ne.1) then
514
515                if (lroot) then
516                  call concoutput_nest(itime,outnum)
517                else
518                  griduncn(:,:,:,:,:,:,:)=0.
519                end if
520
521              else  ! :TODO: check for zeroing in the netcdf module
522                call concoutput_surf_nest(itime,outnum)
523
524              end if
525
526            else
527
528              if (surf_only.ne.1) then
529                if (lroot) then             
530                  call concoutput_nest_netcdf(itime,outnum)
531                else
532                  griduncn(:,:,:,:,:,:,:)=0.
533                end if
534              else
535                if (lroot) then
536                  call concoutput_surf_nest_netcdf(itime,outnum)
537                else
538                  griduncn(:,:,:,:,:,:,:)=0.
539                end if
540              endif
541
542
543            end if
544          end if
545         
546
547          outnum=0.
548        endif
549        if ((iout.eq.4).or.(iout.eq.5)) call plumetraj(itime)
550        if (iflux.eq.1) call fluxoutput(itime)
551        if (mp_measure_time) call mpif_mtime('iotime',1)
552
553
554! Decide whether to write an estimate of the number of particles released,
555! or exact number (require MPI reduce operation)
556        numpart_tot_mpi = numpart*mp_partgroup_np
557
558        if (mp_exact_numpart.and..not.(lmpreader.and.lmp_use_reader)) then
559          call MPI_Reduce(numpart, numpart_tot_mpi, 1, MPI_INTEGER, MPI_SUM, id_root, &
560               & mp_comm_used, mp_ierr)
561        endif
562       
563        !CGZ-lifetime: output species lifetime
564        if (lroot) then
565        !   write(*,*) 'Overview species lifetime in days', &
566        !        real((species_lifetime(:,1)/species_lifetime(:,2))/real(3600.0*24.0))
567        !   write(*,*) 'all info:',species_lifetime
568          write(*,45) itime,numpart_tot_mpi,gridtotalunc,&
569               &wetgridtotalunc,drygridtotalunc
570        !   if (verbosity.gt.0) then
571        !     write (*,*) 'timemanager> starting simulation'
572        !   end if
573        end if
574
57545      format(i13,' SECONDS SIMULATED: ',i13, ' PARTICLES:    Uncertainty: ',3f7.3)
57646      format(' Simulated ',f7.1,' hours (',i13,' s), ',i13, ' particles')
577        if (ipout.ge.1) then
578          do ip=0, mp_partgroup_np-1
579            if (ip.eq.mp_partid) call partoutput(itime) ! dump particle positions
580            call mpif_mpi_barrier
581          end do
582        end if
583
584        loutnext=loutnext+loutstep
585        loutstart=loutnext-loutaver/2
586        loutend=loutnext+loutaver/2
587        if (itime.eq.loutstart) then
588          weight=0.5
589          outnum=outnum+weight
590          call conccalc(itime,weight)
591        endif
592
593
594! Check, whether particles are to be split:
595! If so, create new particles and attribute all information from the old
596! particles also to the new ones; old and new particles both get half the
597! mass of the old ones
598!************************************************************************
599       
600        if (ldirect*itime.ge.ldirect*itsplit) then
601          n=numpart
602          do j=1,numpart
603            if (ldirect*itime.ge.ldirect*itrasplit(j)) then
604!                if (n.lt.maxpart) then
605              if (n.lt.maxpart_mpi) then
606                n=n+1
607                itrasplit(j)=2*(itrasplit(j)-itramem(j))+itramem(j)
608                itrasplit(n)=itrasplit(j)
609                itramem(n)=itramem(j)
610                itra1(n)=itra1(j)
611                idt(n)=idt(j)
612                npoint(n)=npoint(j)
613                nclass(n)=nclass(j)
614                xtra1(n)=xtra1(j)
615                ytra1(n)=ytra1(j)
616                ztra1(n)=ztra1(j)
617                uap(n)=uap(j)
618                ucp(n)=ucp(j)
619                uzp(n)=uzp(j)
620                us(n)=us(j)
621                vs(n)=vs(j)
622                ws(n)=ws(j)
623                cbt(n)=cbt(j)
624                do ks=1,nspec
625                  xmass1(j,ks)=xmass1(j,ks)/2.
626                  xmass1(n,ks)=xmass1(j,ks)
627                end do
628              endif
629            endif
630          end do
631          numpart=n
632        endif
633      endif
634    endif
635   
636
637    if (itime.eq.ideltas) exit         ! almost finished
638
639! Compute interval since radioactive decay of deposited mass was computed
640!************************************************************************
641
642    if (itime.lt.loutnext) then
643      ldeltat=itime-(loutnext-loutstep)
644    else                                  ! first half of next interval
645      ldeltat=itime-loutnext
646    endif
647
648
649! Loop over all particles
650!************************
651    if (mp_measure_time) call mpif_mtime('partloop1',0)
652
653!--------------------------------------------------------------------------------
654! various variables for testing reason of CBL scheme, by mc
655    well_mixed_vector=0. !erase vector to test well mixed condition: modified by mc
656    well_mixed_norm=0.   !erase normalization to test well mixed condition: modified by mc
657    avg_ol=0.
658    avg_wst=0.
659    avg_h=0.
660    avg_air_dens=0.  !erase vector to obtain air density at particle positions: modified by mc
661!--------------------------------------------------------------------------------
662
663    do j=1,numpart
664
665! If integration step is due, do it
666!**********************************
667
668      if (itra1(j).eq.itime) then
669
670        if (ioutputforeachrelease.eq.1) then
671          kp=npoint(j)
672        else
673          kp=1
674        endif
675! Determine age class of the particle
676        itage=abs(itra1(j)-itramem(j))
677        do nage=1,nageclass
678          if (itage.lt.lage(nage)) exit
679        end do
680
681! Initialize newly released particle
682!***********************************
683
684        if ((itramem(j).eq.itime).or.(itime.eq.0)) &
685             call initialize(itime,idt(j),uap(j),ucp(j),uzp(j), &
686             us(j),vs(j),ws(j),xtra1(j),ytra1(j),ztra1(j),cbt(j))
687
688! Memorize particle positions
689!****************************
690
691        xold=xtra1(j)
692        yold=ytra1(j)
693        zold=ztra1(j)
694
695! Integrate Lagevin equation for lsynctime seconds
696!*************************************************
697
698        mp_advance_wtime_beg = mpi_wtime()
699
700        call advance(itime,npoint(j),idt(j),uap(j),ucp(j),uzp(j), &
701             us(j),vs(j),ws(j),nstop,xtra1(j),ytra1(j),ztra1(j),prob, &
702             cbt(j))
703
704        mp_advance_wtime_end = mpi_wtime()
705        mp_advance_wtime_total = mp_advance_wtime_total + (mp_advance_wtime_end - &
706             & mp_advance_wtime_beg)
707
708
709! Calculate the gross fluxes across layer interfaces
710!***************************************************
711
712        if (iflux.eq.1) call calcfluxes(nage,j,xold,yold,zold)
713
714
715! Determine, when next time step is due
716! If trajectory is terminated, mark it
717!**************************************
718
719        if (nstop.gt.1) then
720          if (linit_cond.ge.1) call initial_cond_calc(itime,j)
721          itra1(j)=-999999999
722        else
723          itra1(j)=itime+lsynctime
724
725
726! Dry deposition and radioactive decay for each species
727! Also check maximum (of all species) of initial mass remaining on the particle;
728! if it is below a threshold value, terminate particle
729!*****************************************************************************
730
731          xmassfract=0.
732          do ks=1,nspec
733            if (decay(ks).gt.0.) then             ! radioactive decay
734              decfact=exp(-real(abs(lsynctime))*decay(ks))
735            else
736              decfact=1.
737            endif
738
739            if (DRYDEPSPEC(ks)) then        ! dry deposition
740              drydeposit(ks)=xmass1(j,ks)*prob(ks)*decfact
741              xmass1(j,ks)=xmass1(j,ks)*(1.-prob(ks))*decfact
742              if (decay(ks).gt.0.) then   ! correct for decay (see wetdepo)
743                drydeposit(ks)=drydeposit(ks)* &
744                     exp(real(abs(ldeltat))*decay(ks))
745              endif
746            else                           ! no dry deposition
747              xmass1(j,ks)=xmass1(j,ks)*decfact
748            endif
749
750
751            if (mdomainfill.eq.0) then
752              if (xmass(npoint(j),ks).gt.0.)then
753                   xmassfract=max(xmassfract,real(npart(npoint(j)))* &
754                   xmass1(j,ks)/xmass(npoint(j),ks))
755                   
756                   !CGZ-lifetime: Check mass fraction left/save lifetime
757                   ! if(lroot.and.real(npart(npoint(j)))*xmass1(j,ks)/xmass(npoint(j),ks).lt.0.01.and.checklifetime(j,ks).eq.0.)then
758                       !Mass below 1% of initial >register lifetime
759                   !     checklifetime(j,ks)=abs(itra1(j)-itramem(j))
760
761                   !     species_lifetime(ks,1)=species_lifetime(ks,1)+abs(itra1(j)-itramem(j))
762                   !     species_lifetime(ks,2)= species_lifetime(ks,2)+1
763                   ! endif
764                   !CGZ-lifetime: Check mass fraction left/save lifetime
765                   
766              endif
767            else
768              xmassfract=1.
769            endif
770          end do
771
772          if (xmassfract.lt.minmass) then ! .and. sum(real(npart(npoint(j)))*xmass1(j,:)).lt.1.0) then   ! terminate all particles carrying less mass
773          !            print*,'terminated particle ',j,' for small mass (', sum(real(npart(npoint(j)))* &
774          !         xmass1(j,:)), ' of ', sum(xmass(npoint(j),:)),')'
775            itra1(j)=-999999999
776          endif
777
778!        Sabine Eckhardt, June 2008
779!        don't create depofield for backward runs
780          if (DRYDEP.AND.(ldirect.eq.1)) then
781            call drydepokernel(nclass(j),drydeposit,real(xtra1(j)), &
782                 real(ytra1(j)),nage,kp)
783            if (nested_output.eq.1) call drydepokernel_nest( &
784                 nclass(j),drydeposit,real(xtra1(j)),real(ytra1(j)), &
785                 nage,kp)
786          endif
787
788! Terminate trajectories that are older than maximum allowed age
789!***************************************************************
790
791          if (abs(itra1(j)-itramem(j)).ge.lage(nageclass)) then
792            if (linit_cond.ge.1) &
793                 call initial_cond_calc(itime+lsynctime,j)
794            itra1(j)=-999999999
795            !print*, 'terminated particle ',j,'for age'
796          endif
797        endif
798
799      endif
800
801    end do ! j=1, numpart
802
803    if(mp_measure_time) call mpif_mtime('partloop1',1)
804
805
806! Added by mc: counter of "unstable" particle velocity during a time scale
807!   of maximumtl=20 minutes (defined in com_mod)
808
809    total_nan_intl=0
810    i_nan=i_nan+1 ! added by mc to count nan during a time of maxtl (i.e. maximum tl fixed here to 20 minutes, see com_mod)
811    sum_nan_count(i_nan)=nan_count
812    if (i_nan > maxtl/lsynctime) i_nan=1 !lsynctime must be <= maxtl
813    do ii_nan=1, (maxtl/lsynctime)
814      total_nan_intl=total_nan_intl+sum_nan_count(ii_nan)
815    end do
816
817! Output to keep track of the numerical instabilities in CBL simulation
818! and if they are compromising the final result (or not):
819    if (cblflag.eq.1) print *,j,itime,'nan_synctime',nan_count,'nan_tl',total_nan_intl 
820
821
822  end do ! itime=0,ideltas,lsynctime
823
824
825! Complete the calculation of initial conditions for particles not yet terminated
826!*****************************************************************************
827
828! eso :TODO: this not implemented yet (transfer particles to PID 0 or rewrite)
829! the tools to do this are already in mpi_mod.f90
830! :CUR: check this
831  if (lroot) then
832    do j=1,numpart
833      if (linit_cond.ge.1) call initial_cond_calc(itime,j)
834    end do
835  end if
836
837
838  if (ipout.eq.2) then
839! MPI process 0 creates the file, the other processes append to it
840    do ip=0, mp_partgroup_np-1
841      if (ip.eq.mp_partid) then
842        !if (mp_dbg_mode) write(*,*) 'call partoutput(itime), proc, mp_partid',ip,mp_partid
843        call partoutput(itime)    ! dump particle positions
844      end if
845      call mpif_mpi_barrier
846    end do
847  end if
848
849! eso :TODO: MPI
850  if (linit_cond.ge.1.and.lroot) call initial_cond_output(itime)   ! dump initial cond. field
851
852
853! De-allocate memory and end
854!***************************
855
856999 if (iflux.eq.1) then
857    deallocate(flux)
858  endif
859  if (OHREA) then
860    deallocate(OH_field,OH_hourly,lonOH,latOH,altOH)
861  endif
862  if (ldirect.gt.0) then
863    deallocate(drygridunc,wetgridunc)
864  endif
865  deallocate(gridunc)
866  deallocate(xpoint1,xpoint2,ypoint1,ypoint2,zpoint1,zpoint2,xmass, checklifetime)
867  deallocate(ireleasestart,ireleaseend,npart,kindz)
868  deallocate(xmasssave)
869  if (nested_output.eq.1) then
870    deallocate(orooutn, arean, volumen)
871    if (ldirect.gt.0) then
872      deallocate(griduncn,drygriduncn,wetgriduncn)
873    endif
874  endif
875  deallocate(outheight,outheighthalf)
876  deallocate(oroout, area, volume)
877
878  if (mp_measure_time) call mpif_mtime('timemanager',1)
879
880end subroutine timemanager
Note: See TracBrowser for help on using the repository browser.
hosted by ZAMG