Overcoming computational challenges to realize meter- to submeter-scale resolution in cloud simulations  using the super-droplet method

Matsushima, Toshiki; Nishizawa, Seiya; Shima, Shin-ichiro

doi:https://doi.org/10.5194/gmd-16-6211-2023

Articles | Volume 16, issue 21

https://doi.org/10.5194/gmd-16-6211-2023

© Author(s) 2023. This work is distributed under
the Creative Commons Attribution 4.0 License.

Special issue:

Particle-based methods for simulating atmospheric aerosol...

https://doi.org/10.5194/gmd-16-6211-2023

© Author(s) 2023. This work is distributed under
the Creative Commons Attribution 4.0 License.

Articles | Volume 16, issue 21

Development and technical paper

|

02 Nov 2023

Development and technical paper |

| 02 Nov 2023

Overcoming computational challenges to realize meter- to submeter-scale resolution in cloud simulations using the super-droplet method

Toshiki Matsushima, Seiya Nishizawa, and Shin-ichiro Shima

Download

Final revised paper (published on 02 Nov 2023)
Supplement to the final revised paper
Preprint (discussion started on 02 Mar 2023)

Interactive discussion

Status: closed

RC1:
'Comment on gmd-2023-26', Anonymous Referee #1, 29 Mar 2023

The comment was uploaded in the form of a supplement: https://gmd.copernicus.org/preprints/gmd-2023-26/gmd-2023-26-RC1-supplement.pdf

Citation: https://doi.org/10.5194/gmd-2023-26-RC1
- AC1: 'Reply on RC1', Toshiki Matsushima, 05 Jun 2023
  
  Thank you for taking the time to read through our manuscript and provide us with positive and helpful comments. Please see the attached document for point-by-point responses to your comments.
  
  Citation: https://doi.org/10.5194/gmd-2023-26-AC1
RC2:
'Comment on gmd-2023-26', Anonymous Referee #2, 28 Apr 2023

In the paper "Optimization and sophistication of the super-droplet method for ultrahigh resolution cloud simulations" the super-particle based warm phase microphysical model of Shima 09 is optimized for SIMD by moving branches out of loops, using low or fixed precision if possible and ensuring that sufficiently long vectors are contiguous in the cache. Additionally a second order conservative interpolation is derived and the SD initialization is revised such that SDs with zero multiplicities are avoid. Then the improved version is compare to the base version and a bin and a bulk scheme. The performance is improved by nearly two orders of magnitude and is sustained for the largest stratocumulus simulation so far. Remaining challenges and potential future work are discussed in detail.

The work is original and is opening the door for DNS-like clouds and precipication studies. The model code has been made public. The paper is well writing and interesting to read. Hence, it should be published after minor revisions.

general comments

Most of the model improvements are of "technical" nature. I am not computer scientist, hence I cannot completely judge these achievements.

Still, the performance achievements seem great. However, I would have liked it even better, if the comparison with traditional bulk method would be made fairer. E.g. in the SDM and bin model there is no "cold" microphysics (yet), but the bulk method contains these tracers. Hence, it is likely that the advection cost could be reduced to 4/10. Or in the time to solution comparison there is no discussion that with the bulk methods larger microphysical timesteps are possible and might be even sufficient to reach convergence. Considering that the SDM-new128 might be more costly than the bulk method. However, I still would want to apply it, as it makes, as stated in the paper, far less assumptions.

It is quite a long paper. Consider whether it can be shortened. Personally, I tend to read and cite paper more often, if they are briefly and succinctly. To provide examples, I would not miss the following parts in main body of the paper:

- discussion about the flow solver, i.e. focus on the microphysics (e.g. microphysics - radiation coupling is also not included only mentioned in L914. It is an interesting question how to couple SDM with radiation but clearly out of the scope of this paper). Of course there are a few details that would need to kept, like that a C grid is used and the effective resolution.

- the very details about Fugaku, e.g. L751ff (At that point in the paper I had forgotten what the eco-boost mode does and had to reread the paper to find it on L201. The portion of readers that have access to Fugaku and hence for which these information are relevant, are probably low).

I read section 3 as if the improvements of the SDM are of technical nature and should not deteriorate the physics. Then I was surprised that in only physical comparison between the SDM-orig and SDM-new in Fig. 6 the difference seem quite substantial around t=1200. You argue with different versions of the flow solver. What change prevents you from using the same version? Then also the discussion in L565ff could be removed. To strengthen the confidence in the SDM-new it would be worth to investigate the difference, their cause and relevance a bit more.

specific comments

2.1 Governing equations:

L109-119: The discussion about anelastic vs fully-compressible equation is a bit confusing and maybe misleading. I understand that global communications can be the performance bottleneck. However, as the authors acknowledge, other groups have successfully managed to gain an overall benefit from the larger timestep possible with the anelatic approximation. The SDM timestep seems comparable to Mellado et al. 2018 dynamics time step, while in this paper the dynamics timestep is 16 times lower. L891ff states another disadvantage.

L132: Mention, that this is not a complete list of the warm phase microphysical processes. For example spontaneous and collisional break-up are not implemented, but are relevant for other meteorological situations with high rain rates. Actually, collisional break-up might be most complicated warm phase process, even if can be more naturally implemented in the SDM framework.

L150: Maybe I missed it, has the size of the collision grid ever been discussed in the context of SDM? If it is kept equal to size of the fluid cell, the interaction distance of SDs decreases with mesh resolution. I could even imagine that it could be a reason why the SD statistics in Sato et al. 2017 did not converge even at 5 m resolution. If you think that is a valid discussion point, it would fit in section 6 at L1077.

L155: A philosophical question: Do you view a super-particle as a sample from the real particles or as a statistical average over the multiplicity of real particles. I would prefer the former as the SDM could converge to the full problem for multiplicity -> 1 (even so that is not true for the O(N) collision algorithm). However, in my view turbulent (subgrid-scale) diffusion on the position space should then be included, otherwise initially very close SDs will not separate. This will prohibit the connection to DNS where two-point statistics are of interest. It seems even worse with low or fixed precision arithmetic (L441).

3.1 Model framework of SCALE

L238: 3D/2D decomposition leads to a lot of data transfer via network (argument against the anelastic approximation in section 2.1). SDM physics contributes only to ~50% runtime attributed to SDM. How much this MPI Wait times compared to other technical work as SD sorting?

L245: Another argument for a 2D decomp for the SDM is that particle sedimentation in z direction does not lead to MPI communications.

3.2 Initialization of super-droplets

L281: Could you make it clearer, why constant multiplicity is more natural? I can only understand that in the limit of multiplicity -> 1. It is important as it serves as motivation to develop the new init scheme.

L285: This question is not completely answered in L825 ff. Add in section 6?

L362-363: cache blocking and moving branches out of loops was already the way to go for vector computers, maybe citations would be appropriate.

3.3.2

L385 Important point, could you motivate the consistency of the densities a bit further. L155 states a turbulence scheme is used for tracers but not for the SDM. see comment to L155

3.3.5

L520 citations seem to be appropriated

L527-534 This is not obvious to me, maybe a sketch would help?

4.1

eq. 10 this metric seems to measure the usage of the available computer power, but ignores convergence, i.e. how many tracers/bins/SDs are actually needed.

L582 A short discussion on how much optimization might be possible might be appropriate and link to the discussion in section 5 L953ff.

L583 This comparison seems indeed unfair, especially as you emphasize in several places that SD-new128 has similar performance (see general comment above)

4.2
L625 Why has BIN a O(N**2) scaling? That implies that advection not dominant. If the reason is collisions, a linear sampling might also be possible. Hence, degree of optimization is hard to judge.

4.3

L761 It do not understand the last sentence.

L783 what do you mean by "only SDs in one cell"?

L814 it is unfortunate that the bulk scheme seems to have a bug or inappropriate threshold. That weakens the conclusion in L823 and L1165 considerably.

5.2

Table 3. tracking + condensation + coalescence only adds up to 141 min of 274 in microphysics. How much is due to the 3D/2D conversion + MPI wait steeming from the flow solver decomposition and how much from the technical workload needed for the microphyiscs itself like the SD sorting?

L953ff bin method O(N**2), tracer advection is not even dominant (see comment to L625

L972 0.8*15*33 /= 363

6.1

L990ff are too optimistic. First, instantaneous freezing and melting assumptions are quite restrictive, e.g. leads to complete neglect of wet growth in deep convection. Mixed-phase regimes -38 to 0 are very important and frequent, e.g. riming of ice particle is a main driver for cold phase precip. Hence, I think the coexistence of water and ice particles will be needed often and extension to mixed-phase particle should be the next step after Shima 09 and Shima 20. In conclusion, I feel that the comparison to mixed-phase bulk is not justified.#f

6.3

L1014ff Online postprocessing seems needed, for statistical analyses DSD on the scale of blocks should be sufficient. Of course the checkpoint/restart problem remains if the simulation time is longer than allowed by the compute queue. However, for such large scale applications it might be possible to discuss such rules with the computing center.

technical corrections

L12: Add something like "possible" or "probably" for GPUs, as the effectiveness is not demonstrated in this paper.

Fig 1. end of caption, dotted lines also in (d)

L355 reformulate the sentence to avoid misinterpretation. E.g. move "except for the collision-coalescence process" to a new sentence or put it in parenthesis.
L482 link to L375?

L661 "resists" might not be the best word

Fig 7. larger symbols in the legend

Citation: https://doi.org/10.5194/gmd-2023-26-RC2
- AC2: 'Reply on RC2', Toshiki Matsushima, 05 Jun 2023
  
  Thank you for taking the time to read through our manuscript and provide us with positive and helpful comments. Please see the attached document for point-by-point responses to your comments.
  
  Citation: https://doi.org/10.5194/gmd-2023-26-AC2

Peer review completion

AR: Author's response | RR: Referee report | ED: Editor decision | EF: Editorial file upload

AR by Toshiki Matsushima on behalf of the Authors (01 Jul 2023) Author's response Author's tracked changes Manuscript

ED: Publish subject to minor revisions (review by editor) (31 Jul 2023) by Simon Unterstrasser

Dear authors,

the reviews of the first round were quite positive and I haven't sent out the manuscript to reviewers again.

Instead, I reviewed the revised version. As I am also using particle-based methods in my research, I have some more in-depth comments.

Clearly, the technical improvements show their effectiveness as you nicely demonstrate the performance increase.
And it is really great what you have achieved in technical terms!!
Nevertheless, sometimes it should be better demonstrated that the physical/numerical improvements have a clear benefit for the physical simulation of the cloud.

1.)
From Fig. 6, I understand, that the SD spatial distribution is affected by the interpolation scheme, but what is the effect of it on the cloud as a whole. You could, e.g., add a panel to Fig. 7 showing the differences between CVI-1 and CVI-2. Other model adaptations should be described similarily to better understand the physical implications of your improvements.

You may even add an additional figure with time series of selected total cloud properties to make the comparison between the different model version more quantitative.

2.)
Moreover, I understand that including SGS velocity needs a lot of memory (in the LCM of Sölch & Kärcher, SGS turbulence on SD movement is included. But clearly, those simulations do not run on such extremely large grids).
But the open question is, if the inclusion of SGS velocity wouldn’t be more important than switching from CVI-1 to CVI-2. How can you demonstrate that such sophisticated methods like CVI-2 are really necessary compared to neglecting GSG turbulence? Wouldn't SGS turbulence lead to more SD mixing and potentially affect your cloud evolution?

3.)
You mention and stress that SD density should be similar to air density.
This is a rather strong statement, which I believe is only valid under several assumptions and for a particular SD initialization (which may not be clear to readers).

The initial SD density may not be linked to air density, instead the multiplicities could be accordingly scaled.
Moreover, one could initialize SDs only in a confined region of the whole domain.
Unterstrasser & Sölch (2014) report about SD splitting and merging. In all those cases, SD density and air density are not linked any longer.

I understand that none of the cases is true in your case.
Yet, your statements reads a bit like it is universally valid.
This should be clarified to avoid any misunderstandings.

In the attached document you can find many more comments on language, plot layout and so on.

Best wishes,
Simon

Comments to the Author: PDF

Hide

AR by Toshiki Matsushima on behalf of the Authors (20 Aug 2023) Author's response Author's tracked changes Manuscript

ED: Publish subject to technical corrections (18 Sep 2023) by Simon Unterstrasser

AR by Toshiki Matsushima on behalf of the Authors (20 Sep 2023) Author's response Manuscript

Short summary

A particle-based cloud model was developed for meter- to submeter-scale resolution in cloud simulations. Our new cloud model's computational performance is superior to a bin method and comparable to a two-moment bulk method. A highlight of this study is the 2 m resolution shallow cloud simulations over an area covering ∼10 km². This model allows for studying turbulence and cloud physics at spatial scales that overlap with those covered by direct numerical simulations and field studies.