SKRIPS v1.0: a regional coupled ocean–atmosphere modeling framework (MITgcm–WRF) using ESMF/NUOPC, description and preliminary results for the Red Sea

Sun, Rui; Subramanian, Aneesh C.; Miller, Arthur J.; Mazloff, Matthew R.; Hoteit, Ibrahim; Cornuelle, Bruce D.

doi:https://doi.org/10.5194/gmd-12-4221-2019

Articles | Volume 12, issue 10

https://doi.org/10.5194/gmd-12-4221-2019

© Author(s) 2019. This work is distributed under
the Creative Commons Attribution 4.0 License.

https://doi.org/10.5194/gmd-12-4221-2019

© Author(s) 2019. This work is distributed under
the Creative Commons Attribution 4.0 License.

Articles | Volume 12, issue 10

Development and technical paper

|

08 Oct 2019

Development and technical paper |

| 08 Oct 2019

SKRIPS v1.0: a regional coupled ocean–atmosphere modeling framework (MITgcm–WRF) using ESMF/NUOPC, description and preliminary results for the Red Sea

Rui Sun, Aneesh C. Subramanian, Arthur J. Miller, Matthew R. Mazloff, Ibrahim Hoteit, and Bruce D. Cornuelle

Download

Final revised paper (published on 08 Oct 2019)
Preprint (discussion started on 29 Nov 2018)

Interactive discussion

Status: closed

AC: Author comment | RC: Referee comment | SC: Short comment | EC: Editor comment

- Printer-friendly version

- Supplement

SC1: 'Executive Editor comment on gmd-2018-252', Astrid Kerkweg, 18 Dec 2018
- AC1: 'Reply to executive editor (SC1)', Rui Sun, 17 Apr 2019
- AC4: 'Annotated manuscript', Rui Sun, 17 Apr 2019
RC1: 'Review of “A regional coupled ocean–atmosphere modeling framework (MITgcm–WRF) using ESMF/NUOPC: description and preliminary results for the Red Sea”, by Sun et al.', Dave Gill, 19 Jan 2019
- AC2: 'Reply to Reviewer Comment 1 (RC1)', Rui Sun, 17 Apr 2019
- AC4: 'Annotated manuscript', Rui Sun, 17 Apr 2019
RC2: 'Review of the manuscript', Anonymous Referee #2, 22 Jan 2019
- AC3: 'Reply to Reviewer Comment 2 (RC2)', Rui Sun, 17 Apr 2019
- AC4: 'Annotated manuscript', Rui Sun, 17 Apr 2019
AC4: 'Annotated manuscript', Rui Sun, 17 Apr 2019
AC5: 'Revised manuscript (clean version)', Rui Sun, 17 Apr 2019

Peer-review completion

AR: Author's response | RR: Referee report | ED: Editor decision

AR by Rui Sun on behalf of the Authors (17 Apr 2019) Author's response Manuscript

ED: Reconsider after major revisions (18 Apr 2019) by Olivier Marti

AR by Rui Sun on behalf of the Authors (30 Apr 2019) Author's response Manuscript

ED: Referee Nomination & Report Request started (10 May 2019) by Olivier Marti

RR by Dave Gill (27 May 2019)

RR by Anonymous Referee #2 (04 Jun 2019)

Suggestions for revision or reasons for rejection

The authors of the manuscript coupled an atmospheric model with an oceanic model using ESMF infrastructure. They evaluate the model performance during June 2012, which contained three heat waves over the Arabian Peninsula and the Red Sea. The model simulation lasted one month. The skill was better during the earlier simulation timeframe.

Major comments
The paper still needs focusing. I am still struggling to see the main point of the paper: was it a technical challenge that had to be overcome, or was it a science question that could finally be answered with the coupled system? Is the system designed for seasonal prediction, hence a 30-day run? If it is focused on the heat waves, why is the bulk of evaluation over the Red Sea? When resolving a phenomenon that goes through a diurnal cycle, such as air temperature at 2 meter height, isn’t it imperative that the boundary conditions can resolve the diurnal cycle? Obtaining daily HYCOM values completely misses the diurnal variability, and SST change can be up to a few degrees Celsius, affecting surface heat fluxes.
Please re-read the manuscript carefully before submitting it. The second version still contains typos.
Minor comments
P1 l12: I am not sure if the coupled system has good skill. Some errors are quite big.
P7 l12: The sequential mode might be simpler to start, but the overall performance of the system in your case will suffer – the ocean component has fewer active (water) points on which the computation is performed, thus affecting load balancing. If discussing scalability of the system, this is not the best choice.
P8 l16: If I understand correctly MITgcm also runs on 256x256 grid, of which a large portion are idle (land points)?
P9 l9: The link to HYCOM data is incorrect – it should read ‘/dataserver/’.
P10 Table1: The typo has not been corrected; it still reads ATM.STA instead of ATM.DYN.
P11 l7: For clarity: ‘from the model experiments’, probably better than ‘from various experiments’.
P11 l10: I am not convinced ERA5 air temperature is in good agreement with NCDC ground observations. Figure 6 indicates discrepancies up to 10 degrees Celsius!
P11 l15: Why are T2 differences discussed for hours 36 and 48?
P12 Figure 4: The model is too hot over land at daytime (36 hours) and too cold almost everywhere at night (48 hours). The temperature differences are significant!
P12 l5: The CPL run results are closer to the ERA5 dataset where? Over land, over water?
P12 l8: If SST fields from CPL and ATM.DYN are similar, it does not mean ‘SST in CPL run is tending to be more similar to the realistic’. Please reword.
P12 l13: NCDC has already been defined.
P12 l16: Ground observations and ERA5 are not in good agreement according to Figure 6.
P13 Figure 5: The surface air temperature is persists to be too warm over land at daytime seen at t=9.5 days and t=23.5 days). Care to comment?
P13 l3: What is RMSE difference?
P13 l5: Are you implying that the simulation was not capable of capturing the June 2nd event was not captured in Mecca? I do not understand the meaning of the sentence.
P13 l7: If initial conditions were at fault, the first day maxima would not be captured. Why is the WRF physics scheme in error – it worked on day 1.
P13 l8: Physics schemes do not parameterize extreme events. They parameterize physical processes that can not be explicitly resolved in a numerical model.
P13 l10: …’T2 compare’… should read ‘T2 compared’.
P14 Figure 6: Can you mark the heat event episodes with light gray background?
P14 Figure 6: The discrepancy between ERA5 and observed maxima and minima is troublesome if ERA5 is used to evaluate the model results.
P15 Figure 8: ‘air temperature’ should read ‘air temperature over the Red Sea’. I do not understand why only over water.
P15 Figure 8: Can you comment on the bias and rmse time drift of the ATM.STA results?
P16 l6: It appears that the model captures the general meridional SST gradient in the Red Sea. But the actual temperature differences are quite large (Figure 9).
P16 l14: I am afraid that daily data is not frequent enough to study diurnal phenomena. I am not familiar with HYCOM data on the aforementioned server, but I would be surprised if a global ocean model provides fields only once a day.
P17 l5: ‘This is because the HYCOM data is cooler than GHRSST’. Have you considered the difference in HYCOM bulk SST at the topmost model level, and the actual skin SST measured by satellites, reported by GHRSST? Also, the temperatures might be lower, but they don’t make the data cooler.
P17 l11: Why are surface heat fluxes evaluated only over the Red Sea?
P18 l3: CPL and ATM.DYN runs exhibit more latent heat fluxes coming out of the ocean. Why? And reword please.
P18 l6: Latent heat, or latent heat fluxes (three times).
P18 l8: Why is latent heat flux adequate close to the coastline?
P18 l10: I am not convinced the short and longwave radiation fluxes are captured reasonably well. According to Figure 12, the differences are almost up to 30% of the total forcing.
P18 l15: Yes, the differences are almost 200 W/m2!
P18 l17: Points with discrepancies can be masked out to help with the evaluation.
P18 l21: I do not agree with the conclusion that ‘overall the CPL simulations are capable of well capturing all the components of the surface heat fluxes’.
P18 l24: How is latent heat flux different from evaporation?
P18 l24: Surface winds are winds at 10 meters? Are there any wind observations in the NCDC data? Why not compare sea level pressure? If the pressure field is not correct, that could explain the discrepancy in winds.
P18 l32: Please reword: ‘is small than’.
P21 l5: Please reword: ‘This is because the CPL run over-estimated the SST than the ATM.DYN run’.
P21 l7: Did the model create any precipitation? If there is none observed is not a sufficient reason not to include it in the discussion.
P22 l9: What are overlap cells? Why does it matter they are 25% of the total cells?
P22 l14: Fluctuations in computational time might be resolved by performing a few runs and averaging the execution times. I am still convinced that idle ocean points when using many cores are seriously affecting load balancing and thus overall performance when using the system sequentially.
P23 l9: ‘We hypothesize’ not ‘we hypothesis’.
P23 l11: Your results suggest that using more cores increases the cost of coupling. That is worrysome. Is there an upper limit to this cost?
P24 Table 3: For a clean evaluation of ESMF cost, switch off I/O, because it is handled by each component. Also, use the built-in ESMF Clock, which will tell you exactly where time is being spent.

Hide

ED: Reconsider after major revisions (11 Jun 2019) by Olivier Marti

AR by Rui Sun on behalf of the Authors (31 Jul 2019) Author's response Manuscript

ED: Referee Nomination & Report Request started (31 Jul 2019) by Olivier Marti

RR by Dave Gill (01 Aug 2019)

Suggestions for revision or reasons for rejection

This is a third review of "SKRIPS v1.0: A regional coupled ocean–atmosphere modeling framework (MITgcm–WRF) using ESMF/NUOPC, description and preliminary results for the Red Sea" by Sun et al.

Accept with technical corrections.

This version of the manuscript is more narrowly focused on the implementation and development of the coupled system, with much less emphasis on the validation of a case study. There are a number of locations with improved explanations / descriptions: how the comparison with observations takes place, the purpose for the domain and case study, use and impacts of the validating data, implications of observations and reanalysis data near the coast, and a more clear presentation on scaling and timing performance. Every figure caption has been modified, making them much easier to understand and making them much more standalone.

1) Scientific significance
Does the manuscript represent a substantial contribution to modelling science within the scope of this journal (substantial new concepts, ideas, or methods)?

This is a coupled ocean-atmosphere system, using a state of the art coupler and highly cited numerical models. Significant documentation and support for the individual components exist. This coupled system could substantially lower the bar for users to get into the coupled ocean-atmosphere arena, particularly with users who are already fluent in one of the models.

2) Scientific quality
Are the scientific approach and applied methods valid? Are the results discussed in an appropriate and balanced way (consideration of related work, including appropriate references)? Do the models, technical advances and/or experiments described have the potential to perform calculations leading to significant scientific results?

The reasonable approach to the coupled system would be to test each component separately and then together to determine sensitivity. The authors approached validation in this fashion. They used standard observation and reanalysis to indicate that the coupled system is producing expected results. The test case was long enough to have indicated an incorrect trend. The spatial distribution of errors are explainable and largely expected. Regional climate systems are hampered without access to a coupled ocean-atmosphere system, and this coupled system is directly intended to support that community.

3) Scientific reproducibility
To what extent is the modelling science reproducible? Is the description sufficiently complete and precise to allow reproduction of the science by fellow scientists (traceability of results)?

The authors provided extensive information on the location of component source code repositories, and also to the source of the modified code (to enable the coupled system with existing source code). For the actually running of the system, I think the user community would benefit from more detailed information than what the authors provide: https://scripps-coupled-atmosphere-ocean-model.readthedocs.io/en/latest/tutorial/tutorial_rs.html. This page (these pages) could serve as both a tutorial and a step-by-step procedure to replicate the coupled system results. For example, instead of "generate the mesh", explicit steps could be given.

4) Presentation quality
Are the scientific results and conclusions presented in a clear, concise, and well structured way (number and quality of figures/tables, appropriate use of English language)?

The paper is reasonably well structured. The conclusions draw correctly from the results and discussion. Data that that should be presented in a tabular form for ease of understanding is done so. Several figures suffer from insufficient visual differentiation, but that is similar to most presentations with even a small ensemble. However, as the authors state in comments (not in the paper) "our goal for this work and manuscript is to (1) introduce the design of a newly developed regional coupled ocean-atmosphere modeling system with a state-of-the-art coupler (ESMF with NUOPC), (2) describe the implementation of the modern coupling framework, (3) validate the coupled model using a real-world example, and (4) demonstrate and discuss the parallelization of the coupled model." The "similarness" of a number of figure components points to the authors conclusions that the coupled system is giving results similar to the results of the standalone well-known models.

Technical Corrections:

P1 L12,13
"The coupled model, documentation, and tutorial cases used in this work are available"
What tutorial cases are used in this paper? Either be more specific, or remove this portion of the sentence.

P3 L7
"(Skamarock et al., 2005)."
(Skamarock et al., 2019).

P5 L24
"(Skamarock et al., 2005)"
(Skamarock et al., 2019)

P9 L4,5
"To validate the coupled model, the following sets of simulations using different surface forcings are performed according to the tests (Warner et al., 2010; Turuncoglu et al., 2013; Ricchi et al., 2016)"
Either I don't know what "according to the tests" means, or this should be stated in a more clear fashion. How are these citations related to "according to the tests"?

P10 L1,2
"The ocean model uses the data assimilating HYCOM/NCODA 1/12 global reanalysis data as initial and boundary conditions"
Remove "data assimilating".

P10 L10,11
"Here we used the default width"
Everything else is present tense.

P10 L21
"The simulated SST is also validated against HYCOM/NCODA data to show the increase of the error."
No idea what this means.

P11 L1
http://cdo.ncdc.noaa.gov/CDO/georegion
This link is broken.

P11 L5
"Surface heat fluxes (e.g., latent heat, sensible heat, longwave and shortwave radiations), which drives the oceanic component"
drive

P16 L2
"The SST field snapshots from CPL run on June 2nd and 24th are shown"
Remove "run". The word "run" could be used as a noun or a verb in the sentence, and the meaning of the sentence changes.

P16 L12,13
"To quantitatively compare the errors in SST, the time history of the SST in the simulations (i.e., OCN.DYN and CPL) and validation data (i.e., GHRSST and HYCOM/NCODA) are shown in Fig. 9."
Add a sentence describing the reason for the oscillatory 9a vs 9b (3h vs daily data?).

P17 Fig9b
"Mean SST: CPL 29.28; OC.,DYN 29.29; HYCOM 29.85"
"HYCOM" should be "GHRSST"

P23 L3
"The coupled model is validated by using a realistic application to simulate the heat events in the Red Sea region. "
This is the conclusion so be specific in case readers read only this and the abstract, "The coupled model is validated by using a realistic application to simulate heat events during June 2012 in the Red Sea region."

P23 L15,16
"The CPU time associated with different components of the coupled simulations is also presented, showing the ESMF/NUOPC driver is not a bottleneck in the computation. "
Remove the "is also presented" and state directly "The CPU time associated with different components of the coupled simulations shows the ESMF/NUOPC driver is not a bottleneck in our coupled implementation of SKRIPS. "

P24 L1,2
"Appendix A: Snapshots of Surface Heat Fluxes
The snapshots of the THFs in the simulations at 1200 UTC June 2nd and 24th are presented."
Would you redefine THF in the appendix or do something similar to changing the title of the section to "Snapshots of Surface Turbulent Heat Fluxes".

P24 L2
"The snapshots of the THFs in the simulations at 1200 UTC June 2nd and 24th are presented."
Maybe add "(Fig. A1)" at the end of the sentence. You don't mention A1 until you discuss A3, and then only tangentially.

P31 L33,34
"Skamarock, W. C., Klemp, J. B., Dudhia, J., Gill, D. O., Barker, D. M., Wang, W., and Powers, J. G.: A description of the advanced research
WRF version 2, Tech. rep., National Center For Atmospheric Research Boulder Co Mesoscale and Microscale Meteorology Div, 2005."
Skamarock, W. C., J. B. Klemp, J. Dudhia, D. O. Gill, Z. Liu, J. Berner, W. Wang, J. G. Powers, M. G. Duda, D. M. Barker, and X.-Y. Huang, 2019: A Description of the Advanced Research WRF Version 4. NCAR Tech. Note NCAR/TN-556+STR, 145 pp. doi:10.5065/1dfh-6p97

Referee Report: PDF

Hide

RR by Anonymous Referee #2 (22 Aug 2019)

Suggestions for revision or reasons for rejection

Thank you for substantially improving the manuscript. I have a few comments, which I hope will help you add clarity. Please add in the conclusion that your coupled system gives equal or better results than the standalone components by themselves.
I would suggest adding some explanation when introducing the model evaluation section. Air temperature at 2 m is used to compare CPL with ATM.STA and ATM.DYN, for validation of the atmospheric component. Is the coupled WRF giving similar or better results than the coupled WRF in SKRIPS. SST is similarly used to evaluate the oceanic component, coupled versus standalone. In both cases you see the results are equal or better, which is desirable. Winds and fluxes are used to assess the coupled system.
There are still mistakes in the manuscript that would be picked up by a native speaker (e.g.
P11, L10: Usually the model data is interpolated to validation data points. Is there a good reason you are doing the opposite?
P12, L14: I believe RMSE appears here for the first time, so please move the definition from P14, L3 here.
P13, L2: Replace “warming SST, which warms the T2”, with “a dynamic SST which influences T2”, or something similar. There is warming and cooling of SST in a diurnal cycle which affects T2.
P14, Fig 6: What is ‘v.s.’ – did you mean versus? If so, I believe the short version is ‘vs’. Or simply replace labels with ‘RMSE of maximum temperatures’ and ‘RMSE of minimum temperatures’.
P15, L1: So, you are evaluating capturing the diurnal cycle over land using the three locations (Jeddah, Mecca, Yanbu), and over water using points over the Red Sea. That could be written more concisely.
P17, L12: I believe it is the ‘atmospheric surface heat flux’. Many more instances in the text. Please have a native speaker proof read.
P18, Fig 10: Where is ATM.DYN in the top panel? If it is very similar to CPL, please explain in the text.
P18, L13: Please reword: “because they are initialized in the same way the SST fields are similar”.
P18, L17: Radiations -> Radiation (throughout the text).
P19, L4: Please reword ‘slightly over-estimated’, if the difference can be in excess of 100 W/m^2.
P19, Fig 11: Where is ATM.DYN?
P20, Fig13: Where is ATM.DYN?
P21, L15-25: Did you run the system in sequential or concurrent mode? I think I know.
P21, L25: The cost breakdown among the component should provide you with a good first guess to address load balancing in the concurrent mode.
P21, L27: It is computational cost, not costs. A few more instances on the same page, please have a native speaker proofread the manuscript.
P21, L31-32: Please remove the statement above the ‘purely marine region’. There are many factors determining the cost which you do not explain (e.g. degrees of freedom).
P22, L3: Please replace “is not a bottleneck”, with something like “does not add a significant computational overhead”.
P23, L17: The system has not been used on thousands of cores to suggest it can be used on a supercomputer. In fact, I would argue you showed that the system can be run on a commodity cluster with 500-1000 cores, provided that the problem size is adequate.

Hide

ED: Publish subject to minor revisions (review by editor) (22 Aug 2019) by Olivier Marti

AR by Rui Sun on behalf of the Authors (01 Sep 2019) Author's response Manuscript

ED: Publish as is (04 Sep 2019) by Olivier Marti

AR by Rui Sun on behalf of the Authors (06 Sep 2019)

Short summary

A new regional coupled ocean–atmosphere model, SKRIPS, is developed and presented. The oceanic component is the MITgcm and the atmospheric component is the WRF model. The coupler is implemented using ESMF according to NUOPC protocols. SKRIPS is demonstrated by simulating a series of extreme heat events occurring in the Red Sea region. We show that SKRIPS is capable of performing coupled ocean–atmosphere simulations. In addition, the scalability test shows SKRIPS is computationally efficient.