Articles | Volume 13, issue 11
https://doi.org/10.5194/gmd-13-5567-2020
© Author(s) 2020. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
https://doi.org/10.5194/gmd-13-5567-2020
© Author(s) 2020. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Development and technical paper
| 
13 Nov 2020
Development and technical paper |
| 13 Nov 2020
| 13 Nov 2020
A new end-to-end workflow for the Community Earth System Model (version 2.0) for the Coupled Model Intercomparison Project Phase 6 (CMIP6)
The National Center for Atmospheric Research, Boulder, CO, USA
Alice Bertini
The National Center for Atmospheric Research, Boulder, CO, USA
Gary Strand
The National Center for Atmospheric Research, Boulder, CO, USA
Kevin Paul
The National Center for Atmospheric Research, Boulder, CO, USA
Eric Nienhouse
The National Center for Atmospheric Research, Boulder, CO, USA
John Dennis
The National Center for Atmospheric Research, Boulder, CO, USA
Mariana Vertenstein
The National Center for Atmospheric Research, Boulder, CO, USA
Related authors
Gunther Huebler, Vincent E. Larson, John Dennis, and Sheri Voelz
EGUsphere, https://doi.org/10.5194/egusphere-2025-4435, https://doi.org/10.5194/egusphere-2025-4435, 2025
This preprint is open for discussion and under review for Geoscientific Model Development (GMD).
Short summary
Short summary
Central processing units (CPUs) and graphics processing units (GPUs) are different devices that suit different kinds of work. Using a climate modeling component, we provide a clearer way to tell which device type is faster for a given task. This matters because runs usually use only one device type. Our results are actionable: they guide device choice, report performance gains fairly, highlight code areas to improve, and show how code structure and optimization can change conclusions.
Gunther Huebler, Vincent E. Larson, John Dennis, and Sheri Voelz
EGUsphere, https://doi.org/10.5194/egusphere-2025-4435, https://doi.org/10.5194/egusphere-2025-4435, 2025
This preprint is open for discussion and under review for Geoscientific Model Development (GMD).
Short summary
Short summary
Central processing units (CPUs) and graphics processing units (GPUs) are different devices that suit different kinds of work. Using a climate modeling component, we provide a clearer way to tell which device type is faster for a given task. This matters because runs usually use only one device type. Our results are actionable: they guide device choice, report performance gains fairly, highlight code areas to improve, and show how code structure and optimization can change conclusions.
Fangxuan Ren, Jintai Lin, Chenghao Xu, Jamiu A. Adeniran, Jingxu Wang, Randall V. Martin, Aaron van Donkelaar, Melanie S. Hammer, Larry W. Horowitz, Steven T. Turnock, Naga Oshima, Jie Zhang, Susanne Bauer, Kostas Tsigaridis, Øyvind Seland, Pierre Nabat, David Neubauer, Gary Strand, Twan van Noije, Philippe Le Sager, and Toshihiko Takemura
Geosci. Model Dev., 17, 4821–4836, https://doi.org/10.5194/gmd-17-4821-2024, https://doi.org/10.5194/gmd-17-4821-2024, 2024
Short summary
Short summary
We evaluate the performance of 14 CMIP6 ESMs in simulating total PM2.5 and its 5 components over China during 2000–2014. PM2.5 and its components are underestimated in almost all models, except that black carbon (BC) and sulfate are overestimated in two models, respectively. The underestimation is the largest for organic carbon (OC) and the smallest for BC. Models reproduce the observed spatial pattern for OC, sulfate, nitrate and ammonium well, yet the agreement is poorer for BC.
John T. Fasullo, Jean-Christophe Golaz, Julie M. Caron, Nan Rosenbloom, Gerald A. Meehl, Warren Strand, Sasha Glanville, Samantha Stevenson, Maria Molina, Christine A. Shields, Chengzhu Zhang, James Benedict, Hailong Wang, and Tony Bartoletti
Earth Syst. Dynam., 15, 367–386, https://doi.org/10.5194/esd-15-367-2024, https://doi.org/10.5194/esd-15-367-2024, 2024
Short summary
Short summary
Climate model large ensembles provide a unique and invaluable means for estimating the climate response to external forcing agents and quantify contrasts in model structure. Here, an overview of the Energy Exascale Earth System Model (E3SM) version 2 large ensemble is given along with comparisons to large ensembles from E3SM version 1 and versions 1 and 2 of the Community Earth System Model. The paper provides broad and important context for users of these ensembles.
Andrew Gettelman, Hugh Morrison, Trude Eidhammer, Katherine Thayer-Calder, Jian Sun, Richard Forbes, Zachary McGraw, Jiang Zhu, Trude Storelvmo, and John Dennis
Geosci. Model Dev., 16, 1735–1754, https://doi.org/10.5194/gmd-16-1735-2023, https://doi.org/10.5194/gmd-16-1735-2023, 2023
Short summary
Short summary
Clouds are a critical part of weather and climate prediction. In this work, we document updates and corrections to the description of clouds used in several Earth system models. These updates include the ability to run the scheme on graphics processing units (GPUs), changes to the numerical description of precipitation, and a correction to the ice number. There are big improvements in the computational performance that can be achieved with GPU acceleration.
Stephen G. Yeager, Nan Rosenbloom, Anne A. Glanville, Xian Wu, Isla Simpson, Hui Li, Maria J. Molina, Kristen Krumhardt, Samuel Mogen, Keith Lindsay, Danica Lombardozzi, Will Wieder, Who M. Kim, Jadwiga H. Richter, Matthew Long, Gokhan Danabasoglu, David Bailey, Marika Holland, Nicole Lovenduski, Warren G. Strand, and Teagan King
Geosci. Model Dev., 15, 6451–6493, https://doi.org/10.5194/gmd-15-6451-2022, https://doi.org/10.5194/gmd-15-6451-2022, 2022
Short summary
Short summary
The Earth system changes over a range of time and space scales, and some of these changes are predictable in advance. Short-term weather forecasts are most familiar, but recent work has shown that it is possible to generate useful predictions several seasons or even a decade in advance. This study focuses on predictions over intermediate timescales (up to 24 months in advance) and shows that there is promising potential to forecast a variety of changes in the natural environment.
Ruth Petrie, Sébastien Denvil, Sasha Ames, Guillaume Levavasseur, Sandro Fiore, Chris Allen, Fabrizio Antonio, Katharina Berger, Pierre-Antoine Bretonnière, Luca Cinquini, Eli Dart, Prashanth Dwarakanath, Kelsey Druken, Ben Evans, Laurent Franchistéguy, Sébastien Gardoll, Eric Gerbier, Mark Greenslade, David Hassell, Alan Iwi, Martin Juckes, Stephan Kindermann, Lukasz Lacinski, Maria Mirto, Atef Ben Nasser, Paola Nassisi, Eric Nienhouse, Sergey Nikonov, Alessandra Nuzzo, Clare Richards, Syazwan Ridzwan, Michel Rixen, Kim Serradell, Kate Snow, Ag Stephens, Martina Stockhause, Hans Vahlenkamp, and Rick Wagner
Geosci. Model Dev., 14, 629–644, https://doi.org/10.5194/gmd-14-629-2021, https://doi.org/10.5194/gmd-14-629-2021, 2021
Short summary
Short summary
This paper describes the infrastructure that is used to distribute Coupled Model Intercomparison Project Phase 6 (CMIP6) data around the world for analysis by the climate research community. It is expected that there will be ~20 PB (petabytes) of data available for analysis. The operations team performed a series of preparation "data challenges" to ensure all components of the infrastructure were operational for when the data became available for timely data distribution and subsequent analysis.
Cited articles
Abdulla, G.: Annual Earth System Grid Federation 2019 Progress Report,
available at: https://esgf.llnl.gov/esgf-media/pdf/2019-ESGF-Progress-Report.pdf (last access: November 2020),
2019. a
Atmospheric Diagnostics Results: Atmospheric Diagnostics,
available at: http://webext.cgd.ucar.edu/B1850/PMIP4/atm/b.e21.B1850.f09_g17.PMIP4-midHolo.001.90_109-b.e21.B1850.f09_g17.CMIP6-piControl.001.614_633/ (last access: November 2020),
2019. a
Bertini, A. and Mickelson, S.: CESM Postprocessing (verison 2.2.1),
https://doi.org/10.5065/4XV0-FG55, 2019. a, b
Cheyenne: Computational and Information Systems Laboratory, Cheyenne: HPE/SGI
ICE XA System (Climate Simulation Laboratory), National Center
for Atmospheric Research, Boulder, CO, https://doi.org/10.5065/D6RX99HX, 2017. a
Cinquini, L., Crichton, D., Mattmann, C., Harney, J., Shipman, G., Wang, F.,
Ananthakrishnan, R., Miller, N., Denvil, S., Morgan, M., Pobre, Z., Bell,
G. M., Doutriaux, C., Drach, R., Williams, D., Kershaw, P., Pascoe, S.,
Gonzalez, E., Fiore, S., and Schweitzer, R.: The Earth System Grid
Federation: An open infrastructure for access to distributed geospatial data,
Future Gener. Comp. Sy., 36, 400–417,
https://doi.org/10.1016/j.future.2013.07.002, 2014. a
Daily, J.: pagoda, available at: https://github.com/jeffdaily/pagoda (last access: November 2020), 2013. a
Dalcin, L.: MPI for Python,
available at: https://mpi4py.readthedocs.io/en/stable/ (last access: November 2020), 2019. a
Danabasoglu, G., Lamarque, J. F., Bachmeister, J., Bailey, D. A., DuVivier, A. K., Edwards, J.,
Emmons, L. K., Fasullo, J., Garcia, R., Gettelman, A., Hannay, C., Holland, M. M., Large, W.
G., Lawrence, D. M., Lenaerts, J. T. M., Lindsay, K., Lipscomb, W. H., Mills, M. J., Neale, R.,
Oleson, K. W., Otto-Bliesner, B., Phillips, A. S., Sacks, W., Tilmes, S., van Kampenhout, L.,
Vertenstein, M., Bertini, A., Dennis, J., Deser, C., Fischer, C., Fox-Kember, B., Kay, J. E.,
Kinnison, D., Kushner, P. J., Long, M. C., Mickelson, S., Moore, J. K., Nienhouse, E., Polvani,
L., Rasch, P. J., and Strand, W. G.: CESM-release-cesm2.1.1 (Version release-cesm2.1.1), Zenodo, https://doi.org/10.5281/zenodo.3895315, 2019. a
Danabasoglu, G., Lamarque, J. F., Bachmeister, J., Bailey, D. A., DuVivier, A. K., Edwards, J.,
Emmons, L. K., Fasullo, J., Garcia, R., Gettelman, A., Hannay, C., Holland, M. M., Large, W.
G., Lawrence, D. M., Lenaerts, J. T. M., Lindsay, K., Lipscomb, W. H., Mills, M. J., Neale, R.,
Oleson, K. W., Otto-Bliesner, B., Phillips, A. S., Sacks, W., Tilmes, S., van Kampenhout, L.,
Vertenstein, M., Bertini, A., Dennis, J., Deser, C., Fischer, C., Fox-Kember, B., Kay, J. E.,
Kinnison, D., Kushner, P. J., Long, M. C., Mickelson, S., Moore, J. K., Nienhouse, E., Polvani,
L., Rasch, P. J., and Strand, W. G.: CESM-release-cesm2.1.0 (Version release-cesm2.1.0), Zenodo, https://doi.org/10.5281/zenodo.3895306, 2018. a
Danabasoglu, G., Lamarque, J. F., Bachmeister, J., Bailey, D. A., DuVivier, A. K., Edwards, J.,
Emmons, L. K., Fasullo, J., Garcia, R., Gettelman, A., Hannay, C., Holland, M. M., Large, W.
G., Lawrence, D. M., Lenaerts, J. T. M., Lindsay, K., Lipscomb, W. H., Mills, M. J., Neale, R.,
Oleson, K. W., Otto-Bliesner, B., Phillips, A. S., Sacks, W., Tilmes, S., van Kampenhout, L.,
Vertenstein, M., Bertini, A., Dennis, J., Deser, C., Fischer, C., Fox-Kember, B., Kay, J. E.,
Kinnison, D., Kushner, P. J., Long, M. C., Mickelson, S., Moore, J. K., Nienhouse, E., Polvani,
L., Rasch, P. J., and Strand, W. G.: CESM-release-cesm2.1.2 (Version release-cesm2.1.2), Zenodo, https://doi.org/10.5281/zenodo.3895328, 2020. a
Danabasoglu, G., Lamarque, J. F., Bachmeister, J., Bailey, D. A., DuVivier,
A. K., Edwards, J., Emmons, L. K., Fasullo, J., Garcia, R., Gettelman, A.,
Hannay, C., Holland, M. M., Large, W. G., Lawrence, D. M., Lenaerts, J.
T. M., Lindsay, K., Lipscomb, W. H., Mills, M. J., Neale, R., Oleson, K. W.,
Otto-Bliesner, B., Phillips, A. S., Sacks, W., Tilmes, S., van Kampenhout,
L., Vertenstein, M., Bertini, A., Dennis, J., Deser, C., Fischer, C.,
Fox-Kember, B., Kay, J. E., Kinnison, D., Kushner, P. J., Long, M. C.,
Mickelson, S., Moore, J. K., Nienhouse, E., Polvani, L., Rasch, P. J., and
Strand, W. G.: The Community Earth System Model version 2 (CESM2), J.
Adv. Model. Earth Syst., 12, 1–35, https://doi.org/10.1029/2019MS001916, 2020c. a, b
Deser, C. and Sun, L.: Atmospheric circulation response to Arctic sea ice loss:
sensitivity to background SSTs, in: AGU Fall Meeting Abstracts, vol. 2019,
pp. A51A–03,
available at: https://ui.adsabs.harvard.edu/abs/2019AGUFM.A51A..03D (last access: November 2020), 2019. a
ES-DOC Model: ES-DOC Model,
available at: https://explore.es-doc.org/cmip6/models/ncar/cesm2, last access: November 2020. a
Eyring, V., Bony, S., Meehl, G. A., Senior, C. A., Stevens, B., Stouffer, R. J., and Taylor, K. E.: Overview of the Coupled Model Intercomparison Project Phase 6 (CMIP6) experimental design and organization, Geosci. Model Dev., 9, 1937–1958, https://doi.org/10.5194/gmd-9-1937-2016, 2016. a
Gropp, W., Lusk, E., and Skjellum, A.: Using MPI: portable parallel programming with the
message-passing interface, MIT Press, Cambridge, Massachusetts, London, UK, 1999. a
Harrop, C.: Rocoto, Zenodo, https://doi.org/10.5281/zenodo.890939, 2017. a
Hurrell, J. W., Holland, M., Gent, P., Ghan, S., Kay, K., Kushner, P., Lamrque,
J.-F., Large, W., Lawrence, D., Lindsay, K., Lipscomb, W., Long, M.,
Mahowald, N., Marsh, D., Neale, R., Rasch, P., Vavrus, S., Vertenstein, M.,
Bader, D., Collins, W., Hack, J., Kiehl, J., and Marshall, S.: The Community
Earth System Model: A Framework for Collaborative Research, B.
Am. Meteorol. Soc., 94, 1339–1360,
https://doi.org/10.1175/BAMS-D-12-00121.1, 2013. a
Jacob, R., Krishna, J., Xu, X., Mickelson, S., Tautges, T., Wilde,
M., Latham, R., Foster, I., Ross, R., Hereld, M., Larson, J.,
Bochev, P., Peterson, K., Taylor, M., Schuchardt, K., Yin, J.,
Middleton, D., Haley, M., Brown, D., Huang, W., Shea, D.,
Brownrigg, R., Vertenstein, M., Ma, K., and Xie, J.: Abstract:
Bringing Task and Data Parallelism to Analysis of Climate Model Output, in:
2012 SC Companion: High Performance Computing, Networking Storage and
Analysis, 1493–1494, https://doi.org/10.1109/SC.Companion.2012.282, 2012. a
Juckes, M., Taylor, K. E., Durack, P. J., Lawrence, B., Mizielinski, M. S., Pamment, A., Peterschmitt, J.-Y., Rixen, M., and Sénési, S.: The CMIP6 Data Request (DREQ, version 01.00.31), Geosci. Model Dev., 13, 201–224, https://doi.org/10.5194/gmd-13-201-2020, 2020. a, b
Kornblueh, L., Mueller, R., and Schulzweida, U.: Climate Data Operators,
available at: https://code.mpimet.mpg.de/projects/cdo/ (last access: November 2020), 2019. a
Land Diagnostics Results: Land Diagnostics,
available at: http://webext.cgd.ucar.edu/B1850/b.e21.B1850.f09_g17.PMIP4-midHolo.001/lnd/b.e21.B1850.f09_g17.PMIP4-midHolo.001.601_700-b.e21.B1850.f09_g17.CMIP6-piControl.001.1101_1200/setsIndex.html (last access: November 2020),
2019. a
Mannik, L.: Novel Database and Usage Analytics for the CESM2 Climate Model:
First Steps to Tracking Configuration and Performance,
available at: https://www2.cisl.ucar.edu/siparcs-2019-mannik (last access: November 2020), 2019. a
Meurdesoif, Y.: XIOS, available at: https://forge.ipsl.jussieu.fr/ioserver, last access: November 2020. a
Mickelson, S.: CESM PyConform Input Example, Zenodo, https://doi.org/10.5281/zenodo.3983646,
2019a. a, b
Mickelson, S.: CESM Workflow Documentation,
available at: https://cesm-wf-documentation.readthedocs.io/en/latest/ (last access: November 2020),
2019b. a
Mickelson, S.: Example Cylc Description File,
available at: https://github.com/NCAR/CESM-WF/blob/master/example_cylc_wf/suite.rc, last access: November 2020b. a
Mickelson, S., Paul, K., and Dennis, J.: PyAverager (version 0.9.16),
https://doi.org/10.5065/9zx1-jq74, 2018. a, b
Mickelson, S., Altuntas, A., Bertini, A., Benedict, J., Coleman, D., Fasullo,
J., Feng, R., Hannay, C., Lawrence, P., Lindsay, K., Medeiros, B., Mills, M.,
Oleson, K., Rosenbloom, N., Strand, G., Sun, L., Thayer-Calder, K., Tilmes,
S., and Tomas, R.: CESM CMIP6 Cylc Suites, https://doi.org/10.5281/zenodo.3983653, 2020. a
NCL: NCL, https://doi.org/10.5065/D6WD3XH5,
2019. a
Ocean Diagnostics Results: Ocean Diagnostics,
available at: http://webext.cgd.ucar.edu/B1850/b.e21.B1850.f09_g17.PMIP4-midHolo.001/ocn/diag_work.601.700/MODEL_VS_CONTROL_b.e21.B1850.f09_g17.CMIP6-piControl.001/ (last access: November 2020),
2019. a
Oliphant, T.: Python for Scientific Computing, Comput. Sci.
Eng., 9, 10–20, https://doi.org/10.1109/MCSE.2007.58, 2007. a
Oliver, H., Shin, M., and Sanders, O.: Cylc: A Workflow Engine for Cycling
Systems, J. Open Source Softw., 3, 737, https://doi.org/10.21105/joss.00737,
2018. a, b
Pascoe, C., Lawrence, B. N., Guilyardi, E., Juckes, M., and Taylor, K. E.: Documenting numerical experiments in support of the Coupled Model Intercomparison Project Phase 6 (CMIP6), Geosci. Model Dev., 13, 2149–2167, https://doi.org/10.5194/gmd-13-2149-2020, 2020. a
Paul, K., Mickelson, S., Dennis, J. M., Xu, H., and Brown, D.: Light-weight
parallel Python tools for earth system modeling workflows, 2015 IEEE
International Conference on Big Data (Big Data), Santa Clara, CA, pp.
1985–1994, https://doi.org/10.1109/BigData.2015.7363979, 2015. a, b
Paul, K., Mickelson, S., and Dennis, J. M.: A new parallel python tool for the
standardization of earth system model data, 2016 IEEE International
Conference on Big Data (Big Data), Washington, D.C., pp. 2953–2959,
https://doi.org/10.1109/BigData.2016.7840946, 2016. a
Paul, K., Mickelson, S., Dennis, J., and Hu, H.: PyReshaper (version 1.0.6),
https://doi.org/10.5065/b92r-gt40, 2018. a, b
Paul, K., Mickelson, S., and Dennis, J.: PyConform (version 0.2.8),
https://doi.org/10.5065/9n3z-7x72, 2019. a, b
Perez, F., Granger, B., and Hunter, J. D.: Python: An Ecosystem for Scientific
Computing, Comput. Sci. Eng., 13, 13–21,
https://doi.org/10.1109/MCSE.2010.119, 2011. a
Petrie, R., Denvil, S., Ames, S., Levavasseur, G., Fiore, S., Allen, C., Antonio, F., Berger, K., Bretonnière, P.-A., Cinquini, L., Dart, E., Dwarakanath, P., Druken, K., Evans, B., Franchistéguy, L., Gardoll, S., Gerbier, E., Greenslade, M., Hassell, D., Iwi, A., Juckes, M., Kindermann, S., Lacinski, L., Mirto, M., Nasser, A. B., Nassisi, P., Nienhouse, E., Nikonov, S., Nuzzo, A., Richards, C., Ridzwan, S., Rixen, M., Serradell, K., Snow, K., Stephens, A., Stockhause, M., Vahlenkamp, H., and Wagner, R.: Coordinating an operational data distribution network for CMIP6 data, Geosci. Model Dev. Discuss., https://doi.org/10.5194/gmd-2020-153, in review, 2020. a
PrePARE: PrePARE, available at: https://cmor.llnl.gov/mydoc_cmip6_validator/, last access: November 2020.
a
Sea Ice Diagnostics Results: Sea Ice Diagnostics,
available at: http://webext.cgd.ucar.edu/B1850/b.e21.B1850.f09_g17.PMIP4-midHolo.001/ice/b.e21.B1850.f09_g17.PMIP4-midHolo.001-b.e21.B1850.f09_g17.CMIP6-piControl.001/yrs301-700/ (last access: November 2020),
2019. a
Smith, D. M., Screen, J. A., Deser, C., Cohen, J., Fyfe, J. C., García-Serrano, J., Jung, T., Kattsov, V., Matei, D., Msadek, R., Peings, Y., Sigmond, M., Ukita, J., Yoon, J.-H., and Zhang, X.: The Polar Amplification Model Intercomparison Project (PAMIP) contribution to CMIP6: investigating the causes and consequences of polar amplification, Geosci. Model Dev., 12, 1139–1164, https://doi.org/10.5194/gmd-12-1139-2019, 2019. a
Taylor, K., Juckes, M., Balaji, V., Cinquini, L., Denvil, S., Durack, P.,
Elkington, M., Guilyardi, E., Kharin, S., Lautenschlager, M., Lawrence, B.,
Nadeau, D., and Stockhause, M.: CMIP6 Global Attributes, DRS, Filenames,
Directory Structure, and CVs,
available at: https://www.earthsystemcog.org/site_media/projects/wip/CMIP6_global_attributes_filenames_CVs_v6.2.6.pdf (last access: November 2020),
2017. a
Taylor, K. E., Stouffer, R. J., and Meehl, G. A.: An Overview of CMIP5 and the
Experiment Design, B. Am. Meteorol. Soc., 93,
485–498, https://doi.org/10.1175/BAMS-D-11-00094.1, 2012. a
Williams, D. N.: Visualization and Analysis Tools for Ultrascale Climate Data,
Eos: Earth and Space Science News, 95, 377–378, https://doi.org/10.1002/2014EO420002,
2014. a
Woitaszek, M., Dennis, J. M., and Sines, T. R.: Parallel high-resolution
climate data anslysis using swift, in: MTAGS'11: Proceedings of the 2011 ACM
international workshop on Many task computing on grids and supercomputers, Association for Computing Machinery, New York, NY, USA,
5–14, https://doi.org/10.1145/2132876.2132882, 2011. a
Yellowstone: Yellowstone,
available at: https://www2.cisl.ucar.edu/supercomputer/yellowstone (last access: November 2020), 2017. a
Zender, C.: Analysis of self-describing gridded geoscience data with netCDF
Operators (NCO), Environ. Model. Softw., 23, 1338–1342,
https://doi.org/10.1016/j.envsoft.2008.03.004, 2008. a
Short summary
Every generation of MIP exercises introduces new layers of complexity and an exponential growth in the amount of data requested. CMIP6 required us to develop a new tool chain and forced us to change our methodologies. The new methods discussed in this paper provided us with an 18 times faster speedup over our existing methods. This allowed us to meet our deadlines and we were able to publish more than half a million data sets on the Earth System Grid Federation (ESGF) for the CMIP6 project.
Every generation of MIP exercises introduces new layers of complexity and an exponential growth...
