the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Robust handling of extremes in quantile mapping – "Murder your darlings"
Abstract. Quantile mapping is a method often used for bias adjustment of climate model data toward a reference, i.e. to construct a transformation of the model’s distribution to that of the reference. The main moments of the distributions are typically well transformed by quantile mapping, but statistical uncertainty increases towards the extreme tails, making robust transformations challenging. Because of the limited data at the extreme tails, also an empirical quantile mapping needs to make some estimation or fit a parameterized function for data beyond the calibration data range. The MIdAS bias adjustment platform is here employed to explore different methods for handling the extreme tail, which are evaluated using an indicator for extreme precipitation - the maximum daily precipitation amount per year. Different methodologies are evaluated for a large ensemble of regional climate model projections over Scandinavia. The sensitivity of the empirical quantile mapping for the tails of the distribution is demonstrated, and it is found that the behaviour is significantly different within and without the calibration period, causing severe issues with the temporal consistency of the timeseries. The sensitivity is identified to be due to differences in the activated features of the bias adjustment, within the calibration period where the empirical transfer function is applied, and outside that period, where the extrapolation method is likely applied. This means that the bias adjustment method is in a sense different between different time periods. Currently MIdAS uses separate calibrations for each day of the year, as opposed to e.g. for each calendar month, further aggravates this issue. Further, finding a robust parametrisation for the tail is not straightforward. We identify a two-step solution which works well for this problem: (i) "murder your darlings" by excluding data from the tail data in the calibration period, the extrapolation feature is activated for all time periods, even the calibration period, and (ii) applying an outlier insensitive method for linear regression works well for finding an extrapolation parametrisation for the tail.
- Preprint
(2522 KB) - Metadata XML
- BibTeX
- EndNote
Status: final response (author comments only)
-
CEC1: 'Comment on gmd-2024-98', Juan Antonio Añel, 20 Jun 2024
Dear authors,
Unfortunately, after checking your manuscript, it has come to our attention that it does not comply with our "Code and Data Policy".
https://www.geoscientific-model-development.net/policies/code_and_data_policy.html
You have archived your code on a git repository which is on a server not suitable for scientific publication. Also, you have to publish and share openly all the data used in your study, not only the data to recreate the plots. This includes the SMHIGridClim and Euro-CORDEX CMIP5 data used. We understand that some files used in your study are large (e.g., full output from models). In such cases, instead of storing the complete files, you should at least keep the variables or final fields computed and used in your work.Also, you must include in a potentially reviewed version of your manuscript the modified 'Code and Data Availability' section, DOIs and links for the new code and data repositories.
Please, reply as soon as possible to this comment with the link and DOIs for it so that it is available for the peer-review process, as it should be.
Please, be aware that failing to comply promptly with this request could result in rejecting your manuscript for publication.Juan A. Añel
Geosci. Model Dev. Executive EditorCitation: https://doi.org/10.5194/gmd-2024-98-CEC1 -
AC1: 'Reply on CEC1', Peter Berg, 28 Jun 2024
Dear Juan A. Añel,
thanks for the clarifications of the requirements of the journal, and please accept our sincere apologies for not fulfilling them in the submitted manuscript. We have now updated the repository of the paper, and where we formerly only had the information necessary to reproduce the final analysis and figures of the paper, we have now complemented will all the raw timeseries needed to reproduce the experiments, a zip-file containing the published MIdAS souce code, as well as instructions for how to perform the experiments. Thus, we are now confident that we fulfil the requirements, and the manuscript would only need to be updated with a new doi-link for the Zenodo repository https://zenodo.org/records/12570891 as follows:
@techreport{Berg2024zenodo,
author = {Peter Berg and Johan Södling},
title = {MIdAS bias adjustment of extremes using Theil-Sen extrapolation: Data and plotting scripts for GMD-publication},
institution = {Zenodo},
year = {2024},
doi = {https://doi.org/10.5281/zenodo.12570891}
}We hope that the review process will not suffer from this, and that the link will be forwarded to the reviewers.
With best regards,
Peter Berg and co-authors
Citation: https://doi.org/10.5194/gmd-2024-98-AC1
-
AC1: 'Reply on CEC1', Peter Berg, 28 Jun 2024
-
RC1: 'Comment on gmd-2024-98', Anonymous Referee #1, 24 Jun 2024
Review of gmd-2024-98 manuscript ‘Robust handling of extremes in quantile mapping - “Murder your darlings”’ by Berg et al.
General comments:
The paper gives an interesting insight into the special problems of extreme-value extrapolations/bias corrections. The paper identifies a real problem in the MIdAS method which is also important for efforts not based on that specific software. A series of tests were performed and new approaches are suggested.
Important paper that should be published just to help users of MIdAS. The ideas in the paper will also provide food for thought for general practitioners in the field of climate-change projection; especially at the extremes end of the distributions.
Specific comments:
The paper is set in a language attuned to people who use MIdAS, which is fine for that community, as well as accessible for people in the relevant areas.
I’d like to see a sentence or two more on the subject in Line 71 ‘conservatively remapped’ - how?
Technical corrections:
Line 22: ‘a reference data’ -> perhaps just ‘reference data’
Line 23: ‘in worst case’ -> ‘in the worst case’
Line 31: ‘will act differently’ - ‘will’? - perhaps true. Seems a strong statement.
Line 81: ‘main figures’ -> ‘main Figures’
Figure 1 caption: ‘absolute values’? Do you mean numerical absolute, or a more general phrase relating to ‘levels’ (as opposed to anomalies)?
Line 87: ‘due to the ensemble mean’ … elaborate that thought, perhaps?
Figure 2 legend in both panels, but especially the lower panel, obscures the graphs.
Line 103: ‘forces the’ -> ‘force the’, as ‘experiments’ is plural.
Line 108 was hard for this reviewer to parse.-
AC2: 'Reply on RC1', Peter Berg, 28 Jun 2024
Thank you very much for the positive review,
we would like to clarify that the presented issue is not specific for the MIdAS implementation of quantile mapping, but is inherent to all empirical quantile mapping methods and is a fundamental issue that needs to be dealt with. It is, however, not easily discovered due to the large noise levels, which is why we used a large ensemble of models and averages over a larger domain to visualize the issue.
The conservative remapping was performed using the CDO function 'remapcon' that follows the SCRIP convention. We will provide more details on this in a revised manuscript, along with your technical corrections which are much appreciated.
With best regards,
Peter Berg and co-authors
Citation: https://doi.org/10.5194/gmd-2024-98-AC2
-
AC2: 'Reply on RC1', Peter Berg, 28 Jun 2024
-
RC2: 'Comment on gmd-2024-98', Anonymous Referee #2, 26 Aug 2024
I liked this paper. It is well written, succinct, has a catchy title, and makes an important point. My one complaint – and it is a big complaint – is that this paper only considers linear extrapolation. My intuition is that you wouldn’t need to “murder your darlings” if you fit an extreme value distribution to the tail of the data. I would strongly recommend adding the results from such an implementation to Fig. 2 and Fig 3. I’d be ok accepting the paper without this addition, however, since it already stands on its own as a cautionary tale.
One other big-picture thought is that the efficacy of your extrapolation method at reproducing the real world can be assessed by removing the top 5 percent of your training data and confirming that your extrapolation method correctly predicts that top 5% during the training period. Fortuitously, your Fig 2b shows this in the sense that MIdAS minus origGCM (teal line) by construction handles the top 5% properly during the training period and R5T5 (red line) is what happens when you throw out the top 5% of the data and try to extrapolate it instead. The difference between the teal line and the red line represents the failure of your extrapolation to produce the correct extreme values. It doesn’t look to me like your method is doing a very good job; again, I’m curious whether an extreme value distribution fit would perform better.
Other than these big-picture complaints, I have only minor complaints about wording:
- L14: missing word: “for each calendar month, WHICH further aggravates this issue”
- I think it would be useful if you define quantile mapping in the introduction. This will make your paper accessible to people who are just learning about bias correction.
- L33-34: “…will not be the same method as that which was calibrated…” Could you rewrite this sentence? I don’t understand it at all.
- L35: “One can force the bias adjustment to apply its full range of methods only by…” – I don’t think “methods” is the right word?
- ~L47: I think it would be useful to write out the equation for your linear spline implementation.
- L52: “31 d are used to build the distribution of the reference and model data” – I think you mean 31 days times 29 years of samples, right?
- L65: Since you always use Theil-Sen regression for the top 5% of the data, you could remove the “R5” part of your T%R5 names if you want.
- L73: put an “s” on the end of “unique combination”
- Figs 1-3: in panel b, it would be useful if the y label could be clear that this is a difference in annual maxima.
- L87: “interannual variability is reduced due to the ensemble mean for the model data” – it may be worth explicitly clarifying that reduced variance in no way indicates that the quantile mapping is wrong – it is due to averaging realizations once the remapping is done.
- I’d like to know a bit more about why the original MIdAS approach causes bias corrected future predictions to look ~identical to the original data. It seems like this would only happen if the slopes you’re using for extrapolation are ~identical for the model data and the observation data, which doesn’t seem assured. Could you say more about this?
- ~L63: you could be more clear that you are doing a *single* Thiel-Senn regression using all data from the top 5% of the remaining data.
- On L123 you say that changed slope is a “negative impact” but the following sentence says that whether it is good or bad is up for debate. This is inconsistent.
- L131: The second sentence of this bullet (“To create a robust sample size the bias should be assessed over an ensemble of simulations and/or over a larger set of gridcells”) is a separate thought and should have its own bullet.
Citation: https://doi.org/10.5194/gmd-2024-98-RC2 -
AC3: 'Reply on RC2', Peter Berg, 03 Sep 2024
Thank you very much for the positive review. Here follows a general response to the two main points brought forward, with some additional analysis of the data. We will elaborate more on these points in the revised paper.
Regarding the use of extreme value theory: There is a background philosophy of MIdAS that was not well stressed in the current paper, but more so in the original reference. The philosophy is to have a method that is well applicable across the globe and across seasons. As soon as a distribution is assumed, it becomes problematic to an operational method to follow up on cases where the distribution is not justified. This can occur for different regional climates, or for specific seasons and climate regimes. For this reason, we develop the MIdAS code to be transparent and preferably of low-complexity. The linear model (in the QQ-plot), implicitly assumes that the climate model performs generally in line with the reference data for the tail, although the magnitudes might differ, i.e. a linear offset in the QQ-plot.
The main reason for not using a more standard extreme value theory, such as generalized extreme value or peak-over-threshold, is because of the added complexity it implies for the implementation, the transparency of which distribution (and parameter set) that is used, and the goodness-of-fit in different locations and times. We believe that such approaches will introduce added uncertainty in larger production jobs, and the more simple linear method is preferred, although it might in some occasions have worse performance.
Regarding the performance of the linear fit to the tail: The reviewer makes a good point, and indeed the new methods with a linear fit “RxT5” do in general perform worse than the original “MIdAS” method for the reference period, as shown in the attached figure 1. We have made an analysis on how well the different versions perform for the annual maxima, which is shown in the figure below. In this figure we compare the mean annual maxima for the calibration period with the reference data, and make a box plot for all the ensemble members. Clearly, there is a systematic underestimation in the Theil-Sen methods compared to the original MIdAS method, but not far away from the original empirical method by less than 1 mm d-1 (or less than 0.5% relative bias). Thus, we argue that the linear assumption is acceptable, seen also in light of the answer to the first topic above.
The detailed comments are much appreciated and will be incorporated in the revised manuscript.
With best regards,
Peter Berg and co-authors
Figure caption (attached): The mean annual maxima calculated for the calibration period, and presented as absolute bias (mm d-1) from the reference data (SMHIgridclim), as box-plots of all ensemble members
Viewed
HTML | XML | Total | BibTeX | EndNote | |
---|---|---|---|---|---|
325 | 93 | 26 | 444 | 12 | 12 |
- HTML: 325
- PDF: 93
- XML: 26
- Total: 444
- BibTeX: 12
- EndNote: 12
Viewed (geographical distribution)
Country | # | Views | % |
---|
Total: | 0 |
HTML: | 0 |
PDF: | 0 |
XML: | 0 |
- 1