Each datapoint in our compiled dataset consists of a binary target variable and the corresponding daily multivariate climate input for one growing season at one grid cell. Due to the trends in yields due to improved management practices and breeding that have occurred over the last decades, researchers often consider yield anomalies (after detrending) rather than their absolute values. We test our method on both non-detrended and detrended yields (based on subtracting a seven-year rolling mean, meaning that we discard datapoints from the first six years), in order to assess the utility of the method for analysis of both relative yield failure and the occurrence of yields below an absolute threshold. The target variable is maize yield failure, which we define as any yield below the 10th percentile observed during the training period at that grid cell. We assess the robustness of our method to this choice by reproducing the results using 5th and 15th percentile thresholds. The dataset is split into a training set (the first 30 % of growing seasons, covering 1902–1937 for non-detrended yields and 1907–1940 for detrended, and all grid cells with latitudes above 35° north) and three test sets: a temporal test set (the last 70 % of growing seasons, covering 1938–2016 and 1941–2016, for grid cells of latitude above 35°), a spatial test set (the first 30 % of growing seasons in grid cells below 35°) and a spatiotemporal test set (the last 70 % of growing seasons in grid cells below 35°). Note that, for non-detrended LPJmL yields, the 10th percentile over the training years was zero for 167 grid cells. These low-yielding areas are removed from the training and test set. After processing, the training sets are made up of 50 000–60 000 datapoints, the temporal test sets consist of 115 000–135 000, the spatial test sets 13 000–18 000, and the spatiotemporal test sets 30 000–39 000.

Our predictive variables derive from the same daily multivariate climate data that was used to drive the crop models. For each grid cell and year, we take 90 days (d) of each climate variable before the planting date (to consider e.g. lagged soil moisture effects) and 240 d after, resulting in a 3300-dimensional dataset (as we have ten climate variables). To use an interpretable model with this data, a smaller number of climate drivers must first be defined (generally, by applying aggregation functions over specified time intervals – for instance, the monthly maximum temperatures or number of growing degree days during the growing season). This feature selection step is normally based on expert knowledge. Instead, our proposed method extracts a small number of simple climate drivers of yield failure from daily multivariate weather data based on their robust predictive power in held-out time periods.

The method for identifying climate drivers consists of three steps (Fig. ): first, we generate 100 pools of candidate features by using a set of chosen aggregations (mean, minimum, maximum, number of days below 0, number of days above the 90th percentile, minimum and maximum 5 d mean) over each daily climate variable from 30 randomly-sampled time intervals (with minimum duration of two weeks) during the growing season; second, we select ten features from each pool using sequential forward feature selection, based on the predictive performance of ML models on held-out time periods; finally, we collect the selected features from the pools, and extract a set of ten condensed climate drivers from that collection using agglomerative clustering, based on the aggregation methods, variables and periods of the growing season which are selected most frequently.

When selecting features from each pool (step 2), we use an extremely randomised trees model (extra trees). This model consists of an ensemble of decision trees, similar to the more widely-known random forest, but rather than identifying the optimal split value for each feature at each node, the thresholds are selected at random. Extra trees models are therefore faster to train, and can result in smoother functional relationships when predictions are averaged across trees, leading to improved out-of-sample performance . Like other tree-based models, they handle nonlinear relationships well, are robust to outliers and skewed distributions, are computationally efficient, and can be trained in parallel. We do not tune hyperparameters; all models used consist of 50 trees of maximum depth 30. At each split, half of the predictive variables are considered, and each leaf node must contain a minimum weighted fraction of 0.0001 of all samples (with samples weighted inversely proportionally to class frequency). Features are selected sequentially based on optimising model performance using ten-fold temporal cross-validation, with each test fold consisting of a held-out period of three or four consecutive years.

To obtain a final set of ten climate drivers of yield failure we apply agglomerative clustering to the selected features from 100 pools of candidates (step 3). Features are only clustered together if they substantially overlap in time and use the same aggregation method and climate variable. The ten largest clusters are then condensed into single drivers by using the 10th percentile of starting timepoints and 90th percentile of ending timepoints within the cluster as the new start and end timepoints. To create simple and interpretable models, we then fit logistic regression models using the ten identified drivers. Climate drivers are identified and models are trained using the training data only. We perform this procedure (identify ten features, fit logistic regression model) for both crop models.

We evaluate model performance and the robustness of our approach to different parameter choices in several ways. The ability of the final logistic regression models to predict yield failure over the three held-out test sets is assessed using four performance metrics: (i) the area under the receiver operating characteristic curve (ROC AUC), a ranking metric that reflects the ability of the model to distinguish between binary classes at a range of classification thresholds; (ii) the average precision, which measures the tradeoff between precision and recall at different classification thresholds; (iii) the Brier score loss, which measures the mean squared difference between each predicted probability and the true (binary) outcome; (iv) and the logistic loss (log loss), also known as the cross-entropy loss, which measures the negative log-likelihood of each predicted probability given the true outcome. Note that the prediction task is an imbalanced classification problem, with non-failure years occurring approximately nine times as frequently as yield-failure years. Additionally, we investigate the ability of the model to capture temporal patterns in yield failures over both the training and held-out region by calculating the percentage of grid cells each year predicted to be experiencing yield failure and comparing this to the simulated “ground truth”. As predictions from the logistic regression model range between 0 and 1, we select a threshold to define yield failure that achieves the maximal f-score (the harmonic mean of the precision and recall) over the training set.

We assess if drivers are plausible by checking whether they only include the variables used as input by the crop models (i.e., ps, hurs and huss should not appear, nor should sfcwind for pDSSAT). We also compare their distributions between yield failure and non-yield failure years and assess their consistency with associations captured by interpretable models and expert judgement.

We further test the robustness of the approach by conducting 100 bootstrap repeats, sampling 20 pools (with replacement) from the 100 pools described above. For each repeat, we extract sets of ten climate drivers from the 20 pools, and evaluate the extent to which these sets consist of the variables actually taken as input by the crop models by counting the variables used by the features collected from each pool (in Step 2 of the driver identification method), and from the condensed drivers over all bootstrap repeats. Second, for each of the repeats, we construct models with 5, 10 or 15 identified drivers and evaluate their predictive performance over the three test sets, in comparison to a variety of commonly-used baseline feature sets. An assessment of model performance estimates using up to 1000 bootstrap repeats, for Lasso logistic regression models on the temporal test set for LPJmL, suggests that 100 repetitions is sufficient for stable model performance estimation (Fig. ).

The baseline feature sets consist of mean climate conditions aggregated over different time intervals: either the growing-season (defined as 180 d after planting), or on a quarterly (three consecutive ninety-day intervals after planting) or monthly (six consecutive thirty-day intervals after planting) basis. These are calculated for a subset of the climate variables (pr, tas and rsds) or for all ten. In addition, we include five extreme indicators that are often used in data-driven agricultural yield models . These are calculated over the first 180 d after planting, and consist of the minimum and maximum temperatures experienced, the maximum five-day precipitation sum, the number of days in which the temperature drops below 0 °C, and the number of days in which temperatures exceed the 90th percentile (as defined during the training period for each grid cell). This results in six baseline feature sets containing between 8 and 65 weather features.

As the number of predictors can affect model performance, to facilitate a fair comparison between the baselines and the sets of climate drivers identified using our approach, we use Lasso logistic regression models, which uses L1 regularisation to force the coefficients of less important features to 0, effectively performing feature selection. Predictors are first scaled by removing the median and scaling to the 10th and 90th quantile range (based on training-set statistics), and the value of the hyperparameter α, which controls the level of regularization, is selected using five-fold temporal cross-validation over the training years and regions. We also compare results when including second-order interaction terms and when using more complex extra trees models (with the same hyperparameters as previously described).

Finally, we assess the robustness of the approach to a number of parameter and methodology choices. For these sensitivity tests, performance is measured using a temporal train-test split, with drivers identified and models trained on the first 30 % of growing seasons and tested on the final 70 %. The cross-validation strategy chosen when selecting features from each pool is important . We explore the impact this choice has on the results by comparing model performance scores when using three other cross-validation strategies (random sampling, spatial clusters and clusters in feature-space) in addition to the temporal strategy described above. We also test the impact of using 5, 10 or 15 cross-validation folds. Furthermore, we conduct sensitivity tests in which several parameters of our method are varied both individually and in combination: the number of time intervals sampled in each pool of candidate features, the maximum number of features selected from each pool, and the number of drivers identified.

Observational data is split into training years (1983–2007) and test years (2008–2017). County-level yield failure years are determined based on falling below the tenth percentile of (detrended) yields over the training period. Using maize planting dates from WorldCereal , daily multivariate climate data is obtained for 90 d before planting and 240 subsequent for each year and county.

After processing, the method is applied on the training data to obtain four climate drivers in the same manner as described above. 100 pools of candidate features are obtained based on 20 sampled time intervals per pool, and ten features are selected from each using five-fold temporal cross-validation and extra trees models. The obtained drivers are then used to fit a logistic regression model to the training data, and performance is evaluated on data from the test years.

Maize yields simulated by pDSSAT are more variable in space and time than LPJmL, but both simulations exhibit lower average yields in the western regions of the US and declining annual yields until the late 1930s, followed by an upward trend (Fig. a–c). As management and genetic changes over time are not considered in these simulations, this behaviour is the result of changing climate conditions, and in fact these trends coincide with opposing shifts in average growing-season temperatures (Fig. d). The high spatial variability in simulated yields and temperatures suggests that a space-for-time substitution may be able to improve the ability of data-driven models to extrapolate to warmer (future) years. Despite increasing temperatures in later decades, most locations experiencing an unprecedentedly-warm growing season are unlikely to be outside of the range of previously-experienced conditions at all grid cells until that year.

Composite plots of the growing-season daily climate conditions for normal and yield-failure years suggest that non-causal climate variables are associated with yield failure in these simulations for both crop models (Fig. , with further variables in Fig. ). For example, near-surface relative humidity is not used as input for LPJmL or pDSSAT, but yield failure years tend to have drier growing seasons in both crop models (Fig. g and h). Composite plots of growing-season windspeed show an association between lower winds and yield failure for both models (Fig. i and j), despite the fact that windspeed is an input variable for LPJmL but not pDSSAT. Yield-failure years tend to have higher growing-season temperatures, but there is a wide overlap in temperature conditions between failure and non-failure years (Fig. c and d). Similarly, periods of low rainfall in the first four months after sowing are associated with yield failure, but many non-failure years also experience periods of low rainfall at that time (Fig. e and f). Furthermore, while simulations from the two crop models differ in the proportion of grid cells experiencing yield failure by up to 10 % in some years (Fig. a and b), relationships suggested by all composite plots are visually similar. Composite plots of growing-season daily climate conditions for normal and yield-failure years when yields are detrended are very similar (Figs.  and ).

A logistic regression model fit on the training data using ten identified climate drivers of pDSSAT yield failure as predictors is able to achieve a ROC AUC of 0.83, average precision of 0.32, Brier score of 0.056 and log loss of 0.20 on the held-out test years. Comparable scores are achieved when applied to the held-out spatial test set (covering the same growing seasons as used for identifying drivers and training the model) and the spatiotemporal test set (including only held-out years and locations) (Table ). Notably, the model is able to outperform lasso logistic regression models using all baseline climate feature sets tested for all metrics calculated and for all held-out sets. The model captures the year-to-year variability in yield failure well (Fig. a), with a Pearson correlation of 0.84 between the true and predicted annual proportions of grid cells experiencing yield failure over the training region (above 35° latitude) and 0.91 over the held-out region (below 35° latitude). Notably, model performance does not appear to deteriorate over time: annual ROC AUC scores slightly increase over the decades, while Brier score and log loss decrease. Maps of predictions and ground truth in 2011, in which the majority of grid cells in the lower latitudes experienced a yield failure (Fig. c and d) show general agreement in spatial variability, including over the held-out lower latitudes in which yields were most adversely impacted.

A logistic regression model using ten climate drivers for LPJmL is able to achieve similarly high performance (Table , Fig. ). A Pearson correlation of 0.76 is achieved between true and predicted annual proportions of grid cells experiencing yield failure in the training regions and 0.87 in the held-out lower latitudes. Similar levels of model performance are achieved for both crop models when yields are detrended before defining yield failure (Figs.  and ).

The ten climate drivers of yield failure identified by our method for each crop model consist of variables that are taken as input by those respective models, with the exception of one driver based on mean daily temperature for pDSSAT (Fig. ). However, as pDSSAT takes as input the daily minimum and maximum temperatures and uses them to estimate hourly average temperatures for use in model processes, it could be argued that mean daily temperature is a plausible driver. For both crop models, several drivers are aggregates of daily precipitation and, in combination, span a time period starting before planting and ending four months afterwards, with overlap between drivers. In terms of aggregation methods selected, precipitation drivers identified measure the number of days above the 90th percentile, which can be interpreted as the number of rainy days, the maximum daily precipitation, or the mean daily precipitation over a time interval, which would be proportional to the cumulative rainfall in that period.

Temperature is strongly represented in climate drivers for both crop models, but temperatures earlier in the growing season are more often identified for pDSSAT (four drivers) than LPJmL (one driver). Three of the temperature drivers are almost identical across both models: the number of days of minimum daily temperatures above the 90th percentile from around planting date up until around five months afterwards; the number of days of maximum daily temperatures above the 90th percentile over a three-month period starting around two months after planting; and the minimum temperature experienced over a two-month period starting around ten weeks from planting. Shortwave radiation is included in the set of drivers for both crop models. No identified drivers take into account the climate conditions later than six months from planting, and the only variable considered before planting is precipitation.

Analysis of the relationships between identified climate drivers and yield failure is made possible by the use of simple, interpretable models. Odds ratios associated with the climate drivers for fitted logistic regression models are reported in Table . Odds ratios greater (smaller) than one indicate higher (lower) odds of yield failure given a one-unit increase in the predictor, conditional on the other predictors of the model, while an odds ratio equal to one would suggest no association of that predictor with yield failure, in this model. For both crop models, climate drivers are based on mean precipitation are highly influential, with increased rainfall strongly associated with lower yield failure probability. Additionally, the climate drivers with the strongest positive association with yield failure are temperature-related. For both crop models, however, both positive and negative associations with yield failure are identified for temperature-related drivers, pointing towards nonlinear relationships between growing-season temperature and yield failure probability.

When yields are detrended before applying the method, the precipitation- and temperature-related drivers identified are largely similar to those identified without detrending for both crop models (Fig. ). Climate conditions later than six months after planting are again not considered. However, when yields are detrended, short-wave radiation is not used by any of the climate drivers identified for pDSSAT, and one of the drivers uses specific humidity.

Histograms of some identified climate drivers in yield failure and non-yield failure years for each model are largely in agreement with the associations implied by the odds ratios (Fig. ). For both crop models, yield failure years are more likely to have fewer rainy days between 7 and 13 weeks after planting (Fig. a and b) and more days of high maximum temperatures between 2 and 6 months after planting (Fig. c and d). Normal years are more likely to experience a lower minimum temperature during the second and third month after planting (Fig. e and f). For LPJmL, a higher number of rainy days in the first ten weeks after planting and eight weeks before are associated with decreased chances of yield failure (Fig. g). For pDSSAT, the yield failure distribution of the number of warm days (in which minimum daily temperatures exceed the 90th percentile) in the first five months after planting is slightly higher than normal years (Fig. h).

To evaluate the robustness of the approach we now turn to the results for the 100 bootstrap repeats, where in each repeat 20 pools are sampled from 100 total pools of candidate features. The original pools, by definition, contain an equal number of features based on each of the ten climate variables (Step 1). After selecting ten features from each pool based on cross-validation predictive performance (Step 2) we find that the majority of features selected consist of variables that are used as input by the crop models (Figs. a and a). However, some variables not used by the crop models are also selected, albeit less frequently (surface pressure, windspeed, and specific humidity for pDSSAT and surface pressure and specific and relative humidity for LPJmL).

After condensing the features selected from twenty pools to final sets of ten climate drivers (Step 3), variables not used as input by the crop models are eliminated for pDSSAT in all 100 bootstrap repeats (Fig. b), and for LPJmL, two of the variables not used by the crop model (specific humidity and surface pressure) are each selected in 3 out of 100 repeats, while the remaining variable unused by the crop model (relative humidity) is eliminated for all repeats (Fig. b). This shows that our approach is able to identify climate drivers based on variables that are used as input by the crop models and are therefore more plausibly causally-related to crop failure. When yields are detrended, results differ by crop model. For pDSSAT, specific humidity, which is not used as input by the crop model, appears in 97 of the 100 bootstrap repeats, relative humidity appears in three, and surface pressure is eliminated (Fig. a and b). For LPJmL, all variables not used by the crop model are eliminated for all bootstrap repeats (Fig. a and b).

The duration of the time intervals over which climate conditions are aggregated in the selected features before condensing (Step 2) displays three peaks: at two to four weeks, at two to three months, and at approximately four to five months. This third peak gradually tapers off, with some selected features based on ten or eleven months of climate data. The time intervals over which condensed drivers are aggregated (Step 3), on the other hand, show two distinct peaks for pDSSAT: at approximately eight weeks, and at four to five months (Fig. c). For LPJmL, the second peak, if present, is less visible (Fig. c). Drivers never, or rarely, consider more than six months of climate data. The differences between selected features and the condensed drivers identified are consistent when yields are detrended (Figs. c and c).

For both crop models, Lasso logistic regression models using sets of 5, 10 or 15 identified climate drivers, fitted on training data only, are able to achieve evaluation scores on the temporal, spatial and spatiotemporal test sets consistently higher than those using average growing-season or quarterly climate features and five extreme indicators (Figs.  and –). For pDSSAT, models using only ten identified climate drivers consistently perform better in terms of all performance metrics evaluated and on all test sets than any of the baseline feature sets tested. Models using only five drivers also outperform the baseline feature sets for most metrics and test sets; the only exception is the spatial held-out set, on which one baseline set is able to achieve a higher log loss score (although obtaining worse performance according to the three other performance metrics). For LPJmL, models using five or more drivers outperform all baseline feature sets on all test sets and for all evaluation metrics evaluated, with the exception of the baselines consisting of monthly climate aggregates, which achieve roughly similar scores (depending on the test set and evaluation metric used). For both crop models and all metrics, increasing the number of identified climate drivers used as predictive features is beneficial for model performance on the temporal test set, but can lead to unchanged or decreased performance on the spatial and spatiotemporal test sets (depending on the test set, crop model and evaluation metric used).

If second-order interaction terms are included, models using fifteen identified climate drivers can often outperform, or perform similarly to, models using monthly climate averages on the temporal test set for all evaluation metrics calculated, and can consistently outperform all other feature sets tested (Figs. –). On the spatial and spatiotemporal test set, models using identified drivers reliably outperform baseline sets consisting of annual climate averages plus extreme indicators, and outperform quarterly baseline sets according to most metrics evaluated. The comparison to baseline sets consisting of monthly climate averages plus extreme indicators depends on the crop model and evaluation metric used. Results are similar when using more complex extra trees models, and for both crop models and all models tested when yields are detrended (Figs. –).

The results described so far are based on applying the method to identify drivers of yield failure, where yield failure is defined as yields below the 10th percentile of those occurring during the training years. When reproducing the results using a more extreme (5th percentile) or more moderate (15th percentile) definition of yield failure, we find that the variables, aggregations and time-durations of identified drivers are largely identical, with some minor differences. In general, the results suggest that the method is slightly better able to eliminate variables that were not used as input for the crop models when using higher (more moderate) thresholds for defining yield failure. For example, when using a 5th percentile threshold, specific humidity is sometimes identified as a driver for LPJmL, but not when the 10th or 15th percentile is used.

The ten identified drivers based on 100 repeats are largely similar for all yield failure thresholds tested (Figs. –). In particular, the precipitation drivers and those using the number of days where maximum temperature is above the 90th percentile are almost identical. Almost all drivers identified are based on precipitation, minimum or maximum daily temperatures at different stages of the growing season. However, there are some differences, particularly in the drivers identified based on the 5th percentile, which features long-wave radiation for pDSSAT and specific humidity for LPJmL.

If a more extreme definition of yield failure is used (5th percentile), we observe generally poorer predictive performance across the held-out test sets, particularly in terms of average precision, for all models, whether using identified drivers or baseline climate feature sets (Fig. ). This is likely because the task and the training set becomes more imbalanced, by definition, with fewer examples of yield failure. Correspondingly, when using a (more moderate) 15th percentile threshold, we observe overall higher scores across the models, metrics and test sets (Fig. ). When comparing the relative performance of models using our drivers to those using baseline feature sets, however, the results are essentially unchanged for all thresholds: models using drivers identified using our proposed method as predictive variables usually outperform those using standard sets of climate indices.

Finally, we find that the results of the method are sensitive to the cross-validation method used when selecting features in Step 2, both in terms of climate variables selected and the test performance of models using sets of identified drivers. Using temporal cross-validation results in the highest proportion of potentially-causal variables used in features selected from pools (Step 2) and in identified drivers (Step 3), and this is consistent for both crop models (Figs.  and ) and also when yields are detrended (Figs.  and ). Spatial cross-validation and feature-cluster cross-validation results in a lower proportion of potentially-causal variables across all settings, and random cross-validation the lowest. Similarly, identified driver sets selected using temporal cross-validation in general achieve the highest test scores and those selected using random cross-validation the lowest (Fig. ). This general behaviour is robust to the evaluation metric, the model used (Lasso logistic regression with and without second-order interactions or extra trees), the crop model and if yields are detrended.

In comparison to the choice of cross-validation strategy, the performance of the identified driver sets is relatively insensitive to all other parameters tested. Increasing the number of time intervals sampled per pool slightly improves model performance with resulting sets of drivers for pDSSAT, but not LPJmL. Interestingly, selecting fewer features from each pool leads to increased performance of identified driver sets in Step 3.

Using the same methodology for observational US county-level data and daily meteorological data from eight variables, we obtain four climate drivers of maize yield failure (Fig. ). The only variables used are precipitation, maximum vapour pressure deficit and maximum temperature, and the drivers cover a time period starting approximately one month before planting until around six months afterwards. Precipitation is relevant for a three-month period starting one month prior to planting, and the number of days in which the maximum temperature exceeds the 90th percentile are relevant from one month from planting until six months from planting. Two drivers are based on vapour pressure deficit conditions in two distinct phases of the growing season and based on different aggregation methods, with the number of days of extreme vapour pressure deficit conditions most relevant for a two-month period starting approximately two months after planting.

A logistic regression model with the identified four drivers as predictors, fitted to data from the training years, achieves a test ROC AUC score of 0.85, average precision of 0.54, Brier score of 0.073 and log loss of 0.26, and the Pearson correlation of predicted versus ground truth annual proportions of counties experiencing yield failure is 0.98 (Fig. ). Notably, the test years include a year which was heavily impacted by drought (2012), which is well-captured by the model (achieving an average precision in that year of 0.83).