The authors show the value of a two out of sample approach for statistical modelling. The formal approach is convincingly motivated and presented. The substance of the modelling, however, is completely inadequately presented. One would expect a brief characteristic of the modelled crops and their cropping specifics (annual, perennial; sensitive phases for weather dependence), a characterisation of the climate at the site considered, information about the crop relevant weather variability in the periods considered, a motivation of the predictors, and the margins for determining the predictors, a discussion of model errors. The superficiality of the problem analysis becomes particularly clear in the classification of the model quality: 

'It was shown that monthly mean precipitation and temperature could explain more than 30 % of the coffee yield anomaly variability. The 70 % remaining variability is due to non-climatic factors (agricultural practices, diseases, or political and social context). '

This conclusion presupposed that the authors had fully explained the weather-related yield variability with their approaches. However, this cannot be assumed given the selection of climate variables, their coarse resolution, the static view on the underlying processes, the negligence of their interplay, the linearity restrictions applied, the exclusion of spill over effects from previous seasons and of soil characteristics.

Nevertheless, the presented approach shows which formal possibilities exist to obtain a 'best' model without an in-depth examination of the object to be modelled. This is quite interesting for practical statistical modelling. However, the examples presented here (maize, coffee) should be motivated much more comprehensively and classified in a more differentiated way. This does not mean that the authors have to make a comprehensive modelling claim.  But they should be able to justify and classify the range of the model configurations considered in their modelling attempt.  

The general quality of the manuscript is currently at the border between ‘fair’ and ‘poor’, but has potential for moving to the better direction when the authors motivate more explicitly their modelling approach for the selected climates and crops.

General comments

This paper addresses the very important question of model complexity. How can one best choose the level of model complexity when the objective is to minimize error of prediction for out of sample cases. The paper presents two practical cases (prediction of coffee yield in Viet Nam, prediction of maize yield in France) and two prediction methods (linear regression, artificial neural network). The main objectives of the paper are to explain the leave two out cross validation approach (LTO) as a method of evaluating and choosing between models of different levels of complexity, and to compare it with the leave one out cross validation approach (LOO). The presentation of LTO is useful and interesting, but I have some issues with the way the linear regression examples are formulated. (I do not comment on the use of neural networks, with which I am not very familiar).

Specific comments

The authors consider models with a fixed number of explanatory variables and take as the measure of model complexity, the number of potential explanatory variables from which the explanatory variables in the model are chosen. This is not usually the way the problem is formulated, at least in linear regression. In general, the list of potential explanatory variables is  fixed, and the question is how many and which to include in the final model. Then the larger the number of explanatory variables chosen, the more complex the model. Comparing LTO and LOO as a function of the number of  potential explanatory variables may then not be very relevant to the problem of determining model complexity for a linear model. A more relevant question would be: How do LTO and LOO compare, when the number of potential explanatory variables is fixed, and the objective is to choose the best ones to include in the regression model?

In addition, though LOO can be used for model selection (choosing best explanatory variables) in linear regression, there are other approaches which are probably more common, such as forward regression, stepwise regression, the Akaike Information Criterion etc. How does LTO compare with other methods of model selection?

Technical corrections

L138 Problem with English

L145 Should be “to choose”

L163-164 I don’t understand the sentence

Section 3, introduction. It seems to me that much of this is said elsewhere. Maybe combine with section 2.3?

Fig 3. I find the portrayal of distribution functions confusing. First of all, in the figure the distribution functions are continuous, while in practice they are necessarily discrete. Secondly, as far as I can tell, the distribution functions are totally irrelevant. Only the average error is of interest.

Section 3.2.2. Talking of a “testing dataset” is a bit confusing (is this the full set of testing  data, which is identical to the available data, or is this a single value for each fold). Perhaps refer to the “testing value” or “testing datum” when talking about a single fold.

L266 “implemented” is probably the wrong word.

L362 “request” is probably the wrong word

L366 I don’t understand this sentence.   

The authors discuss an important aspect of statistical analysis with limited data availability and exemplify the shortcomings of the more frequently used ‘leave one out’ cross validation compared to the ‘leave two out’. Two case studies are used, an annual crop in Europe and a perennial crop in Vietnam. Two different statistical approaches are used, linear regression and neural network. The two case studies convincingly demonstrate how to select the best model for forecasting crop yield with limited data availability, while considering the problem of overfitting.  In general the study is well designed and clearly communicated, although substantial English revision is required.

Some general observations:

• Line 132: the model architecture is explained as consisting of i) number of potential predictors, and ii) the number of inputs. Please specify clearer what the difference is between predictors and inputs. The authors also mention model types: Some models require more parameters to estimate even with equal numbers of predictors. This should be described in clearer terms.
• The value of using LIN3 and LIN5 are not clear to me, which is related to the previous comment: the authors explain them as linear regression models with three and five inputs, respectively. Could you explain what the value is of this experiment? What added insights do we gain when comparing LIN3 and LIN5? Do we not already get all the insights from adding the number of potential predictors to LIN3?
• There is quite a bit of redundancy – the authors explain the problem several times, thereby repeating themselves. For example, section 3 (lines 168-181) already has been elaborated in previous sections. More concise presentation of the problem and methods would benefit the manuscript
• Even though the main aim of the study is to compare the LOO and LOT, presenting the chosen predictors and final model would be appreciated. Any reader familiar with the case study crops will be interested to understand what climate descriptors were tested and selected in the final model. This transparency is further necesary for making the study reproducible.