The metrics used in this evaluation paper are defined in the supplement and their description does not include reference to previous work. This leads me to think that these metrics have been developed here specifically for evaluating these models. Even if this is not the case, GMD has a general geoscientific audience who will not necessarily be aware of the norms in the field. The definitions, plus full context and reasoning for the choice of metrics should appear in the main text of the manuscript, not in the supplement. Likewise for the description of the datasets used - please describe these fully in the main manuscript. The extra figures and tables can probably remain in the supplement.

On the details of the metrics, for example, I know there are arguments on both sides, but for a model-data comparison which surely aims towards consideration of how models may be improved, RMSE not MAE would be my natural choice. (See for example, doi:10.5194/gmd-7-1247-2014, “in the data assimilation field, the sum of squared errors is often defined as the cost function to be minimized by adjusting model parameters. In such applications, penalizing large errors through the defined least-square terms proves to be very effective in improving model performance.”)

Although it may be premature to include data uncertainties in the numerical metrics, more discussion of these uncertainties is required. In particular, the point that reviewer 1 makes about comparing the variation within the ensemble with the uncertainty in the data is valid. This is something it is important to have a feel for. Again, remember that some of your readers will be generally interested in model-data evaluation, but innocent of the details of fire-enabled models and the relevant data.

Section 3.1
“The simulated modern (2002-2012) total global annual burnt area is between 39 and 536 Mha (Table 2). Most of the FireMIP models are within the range of the remotely sensed observed burnt area (354 to 468 Mha a–1).”

This does not seem to me to be a fair description of the ensemble. Table 2 suggests that two models are very far from the observed range but the rest are impressively close to the quoted range, although I may be misunderstanding the data uncertainty (which I am assuming is 354-468 at 1 standard deviation).

Reviewer 2 wrote:
“Lines 345-349 “there is a positive relationship between simulated burnt area scores and the seasonal concentration of GPP (R2 = 0.30-0.84) and, to a lesser extent, the seasonal phase of GPP (R2 = 0.09-0.24). This supports the idea that seasonal vegetation production and senescence, which have an important influence on fuel loads, drive the interactions between vegetation and fire within each model” – I doubt this statement. It is more likely that similar dynamics of burnt areas and the seasonal concentration of GPP/ the seasonal phase of GPP are related to dependence of both areas burnt and GPP variables from soil moisture in the fire models and DGVMs. Please, either prove, or delete this paragraph. “

I don’t understand your response to this, as I do not think the reviewer is, as you assert, suggesting that “this feature is solely induced by soil moisture”...? Sorry if I am being hopelessly stupid, but I remain concerned that if this reviewer cannot follow your justification for the existence of this positive relationship, then other readers will also have the same problem. I would have thought that some extra explanation at least should be added to the manuscript even if you cannot go as far as to totally “prove” it.

Thank you for making the benchmarking code available and including a nice readme file. It is not clear to me whether turnkey code for the creation of the figures has also been included. If not, please add this. If it is already there, please flag it clearly in your readme file and also in the Code and Data availability section.

Good job (as they say in the USA)! I find it hard to express how much improved this paper is. Suddenly it looks like a proper GMD paper.

So, I can't help but accept as-is... ;-)