The study presented in this article is of critical relevance to the future informed development and review of the abundance of models emerging within the domain. In addition to performing an ablation study to critically analyze several key design decisions of the original PGW model, the authors contribute notable improvements of architecture and training procedures that overall make model training significantly more computationally efficient while maintaining comparable quality. Overall, this is a worthy effort not only to understand and analyze PGW, but may also inspire similar work on other models in the domain.

There are some minor inconsistencies between what is written in analysis and procedures in the text and the corresponding plots. This does not invalidate the main conclusions of the paper per se, but needs to be checked.

That the apparently intuitive benefit of taking in the vertical dimension with the 3D transformer into account appears not to be essential is a noteworthy and surprising finding. The speculations given in the discussion session about this call for further analysis and care with future models.

I feel that the most concerning shortcoming of the work is indeed the missing comparison of a (indeed costly) non-subsampled version with the original PGW, as also indicated in the discussion section; I agree it should probably not invalidate the benefits of the optimizations done, though some doubt remains. Still, the findings remain indicative even without this (still better RMSE than IFS).

Detailed comments:

- p. 2: new training procedure 30% faster - compared to 2D or original 3D?

- p. 3: what was the number of compute nodes? were local SSDs used in some form? if not, is mentioning them relevant to comparable studies (I believe such hybrid setups would be very peculiar to use)?

- p. 5, fig. 1b: These plots do not show perfect sin/cos functions, they are skewed; reading the explanation in 2.6, I don't understand why. I believe this comes from tweaking them to match the original weight sums, but then I'm missing an explanation for this particular tweak. Also, they are not in phase as explained (l. 134); e.g., the maxima of U/V are slightly shifted (epoch 200 in (a), epoch ~185 in (b)).

- p. 8, l. 172: If I read figure 6 correctly, PGW-Lite failed to converge for sizes of 16, 32, and 480. For 64, it converged (hard to read the figure here but I believe there's a dashed red line just behind the solid red/orange line). This is in contradiction to what is written in the text.

- p. 8: The point of the minibatch study appears to be to 1.) analyze how minibatch sizes affect attainable loss/convergence and 2.) make a direct comparison on this between the 2D and 3D transformer approaches. For me as reader it would have been good to point this out here (it only became clear when reading discussion and conclusion) because it affects how one reads the text and plot.

- p. 8: Would different random seeds have a significant effect on convergence/attainable loss?

- in general, while zooming helps, the colour scheme (use of yellow) and size of figures 6 and 7 makes them hard to read, particularly given that many lines relevant to discussion overlap

In the manuscript entitled “Architectural Insights and Training Methodology Optimization of Pangu-Weather”, the authors present a two-dimensional attention mechanism (2D-Transformer) with an Earth-specific positional bias term and relative bias which accounts for weather states being related to the absolute position on Earth. The 2D-Transformer performs more effectively, reducing computational requirements by 20-30%, decreasing the model size by nearly 40%, and increasing the robustness of the Pangu-Weather model's convergence significantly. The ablation study determined a new training process to accelerate the convergence for the 2D-Transformer model without any further hyperparameter tuning.

General comments:

Figure 3 is an important chart supporting the validity of this research. However, it does not include specific humidity, Z500, or V10, as mentioned earlier in this paper. Including these variables could strengthen the argument for the effectiveness of the 2D-Transformer. Due to the 6-hour subsample, readers ultimately do not know if the 2D-Transformer has improved the forecast accuracy of the original Pangu-Weather model. Including such comparisons could significantly increase the citation rate of this paper. There are still certain changes and clarifications that the authors should address prior to publication. For these reasons, I believe that the manuscript can be accepted for publication. Below, I have some specific comments to the authors.

Specific comments:

• Line #2 - #5, the sentence is too long and difficult to read. It can be revised to “The Transformer-based PGW introduced novel architectural components, including the three-dimensional attention mechanism (3D-Transformer) in the Transformer blocks. Additionally, it features an Earth-specific positional bias term that accounts for weather states being related to the absolute position on Earth.”

• Line #24, “the authors also admit” could be replaced with more specific wording, such as “previous studies have shown”. The same issue appears in Line #91, where the architecture described “by the authors” could be replaced with “in this study.” This sentence reads as if the ablation study is original to this paper and not derived from the model itself. If this is the case, some references could be cited here as evidence to support the experiment design.

• Line #27, the published model cannot be run, what is the reason? Is it also caused by modularized manner issue? Does it conflict with reproduction introduced in section 2.3?

• Figure 1, 4, 5 and 7 could be appropriately enlarged. Some images are difficult to discern even when enlarged. For Figure 4 (a), the authors can separate the lines by adjusting the y-coordinates.

• Figure 1, the cosine functions for each variable with weights from Bi et al. (2023) could be listed to explain the normalized process at each epoch. In Figure 1(b), it would be helpful to present the equation for the sloped MSP graph to facilitate understanding.

• Figure 6: Could the author explain the reason for the failure to converge based on PGW-Lite structure with minibatch sizes of 32, 64, and 480? PGW-Lite with minibatch size 720 eventually converged. Could the authors explain this unexpected result? Part of the reason is explained in Line #229. It is not necessary to strictly separate the results and discussion sections. Explaining part of the findings in the result section can enhance the content.

• Line #255, the sentence could be updated to “since wind vectors, acting as pressure gradients, can drive certain atmospheric processes, such as advection terms in atmospheric variables.”

Following are suggestions and do not affect the validity of the argument in this paper.

• Line #98: the models could be compared in more details in Table 1, like the hidden dimension, etc.

Technical corrections:

• Figure 5 y axis could be updated into “normalized RMSE”

The study „Architectural Insights and Training Methodology Optimization of Pangu-Weather“ contains an ablation study for a modification of the Pangu Weather model named PanguLite. Furthermore, it presents insights on training strategies for PanguLite and a two 2d variant of the Pangu that performed best in the ablation study.

The paper is very well written and gives valued insight into the architecture and performance of the models described. Currently, research on ML models in numerical weather prediction is highly successful, and probably soon, AI methods will be used in the operational routines of national weather services. This work contributes to examining the performance and behavior of such models, making their design and training more effective and reliable. Additionally, the authors explain the function of individual components, thus bridging the gap to classical modeling, where a system understanding is a crucial aspect of mathematical-physical modeling.

I recommend the study for publication with minor modifications addressing the following questions and remarks.

1. The model names used in the tables and in the text are not consistent. Table 1 and Table2 might be merged with unique model names.
2. Visualisation of the modification in model architecture: I find the modularised code in the repository very helpful. Could the code be included as pseudo code in the paper giving a comparative overview of the architectural details of the different models? This would be helpful in connection with the graphics in the original Pangu Publication (Fig. 2).
3. Parameter numbers in Table 1:  The numbers of parameters for the 2d attention model s larger than for the 3d attention (PanguLite) in Table 1. This is counter intuitive. Is it due to the fact that the hidden dimension C was enlarged? What was the reasoning behind that choice? Could it be chosen such that the overall parameter size would match that of PanguLite again? Is this dimension C the same for PanguLite and Pangu? Could the authors extrapolate the parameter numbers in Table 1 for the original Pangu model with the original batch size?

1. Parameter numbers in Table 3 (relating to Remark 3): In paragraph 2.5 the authors state that reducing the model size allows for larger local batches. Hence, PanguLite should have larger local batches than the 2d version. In Table 3 it is the other way round. Could the authors please clarify this?

1. Fig. 1: As the curves for U and V are indistinguishable, one colour for both curves would render the figure clearer. Furthermore, plot a) and b) should display temperature and wind in the same colour.
2. Figure 7: What does it mean that the reference model is PanguLite? The text implies that the two curves show both the 2d model with different training losses.