Paper Review: "An Effective Communication Topology for Performance Optimization: A Case Study of the Finite Volume WAve Modeling (FVWAM)"

This paper presents an implementation of halo-exchanges in the FVWAM model using MPI's distributed graph topology and performance comparison over baseline implementation using point-to-point communication primitives. 

The paper provides a detailed comparison from tests with 512 to 32,768 processes and shows that the speedup from the distributed graph topology with and without reordered processes ranged from 1.28 to 5.63.

There is some important context missing from the article that can shed light on the significance of the performance improvements.

* What is the network interconnect and topology of the target system?

* Were experiments repeated with different node allocations assuming there is a batch system scheduling resources?

* Were the different experiment types (point-point, distributed, distributed with reordering) conducted using the same node allocation for consistency? 

* Were the experiments at each processor count (e.g., 512) conducted multiple times to rule out network variability and interference from other traffic on the network?

* Is there any performance variability across runs? 

* Can the authors elaborate differences of their approach if any with using the MPI-3 neighbourhood collectives?

The authors allude to the following factors as the primary contributors to the improvement: 

> First, as the number of processes increases, the volume of exchanged data decreases, thereby reducing the speedup ratio achieved by the distributed graph communication topology. 

It appears that the application is network bandwidth bound at low processor counts. It would be very enlightening to provide details of the communication volume and interconnect specifications to confirm if that's the case.

> Second, received data are continuously searched and inserted into wave action (N) at once in the distributed graph communication topology, which can improve cache hit rates.

The presumption about improved cache hit rates can be confirmed by obtaining hardware performance counter information. I'm skeptical that cache performance played such a big role. The improvement could better be explained by MPI library implementation ordering the communication operations optimally.

The biggest weakness of this work is the limited performance data from just one platform and MPI implementation. It would highly strengthen the work if the performance optimization can be demonstrated on multiple machines with different interconnects and topologies and recent versions of community standard libraries (OpenMPI, MPICH) or recent vendor implementations. It would make the case for neighbourhood collectives for earth system workloads stronger. 

There is work illustrating performance improvements from reordering MPI processes taking network topology into account. e.g., https://dl.acm.org/doi/10.1145/2851553.2851575

What is the current reordering strategy in case I missed? Did the authors consider any advanced reordering strategies? 

The paper refers to pre-posting receives using MPI_Irecv. However, they mention

> An alternative is to call the non-blocking communication interface MPI_Isend for sending data, but it is infrequently utilized due to the increased complexity that introduces to the sending operation

It's not inherently that complex as a lot of applications use non-blocking sends effectively. I wonder what the performance impact would be if the authors used non-blocking operations.

I was looking forward to the article to hear about novel techniques that could improve communication performance at scale. However, it was slightly disappointing to see that the benefit from the proposed optimization dramatically tapers off as we go from low (512) to high (32756) number of processes.

On heterogeneous GPU based supercomputers like the Frontier Exascale system, the number of nodes is relatively low (9,408) due to the fat node architecture compared to CPU based supercomputers like Fugaku (158,976 nodes) out there. In this overall context, the benefit of a communication optimization if more relevant at scale when there are potentially hundreds of thousands of MPI endpoints at scale (e.g., 600k on Fugaku with 4 MPI ranks per node mapping optimally to the NUMA domains there).

Ref: https://docs.olcf.ornl.gov/systems/frontier_user_guide.html

https://www.fujitsu.com/global/about/innovation/fugaku/specifications/

Pg 15, lines 295-302: 

To conclude, I understand the motivation of authors to improve their production simulations performance and the relative significance for their workload. Additional performance data would be highly informative and make this more generally applicable.

Minor comments:

Pg 2, line 29: There are better references than Sukhija et al., 2022 for the Frontier Exascale supercomputer. I suggest using one of the papers from the Supercomputing conference. https://dl.acm.org/doi/abs/10.1145/3581784.3607089

> Sukhija, N., Bautista, E., Butz, D., and Whitney, C.: Towards anomaly detection for monitoring power consumption in HPC facilities, in: 380 Proceedings of the 14th International Conference on Management of Digital EcoSystems, pp. 1–8, 2022

The performance sections in the paper are a bit verbose and redundant pointing to the information in the figures. It might be better to be succinct in highlighting the results and elaborate further on the reasons behind the improvement.