Development of efficient GPU parallelization of WRF Yonsei  University planetary boundary layer scheme

Huang, M.; Mielikainen, J.; Huang, B.; Chen, H.; Huang, H.-L. A.; Goldberg, M. D.

doi:https://doi.org/10.5194/gmd-8-2977-2015

Articles | Volume 8, issue 9

https://doi.org/10.5194/gmd-8-2977-2015

© Author(s) 2015. This work is distributed under
the Creative Commons Attribution 3.0 License.

https://doi.org/10.5194/gmd-8-2977-2015

© Author(s) 2015. This work is distributed under
the Creative Commons Attribution 3.0 License.

Articles | Volume 8, issue 9

Development and technical paper

|

30 Sep 2015

Development and technical paper |

| 30 Sep 2015

Development of efficient GPU parallelization of WRF Yonsei University planetary boundary layer scheme

M. Huang, J. Mielikainen, B. Huang, H. Chen, H.-L. A. Huang, and M. D. Goldberg

Download

Final revised paper (published on 30 Sep 2015)
Preprint (discussion started on 21 Nov 2014)

Interactive discussion

Status: closed

AC: Author comment | RC: Referee comment | SC: Short comment | EC: Editor comment

- Printer-friendly version

- Supplement

RC C3236: 'Development of Efficient GPU Parallelization of WRF Yonsei University Planetary Boundary Layer Scheme', Anonymous Referee #1, 31 Jan 2015
- AC C3611: 'Reply', Melin Huang, 07 Apr 2015
RC C3711: 'Development of efficient GPU parallelization of WRF Yonsei University planetary boundary layer scheme', Anonymous Referee #2, 06 May 2015
- AC C3721: 'Responses to comments of the Anonymous Referee #2', Melin Huang, 27 May 2015

Peer-review completion

AR: Author's response | RR: Referee report | ED: Editor decision

AR by Melin Huang on behalf of the Authors (11 Jun 2015) Author's response Manuscript

ED: Referee Nomination & Report Request started (29 Jun 2015) by Adrian Sandu

RR by Anonymous Referee #1 (15 Jul 2015)

RR by Anonymous Referee #3 (21 Aug 2015)

Suggestions for revision or reasons for rejection

The authors of this paper design and implement the YSU PBL scheme of WRF model on GPU and compare the performance on CPU using single core and multi-cores. The evaluations illustrate the significant performance benefits of GPUs over CPUs. As a researcher of computer science, my comments focus on the designs, implementations, and optimizations on GPU.

1. One major concern is whether it is fair to compare the highly optimized GPU code with the serial or not highly optimized code on CPU. I happen to know a previous research "GPU Acceleration of Numerical Weather Prediction", where the speedup of WRF on GPU over CPU was much lower than the performance numbers reported in this paper, although the model may be different and the GPU used in this paper is latest. I suggest the authors cite this paper and discuss the differences, from the scheme of WRF model to the optimization methods.

2. Section 3.2, "once a kernel is successfully launched, to obtain the correct GPU execution
runtime, command cudaThreadSynchronize() must be called." It is not necessary to call cudaThreadSynchronize() function if only synchronized kernels are launched.

3. Section 4.2, Optimization with more L1 cache. Why not use the shared memory explicitly in the implementation? I find the same question was asked by a reviewer in the previous iteration. Although the authors explained they used cudaFuncCachePreferL1 and cudaFuncCachePreferShared to change the sizes of L1 cache and shared memory, I want to figure out: on K40 GPU (Kepler architecture), the L1 cache is designed to hold the register data spilling, and L1 cache will be bypassed for the global memory data load/store (different with previous generations of GPUs). As a result, for the data access with good data localities, two mechanisms need to be investigated, a. using different numbers of register variable, b. using shared memory explicitly. I suggest the authors use NVIDIA Visual Profiler to collect profiling data, e.g., global load efficiency and kernel occupancy, and then discuss whether it is enough only using L1 cache / shared memory configuration and skip the shared memory in the implementation.

4. Section 4.7, Synchronous I/O and Asynchronous I/O. "It indicates that using asynchronous access, together with coalesced memory transfer, seems to help reduce the total runtime for this scheme." Table 5 cannot support such a claim. The best speedup 36.9x was obtained when kernel execution on GPU with Sync I/O. It is anti-intuition because the input data size is more than 300 MB and the output data size is more than 100 MB. Overlapping kernel execution and data transfer may lead to good performance. Another question is: the presentation of Table 5 is misleading. Non-coalesced and coalesced memory access represent the data access pattern when kernel threads access the global memory, not for the data transfer on PCIe (host to device or device to host).

Hide

ED: Publish subject to technical corrections (03 Sep 2015) by Adrian Sandu

AR by Melin Huang on behalf of the Authors (11 Sep 2015) Author's response Manuscript

Short summary

To expedite weather research and prediction, we have put tremendous effort into developing an accelerated implementation of the entire WRF model using GPU massive parallel computing architecture. This paper presents our efficient GPU-based design on WRF YSU PBL scheme. Using one NVIDIA Tesla K40 GPU, the GPU-based YSU PBL scheme achieves a speedup of 193x with respect to its runtime on 1 CPU core. We can even boost the speedup to 360x with respect to 1 CPU core as two K40 GPUs are applied.