Fast IR Transfer Calculations using GPUs: JURASSIC-GPU v2.0

P.F. Baumeister & L. Hoffmann

Referee comments

Summary

The paper describes the JURASSIC radiative transfer model, which can be used to

simulate the measurements of hyperspectral infrared instruments in both

limb and nadir-viewing geometries. This is a fast, band-averaged RTM using

the Emissivity Growth Approximation to approximate transmittances of complex

paths along the line of sight.

Noting the expected large increase in the data volume from hyperspectral

satellite instruments in the next few years, the second part of the paper

describes some experiments implementing the code on a GPU system. However, this

it outside my area of expertise so I limit my comments on this part to a few

trivial, grammatical suggestions.

Main Comments

1) There should be a bit more acknowledgement and review of the approaches used

in some of the other fast RTMs already used operationally. For example RTTOV

which, like JURASSIC, is a band-averaged model, widely used in NWP applications,

which has a different technique ('predictors') for evaluating transmittance of

complex paths. Also the OSS model from AER, a high-speed monochromatic model

which is a candidate for operational processing for future instruments.

2) Clearly the number of path segments has a significant impact on

computation. However it seems that the JURASSIC ray-tracing doesn't distinguish

between the step size required for accurate ray-tracing (including,

particularly, the integration of absorber amount within each segment), and the

segmentation used for the subsequent EGA calculation and radiative transfer.  In

particular if a coarser spacing can be applied to the latter, there would be a

significant saving in CPU time.  Even so, I am surprised that the ray-tracing

takes up so large a fraction of the computation time.

3) Regarding the 'continuum approximation' (L220 onwards). I believe a similar

assumption is possible if the spectral features from the different absorbers are

uncorrelated across the band. However, with hyperspectral instruments you are

dealing with bands of less than 1 wavenumber, where there may be just a couple

of individual lines, perhaps one from the target molecule and one from the an

interfering molecule. So both the uncorrelated and continuum assumptions would

seem to break down. Is this a fundamental limitation on the accuracy of JURASSIC

compared, say, to other band models such as RTTOV, or do have you some proposed

mechanism for dealing with such cases?

4) Figure 1 and the associated text (L238) are very misleading. This suggests

the Beer-Lambert Law is being used, ie complex path transmittance is simply the

product of component homogeneous path segment transmittances. This is, of

course, only true for monochromatic transmittances, not band-averaged

transmittances as shown here.  However, the error seems confined to just this

part of the manuscript and is not, in fact, how JURASSIC actually evaluates path

transmittances (as detailed in 2.6).

5) There are various references to using the 'bisection' method for

interpolating the look-up tables (L256, L385). If I understand correctly this is

basically an iterative approach, progressively dividing the tabulation axes by

factors of two, and a significant CPU overhead.  However, since your tabulation

axes seem to be at regular intervals, surely you can evaluate your required

interpolation nodes directly?

Minor comments

L18: Suggestion: replace 'of about 4 and 15 um of wavelength' with 

     'from 4-15 um'

L19: Suggestion: start sentence 'Here, in the longwave part ...' to clarify that

     you regard the mid-IR as part of the longwave region

L30: 'next to' ? Do you mean 'as well as' ? There should probably be a reference

     or two to go with this statement.

L34: Perhaps also mention IASI-NG which will have a higher data rate than IASI,

     and the Chinese FengYun-4 GIIRS instrument already in orbit.

L46: 'Massive amounts'. Perhaps 'a large number' sounds less excitable.

L47: Suggestion '... provide a speedup factor of more ...'

L60: 'Russell' (two 'L's) (also missing in several other places)

L69: 'Michelson' (one 's')

L71: Presumably 'for the Atmosphere' still contributes a part of the 

    CRISTA-NF acronym

L101: It seems strange that JURASSIC should used a fixed value of Re. Can this

      really not be varied to match a particular viewing geometry (preferably

      without having to recompile the entire code)?

L115: Suggest '4--15' (in LaTeX, or en-dash) to indicate continues range, rather

     than '4...15' which implies a sequence.

L158: Delete 'being' - superfluous.

L175: The O2 and N2 continua arise from 'Collision Induced Absorption' which, in

    my mind, is a very different process from the CO2 and H2O continua which are

    largely the accumulation of line wings. Nowadays, HITRAN has a completey new

    file category for CIA (including these O2 and N2 continua), in parallel with

    the line database and molecular cross-sections.

L183: emissivity, in this equation, should be defined either before or

      immediately afterwards. L187 is a bit too remote.

L219: '...small, if...' the comma is superfluous (in English grammar if not

     German!) 

L226: How do you establish the pressure and temperature associated with the

      homogeneous conditions?

L245: It might be useful to mention what units 'u' is measured in. And there

     should be an integral before the 'ds', along the length of the 'cell'.

L275: Reference to Eq(19) suggests that a 'Delta e' term might be included after

      all.  I assumed (from L.221) that it was not.

L289: epsilon_2 - you seem to have switched notation. Here I understand this to

     represent the emissivity of both path segments combined but in L282 it

     would be the emissivity of just segment#2

L292-294: Is the appearance of the wavenumber nu significant or an oversight? In

     any case, I couldn't understand the point being made here - please try 

     rewording it.

L303: Suggestion: 'computer' or 'computationally intensive' (unless 'compute

     intensive' is generally accepted usage - it's not really my field!).

 

L377: 'monotonically' rather than 'monotonously' (which has now acquired a quite

     different meaning in English, as in 'tediously' or 'boringly').

L396 (and following). According to Wikipedia, kB and MB seem to be the

    recommended abbreviations.

L448: 'Wizard' (one 'z')

L447: (pedantically) 'fewer' rather than 'less' for discrete, countable items.

      Also L513, L519.

L473: Suggest 'from 2120 to 2605' rather than 'at 2120 to 2605', and again later

     in the same line.

L489 (and subsequently) I'm not keen on the introduction of a new unit, the

     'kray'. I'd prefer 6.2x10^3 rays/s  

L493: 'After loading ... this context'. Wording could be improved ('yet' can

    mean both 'not yet' and 'although', which makes the overall meaning 

    ambiguous). 

L503: 'deduct' - do you mean 'deduce', or did you really mean 'deduct' as in

    'subtract' ?

L514: '2k' expand to 2000 (as with 1000 in previous line).

L528: 'performance ... 11% higher' Does this mean that the required time is 11% 

     longer or shorter? 'higher' could be interpreted either way.

L573: '... restructuring ... has ...' (singular)

L575: ' ... both a GPU ...' (remove superfluous comma after 'both')

------------------

The authors present a detailed manuscript regarding the implementation or adaptation of the JURASSIC (non-scattering) radiative transfer code/model for use with graphics processing units.

The first few sections give a short, but rather dense introduction into the method of emissivity growth approximation, a technique originally used for limb-type observation geometries, but also compatible with nadir-type observations. Experts in that field will probably have no trouble following Section 2, however outsiders might find that section a little jarring. There are some minor (apparent) inconsistenties in notation, but it is clear the focus of the paper is Section 3, with Section 2 being more of a recap - the authors provide enough references for further reading on the EGA method.

From Section 3 onwards, the manuscript delves into the challenges of transferring the JURASSIC implementation onto GPU architecture. This portion (which is also the central part of the publication) highlights some very interesting aspects of GPU implementations of an established code base, which itself is an interesting read to anyone who is looking to attempt a similar task (even if it is a different forward model). Towards the end, the authors make fitting comparisons of their GPU implementation compared to the CPU-only code, and then finally conclude with some easy-to-grasp numbers about the overall performance gain and an estimate on power consumption.

The whole manuscript was generally pleasant to read and offers interesting insight into the difficultions and considerations of writing GPU-accelerated scientific code. While it stays focused on the chosen algorithm itself, there are definitely some take-aways for other algorithms. 

Minor suggestions:

General:

There seems to be no mention of linearization of the JURASSIC model for use in inversions. I suspect there is a straightforward way of calculating various Jacobians. I understand the paper focuses on the forward model aspect exclusively, however it would be nice to have it mentioned somewhere in the text (even if it is an acknowledgement that linearization has not been investigated).

The Greek “mu” when used in units should be replaced by the \upmu (or equivalent) symbol.

L155:

(Very minor) “T” in the Planck function is not explained in text.

L245:

The equation should probably either say “du = qp / (kT) ds” or have an integral wrapped around the right-hand side.

L250++:

Indices i,j,k are a little confusing as they’re not explicitly written down.

L275 and following:

The emissivity notation (p,T,u) changed to (u,p,T) and then further (p, T, p, T, u) further down. This is a little confusing as well and would benefit from some more explicit explanation. 

L350:

(comment) I'm somewhat surprised that the emissivity look-up tables are not manually copied into GPU memory; I might be missing something, but it seems like the look-up tables would be one constant array (per species) and could be kept in memory for any given wavenumber window of the user's choice. In Section 3.2.3, the authors investigate cold/warm caches, so there must be a reason as to why this was not done - but if feels like a somewhat obvious choice to load the tables into GPU memory at the very start of the algorithm.