Ensemble-based DA is a useful approach to improve the forecast accuracy of volcanic ash transport. However, if it is time-consuming, it cannot be taken as efficient due to the high requirement on speed for volcanic ash DA (see Sect. ). Based on this consideration, we need to analyze the computational cost of a conventional volcanic ash DA system.

As introduced in Sect. , the total execution time of conventional EnKF comprises four parts, i.e., initialization, forecast, analysis, and other computational cost. The initialization time includes reading meteorological data, initializing model geographical and grid configurations, reading emission information, initializing stochastic observers for reading and transforming observations to the model grid, and initializing all the ensemble states and ensemble means. The forecast time is obtained from Eq. (), while the analysis time corresponds to the computational sum from Eqs. () to (). The other computational time includes script compiling, setting environment variables, and starting and finalizing DA algorithms.

The evaluation result of the conventional EnKF is shown in Table  (the middle column). It can be seen that the total computational time (4.36 h) is relatively large compared to the simulation window (3.0 h, i.e., from 09:00 to 12:00 UTC, 18 May 2010), which is too much in an operational sense. Therefore, in this study, we aim to accelerate the computation to within an acceptable runtime (i.e., requiring less runtime than the time period of the DA application).

It can also be observed from Table  that the main contribution to the total execution time is the analysis step. Compared to the initialization and forecast time, the analysis stage takes 72 % of the total runtime. Due to the expensive analysis step, although some approaches (such as MPI-parallel I/O , domain decomposition ) can potentially accelerate the initialization and forecast step, the effect on the final acceleration of the total computational cost is little. Therefore, to get an acceptable computational time, the cost reduction in the analysis step is the target. One may wonder that since the number of observations is small, why does analysis take so much time? The large state vector seems to be left responsible for the problem. To know the exact reason, the detailed computational cost of the analysis step must be evaluated.

We start with the formulations of the analysis step. The analysis step is represented by Eq. (), which can be written in a full matrix format with Eq. (), An×Na=An×Nf+Kn×m(Ym×N-Hm×nAn×Nf), where the subscripts represent the matrix's dimensions. Af and Aa represent the forecasted and analyzed ensemble state matrix, and are respectively built up from ξf and ξa with N ensembles. The measurement ensemble matrix Y is formed by an ensemble of y+v (see Eq. ). H is the observational matrix, which is used to select state variables (at measurement locations) in the full ensemble state matrix corresponding to the measurement ensemble matrix Y. n is the number of model state variables in a three-dimensional (3-D) domain, i.e., ∼106 in this study (see Sect. ). m is the number of measurements at one assimilation time, which depends on the measurement type. For aircraft in situ measurements used in this study (see Fig. c), two measurements are made at each time by one research flight, so that m is 2 here. N is the ensemble size and is taken as 100 in this study. As described in Eq. (), the ensemble perturbation matrix Lf in EnKF can be re-written as Ln×Nf=An×Nf-A¯n×Nf=An×Nf(IN×N-1N1N×N)=An×NfBN×N, where I is an N×N unit matrix and 1 is an N×N matrix with all elements equal to 1. Thus, Lf=AfB where BN×N is introduced to represent (IN×N-1N1N×N), so that HLf=OfB, where Om×Nf is used to represent (HAf). Here we explicitly express Lf and HLf in the form of Af and Of, respectively. This is because in our volcanic ash DA system, Af and Of are two of the three inputs (another one is the measurement ensemble matrix Y for the analysis step). These are the three inputs used for actual computations in the analysis step. As shown in Fig. a, Af is obtained from the forecast step, and Of and Y are acquired from our stochastic observer module (see Fig. a) which is used for a volcanic ash transport model to integrate geophysical measurements. With the input Y, the measurement error covariance R, as introduced in Eq. (), can then be computed with Rm×m=1N-1(Ym×N-Y¯m×N)(Ym×N-Y¯m×N)′=1N-1(YB)(YB)′.

Based on previous definitions and Eqs. () to (), the analysis step can be reformulated as follows: An×Na=Af+K(Y-HAf)=Af+PfH′(HPfH′+R)-1(Y-HAf)=Af+1N-1Lf(HLf)′[1N-1(HLf)(HLf)′+1N-1(YB)(YB)′]-1(Y-HAf)=Af+AfB(OfB)′[(OfB)(OfB)′+(YB)(YB)′]-1(Y-Of)=Af{I+B(OfB)′[(OfB)(OfB)′+(YB)(YB)′]-1(Y-Of)}=An×NfXN×N, where XN×N={I+B(OfB)′[(OfB)(OfB)′+(YB)(YB)′]-1(Y-Of)}. Equation () shows how the analysis step is performed in a volcanic ash DA system. In order to accelerate the analysis step, the most time-consuming part must be reduced. Figure b shows estimations of the computational cost for each procedure in the analysis step. Considering that the state number n (∼106) is significantly larger than the measurement number m (m=2 here) and the ensemble size N (N=100), the most time-consuming procedure in the analysis step is thus the last one, that is Aa=AfX with a computational cost of O(nN2). Therefore, in our volcanic ash DA system, this part is the most time-consuming part in the analysis step. Note that the procedure [(OfB)(OfB)′+(YB)(YB)′]-1 for singular value decomposition (SVD) in our study is not time-consuming, which is different from applications of reservoir history matching . This is because of the SVD procedure costs O(m3), and due to the measurement size on the order of the size of the state in those cases, the SVD procedure thus requires a huge computational cost for reservoir DA.

Analysis in the previous section shows that Aa=AfX is most expensive in the analysis step. Each column of Af is constructed from a forecasted ensemble state; thus, the dimension of Af is n×N. In each column, the element values correspond to volcanic ash concentrations in a 3-D domain. Figure  shows the coverage of all ensemble forecast states at a selected time, 10:00 UTC, 18 May 2010, without loss of generality. A common phenomenon can be observed: that is, only a part of the 3-D domain is filled with volcanic ash. The ash clouds only concentrate in a plume which is transported over time. This is because volcanic eruption is a fast and strong process. The advection dominates the transport, and the volcanic ash plume is transported with the wind. This is a particular characteristic of volcanic ash transport, in contrast to other atmospheric-related applications such as ozone , SO2 , and CO2 . For those applications, the concentrations are everywhere in the domain, the emission sources are also everywhere, and observations are available throughout the domain too (especially for satellite data), whereas for application of volcanic ash transport, the source emission is only at the volcano; thus, usually only a limited domain is polluted by ash. As shown in Fig. , in the 3-D domain with a grid size of 3.888 × 106, the number of grids in the area with volcanic ash is counted as 1.528 × 106, whereas the number of no-ash grids is 2.36 × 106. Note that shown in the figure are accumulated ash coverages of all ensemble states; thus, in the no-ash grids, there is no ash for any of the ensemble states. Thus a very large number of rows in Af are zero corresponding to the no-ash grids. These zero rows in Af have no contributions to Aa=AfX, because a zero row in Af always results in a zero row in Aa. Therefore, for the case of Fig. , 2/3 of the computations are redundant and can be avoided. To realize this, one may think to limit the domain for the entire assimilation step; then, the number of zero rows certainly would be largely reduced. This is actually incorrect, because these zero rows are changing along with the transport of ash clouds, and are not constant at each analysis step. So the full domain must be considered and it should be adaptive (choose different zero rows according to different Af at different analysis times).

Here we introduce item nnoash to represent the number of zero rows in the ensemble state matrix Af, and use nash to represent the number of other rows (also, nash represents the grid size of ash plume). When computing Aa=AfX, to avoid all the computations related to nnoash rows with zero elements, the index of other nash rows must first be decided. This index is meant to reduce the dimensions of Af. After getting a Aa with a dimension of nash×N, the index will be used again to reconstruct the full matrix Aa with the dimension of n×N. Based on this idea, we propose a mask-state algorithm (MS) which deals with the time-consuming analysis update. MS includes five steps. i.

Compute ensemble mean state A¯f. The mean state A¯n×1f can be easily computed by averaging An×Nf along N columns. Due to all elements in An×Nf corresponding to ash concentrations, all elements in An×Nf are larger than or equal to zero. The index of non-zero rows in A¯n×1f is thus equivalent to that in An×Nf. The computational cost for this step is O(nN).

According to the derivations of MS, the computational costs related to zero rows are avoided. Here the “zero rows” do not equal “zero elements”. The former corresponds to the regions where there is no ash for all the ensemble members, while the latter also counts the no-ash regions specifically for some ensembles. Certainly the consideration of all “zero elements” can include all the sparsity information of the ensemble state matrix, but extra computations and memories must be spent on searching the full matrix An×Nf with a computational cost of O(nN) and storing a mask-state matrix with dimensions of n×N. This is expensive compared to constructing the mask array in procedure (ii). Actually, after a careful check of the volcanic ash ensemble plumes, there is no “bad” ensemble which is really different from others. Although the concentration levels in ensemble members are distinct, the main direction and the occurrence to the grid cells are more or less the same. This means that the “zero rows” actually more or less equal “zero elements” but are much faster than the way with “zero elements”, which confirms the suitability and advantage of procedure (ii). Probably when there are big meteorological uncertainties, the “zero elements” will be much larger than “zero rows”. In this case, how to make use of the sparsity information in the ensemble state matrix will be considered in future.

Based on procedures of MS, the computational cost of Aa=AfX can be reduced. However, without a careful evaluation, we cannot conclude MS is fast, because the algorithm also employs other procedures. If these procedures (i), (ii), (iii), and (v) are much cheaper than the main procedure (iv), MS can definitely speed up the analysis step, and vice versa. Now we analyze MS's computational cost, which can be summed as O(nN) + O(n) + O(nashN) + O(nashN2) + O(nN), i.e., O(nN+nashN2). Thus, the computational overhead involved in transforming the full matrix to a small one (i.e., O(nashN) for procedure (iii)) has little effect on the total computation cost of MS (i.e., O(nN+nashN2)). However, the computational overhead of transforming the small matrix to the full one (i.e., O(nN) for procedure (v)) does contribute a part, which cannot be ignored, to the total MS's computational cost. The computational cost without MS is O(nN2).

The comparison between both costs (with and without MS, i.e., O(nN+nashN2) and O(nN)) indicates when the number of non-zero rows (nash, i.e., the number of grids with ash) of the forecasted ensemble state matrix satisfies nash<N-1Nn; then, MS can accelerate Aa=AfX. Here, O(nN+nashN2) and O(nN) are on the same order when nash<N-1Nn. The larger the difference between nash and N-1Nn, the better the speedup can be achieved. According to this analysis, and the characteristic (e.g., nashn approximately equals 13 in this case) of volcanic ash transport as described in Sect. , the relation is certainly satisfied and is actually nash≪N-1Nn (significantly smaller) for our study. Therefore, for our volcanic ash DA system, with MS, the computational cost for the time-consuming part Aa=AfX is O(nashN2), which is much reduced compared to O(nN2) with conventional computations.

The relation nash<N-1Nn indicates whether we would have speedup by the MS method; actually, it can be extended to Eq. (), Sms=O(nN2)O(nN+nashN2)=Onnash, which explicitly specifies the expected amount of speedup (Sms) of Aa=AfX by the MS algorithm. In this case study, N is taken at 100 and nashn≈13, so Sms is approximately 3.0.

According to Amdahl's law , the total computational speedup (Stotal) by MS can be predicted by Eq. (), Stotal=1(1-pms)+pmsSms, where pms is the proportion of the computational cost of Aa=AfX in the overall DA computations. It has been evaluated that the computational cost of Aa=AfX dominates the analysis step (see Fig. b); thus, the proportion of the computational cost of Aa=AfX approximates the proportion of the analysis step in the total DA computations (i.e., pms≈72 % in this case, as described in Sect. ). Therefore, based on Eq. (), the maximum (“ideal”) computational speedup can be predicted to be 11-pms (i.e., ≈3.57 for this case study) when Sms approximates infinity. However, this is not the actual speedup because Sms is in fact specified by Eq. (). Based on the discussions above, Stotal can therefore be estimated by Eq. () at ≈2.0 in this case.

Analysis of the algorithmic complexity of MS shows that MS is an efficient approach to reduce the computational cost of the time-consuming Aa=AfX. Now MS will be applied in the real volcanic ash DA system, to investigate whether in practice it can speed up the analysis step well. We perform MS in the conventional EnKF, which means initialization and forecast steps are all computed as the conventional EnKF. The only difference between MS-EnKF and conventional EnKF is that in the former MS is employed for the analysis step, and in the latter is the standard analysis step. The result and related specifications are shown in Table . As introduced in Sect. , the forecast step has been configured with the conventional parallelization; thus, N+2 (102 here) cores are actually used (one core for the DA algorithm, the other N+1 cores for the parallel forecast of N ensemble members and one ensemble mean). It can be seen from Table  that MS indeed largely accelerates the analysis step (as expected, by a factor of about 3.0 for this study), which confirms the theoretical cost evaluation. The detailed experimental time for each step of MS is shown in Table . As expected, the dense–dense matrix multiplication in step (iv) takes the largest part (i.e., 0.8474 h for this case study) of the total computational time (0.87 h) of MS. However, step (iv) has been a big improvement compared to the case without MS (3.14 h; see Table ), which is because the computational time for the other steps (e.g., steps (i–iii) cost only 0.0156 h to reduce the size of the ensemble state matrix) is little and ignorable. Note that the total computational time of Aa=AfX with MS (i.e., 0.87 h in Table ) is not exactly equal to the computational time of the MS-EnKF analysis procedures (i.e., 0.88 h in Table ). The subtraction (i.e., 0.01 h) corresponds to the summed computational time of all the other analysis procedures (i.e., procedures 1–8) except for Aa=AfX (see Fig. b and Table ).

MS is now experimentally proven as efficient to significantly reduce the computational time for the analysis step during volcanic ash DA. Note that it can also be observed that the computational time for the “other” parts in Table  (such as operations for setting environmental variables, starting and finalizing DA algorithms, as mentioned in Sect. ) is slightly reduced by the MS method (i.e., 0.03 h in this case). This is because in the conventional EnKF, the ensemble mean state A¯f is calculated in the “other” parts as an output to finalize the DA algorithms, while in MS-EnKF, the calculations of A¯f are needed and directly involved in the “Analysis” part.

The result shows that, benefitting from the success of a reduced analysis step, the overall computational cost indeed gets significantly reduced. The total execution time is 1.95 h, which is less than the simulation window of 3 h (09:00–12:00 UTC, 18 May 2010). This result satisfies our goal to accelerate the computation to an acceptable runtime (i.e., requires less runtime than the time period of the DA application). Therefore, aviation advice based on the MS-EnKF can be provided as not only accurate, but also sufficiently fast. Note that the result (1.95 h) is obtained after the volcanic ash is transported to continental Europe. If the assimilation is performed in the starting phase of volcanic ash eruption (when aircraft measurements are available), a more significant acceleration would be obtained. This is because in this case the volcanic ash is only transported in an area near to the volcano; thus, the number of no-ash grid cells will take a large proportion (much higher than 2/3 for this case study) of the full domain.

There is another interesting point. According to Fig. , the ash grids comprise 39.3 % of the total grids. Thus, the minimum computing time by using MS to utilize this model's characteristic should be ≈1.234 h (i.e., 0.393 × 3.14 h). However, the experimental result shows that the computational time goes down to 0.88 h (see Table ). One reason for this time decrease is that when the size of the matrix is reduced, the memory access cost also goes down (e.g., through better cache usages). Another possible reason is that the ash grid number actually decreases with time (not always taking 39.3 % of the total grid number), due to ash sedimentation and deposition processes .

Note that in this study we only perform the commonly used ensemble parallelization for the forecast step (already efficient compared to the expensive analysis step) but do not choose model-based parallelization (e.g., tracer or domain decomposition). As specified in Table , no parallelization is implemented on the six tracers. This is because due to the important aggregation process , there are big dependencies between different ash components and thus it does not make much sense to parallelize them. As for domain-decomposed parallelization , it is not efficient for our application. This is because volcanic ash is special in the sense that the model is only doing computations in a small part of the domain (i.e., there are no data in a rather large part of domain), and this active part is continuously changing. Thus, a fixed domain decomposition is not very useful here because of the changing plume position. In this sense, some advanced approach such as adaptive domain-decomposed parallelization should be adopted to achieve additional acceleration to the volcanic ash forecast stage. This is an interesting subject for future application, when a more complicated model is employed, only ensemble parallelization may be not enough for the forecast stage.

According to Sect. , MS has proven to be capable of solving the computational issue of Aa=AfX. Motivated by the model's characteristics, MS was proposed from an application's perspective and achieved a good result by managing the irregular sparsity in our complicated volcanic ash DA system. The main reason why MS is efficient is that the sparsity of Af can be well utilized by MS. In Sect. , we only performed the comparison between MS and the case of full storage dense matrices. However, the problem abstracted here (AfX) is actually a sparse–dense matrix multiplication (SDMM) problem, since Af is sparse and X is dense (see Eq.  for X). Thus, one may wonder what the result would be if the comparison of MS is made to more standard sparse matrix methods, such as compressed sparse row (CSR)-based methods , which are commonly used for sparse matrix vector/matrix multiplication.

Before we make the comparison, we need to first address the intrinsic problem when considering standard sparse matrix methods in EnKF for Aa=AfX. The issue is that it is not possible to directly generate a sparse storage format (e.g., CSR) of Af without first generating the full matrix Af. This is mainly because Af comes from the model-driven ensemble forecast step, where each Af column corresponds to one member of the ensemble. During model forecast, we know there are indeed no-ash grids. However, it is not certain where the plume is exactly after one forecasting time step. This is highly dependent on the weather conditions and the model processes (e.g., advection and diffusion for horizontal grids, sedimentation and deposition for vertical grids). Thus, a fixed and wide domain is usually needed by the model to avoid complications, resulting in the generation of the full storage of Af (to be used in Aa=AfX). Therefore, if we want to implement a CSR storage format for the sparse matrix Af, we must first generate the full storage Af from the ensemble forecast step, and then we generate the three CSR arrays based on Af.

Generating CSR arrays is usually much more expensive (computationally) than a single sparse matrix-vector multiplication (SpMV). Thus, if we generate CSR arrays for only performing one-time SpMV, it would be meaningless from HPC's point of view. Fortunately, this is not the case for Aa=AfX (i.e., SDMM), which can actually be considered as N-times SpMV. (Here, X has N columns, and one SpMV means the multiplication of Af by one column of X.) Thus, CSR-based SDMM might also be a candidate in reducing the computation time of Aa=AfX. It remains interesting to compare the performance of CSR-based SDMM and MS in dealing with Aa=AfX for our study case.

To implement CSR-based SDMM for Aa=AfX, the three CSR arrays for Af (denoted val, col_idx, and row_ptr in this study) need first to be generated. The array val of size nval stores non-zero values of Af, where nval≈nashN. (In this study case, n = 3.888 × 106, nash≈13n, and N = 100.) The array col_idx of the same size nval stores the column index of the non-zeros. The array row_ptr saves the start and end pointers of the non-zeros of the rows in Af. The size of row_ptr is n + 1.

After the above three CSR arrays are generated, CSR-based SpMV can be performed for multiplying Af by a column vector v in X (see Algorithm 1 in Fig. a). With that, Algorithm 2 (Fig. b) can be implemented by looping Algorithm 1 for N times to obtain Aa=AfX. The experimental result of CSR-based SDMM is shown in Table , where all the environmental conditions (such as the DA system, the programming environment) are the same as the case of MS. This gives a fair comparison between CSR-based SDMM and MS. In addition, for a pure algorithmic comparison with the serial MS, here the CSR-based SDMM is also performed in a serial case.

From Table , we can first confirm that the computational time (i.e., 0.0407 h) for the generation of the three CSR-based arrays (val, col_idx, and row_ptr to represent the sparse matrix Af) indeed takes more time than the computational time of one CSR-based SpMV (i.e., 0.0117 h). Thus, there is little value in performing sub-step (i) (see Table ) if only one SpMV (i.e., sub-step (ii)) is needed. However, to get all N (i.e., 100) columns of Aa, the sub-step (ii) is looped for N times, resulting in an ignorable impact of sub-step (i) on the total computational time (i.e., 1.21 h) of CSR-based SDMM.

The result of CSR-based SDMM also shows that the standard sparse matrix methods can reduce the computational time of Aa=AfX, by comparing with the conventional way in Table . However, it can also be observed that the computational time of CSR-based SDMM is larger than MS (i.e., 1.21 h versus 0.87 h in Table ). Thus, although application of sparse matrix multiplication methods is positive, it is still slower than MS on our problem.

In the CSR-based SDMM, only non-zero elements in Af participate in the multiplication between Af and X; thus, redundant computation (related to zero elements in Af) is avoided. So the computation time of AfX is reduced with CSR-based SDMM. In the following, we analyze the performance difference between CSR-based SDMM and MS.

Firstly, from the programming's perspective, in CSR-based SDMM, the loop number for the rows of Af is from 1 to n (see Fig. a), while the corresponding loop number in MS is from 1 to nash (see step (iv) of MS in Sect. , nash≈(1/3)n). Although only non-zero elements are used in the multiplication in CSR-based SDMM, the length of the outer loop is still n (much larger than nash), which is the essential reason that MS is faster than CSR-based SDMM. Note that as discussed in Sect. , there are many zero rows in Af; thus, CSR-based SDMM actually does nothing when it comes to a zero row, but still needs to execute the loop. Within each loop number, it has to check the information from row_ptr (size n+1), where the value corresponding to a zero row is usually set to be the value in row_ptr corresponding to the first subsequent non-zero row.

Secondly, with respect to the algorithm, CSR-based SDMM utilizes the sparsity of Af by its generation of three CSR arrays, while MS not only utilizes the sparsity information of the sparse matrix Af, but also utilizes the consistency of ensemble forecasts; that is, ensemble forecasted states are not consistent in values but usually consistent in non-zero locations. This is a typical property in ensemble-based DA, resulting in N ensemble plumes being different in concentration values but having similar transport directions/shapes (see Sect. ). Thus, most of the zero elements in Af are actually in zero rows of Af for an EnKF application, which leads to a small number of non-zero rows (nash) compared to the full number of rows (n) of Af. Therefore, only considering nash rows in Ãnash×NfXN×N (see step (iv) of MS in Sect. ) is more advantageous for an EnKF application than considering all n rows in CSR-based SDMM. Based on the above analysis, MS can be considered a specific sparse matrix method, which typically works for ensemble-based DA applications.

It is useful to apply standard sparse matrix methods (e.g., CSR-based SDMM) for our assimilation application. The accelerated analysis step by CSR-based SDMM (1.22 h; see Table ) also reduces the total computational time (i.e., 2.29 h; see Table  for the computational time of initialization, forecast, and others) to an acceptable level (i.e., less than 3 h for our case study). In practice, due to the better performance of MS than CSR-based SDMM, we will use MS as a better choice for assimilation applications. In addition, we do not only intend to present MS, but also intend to reveal which part is the most time-consuming part for plume-type assimilation of in situ observations.

For volcanic ash forecasts, only a relatively small domain is polluted compared to the full 3-D domain, so that MS can work efficiently. Using MS is also applicable for many other DA problems, where the domain is not fully polluted by the species. It does not matter what the emission looks like and whether the releases are short- or long-lived species. Given an assimilation problem, the only restriction for MS to gain an acceleration is whether the whole domain is fully polluted or partly polluted. The assimilation problems where MS can achieve the acceleration effect on the computations of Aa=AfX include all the volcanic-related ash/gas assimilations, e.g., assimilation of satellite data/LIDAR data/in situ data; (sand/desert) dust-storm-related assimilation; tornado-related assimilation; assimilation of exploding nuclear plants or factories; chemicals or oils leaking into seas; global (forecast) fire assimilation; and assimilation of environmental pollutant transport, e.g., severe smog. In addition, for DA applications (e.g., ozone, SO2) where pollutants spread over the whole domain, usually the focus is only on the high concentrations, and a threshold can be set to ignore the very low values without losing the necessary assimilation accuracy. In this case, MS can also lead to a potential acceleration since many very low concentrations can be explicitly truncated to be zeros.

It has been analyzed that when the number of non-zero rows (nash, i.e., the number of ash grids in a 3-D domain) of Af satisfies nash<n, MS can work faster than standard EnKF. For volcanic ash application, because nash is much less than n, the acceleration is quite large. Hence, in this case, we propose to embed MS in all ensemble-based DA methods because it is fast and the implementation using MS is exact to the standard ensemble-based methods; i.e., it does not introduce any approximation in view of MS procedures. Actually this proposal can be extended to all real applications, even if the condition is not satisfied. This is because in this case the computational cost of MS for Aa=AfX becomes O(nN2), which is the same as that of using the standard assimilation (shown in Fig. b). Therefore, if the state numbers are equal to or close to the total number of grid points in the domain, the added computational cost of using MS is very small (negligible), so that the computational time with MS is almost the same as the time of using the standard approach, whereas when the condition nash<n is satisfied, MS will accelerate the analysis step. Thus MS is generic and can be directly used in any ensemble-based DA, and this acceleration can be automatically realized for some potential applications, without spending time investigating whether the condition is satisfied. In a real (or operational) 3-D DA system, MS can be easily included; i.e., we only need to invoke the MS module when computing Aa=AfX, without any other change to the current framework. Note that MS is applicable in ensemble-based DA but not in variational-based DA. This is because in a variational-based DA system, the minimization of a cost function is mostly operated within several/many continuous time steps ; thus, it is convenient to always use the full (i.e., non-masked) domain to represent different state matrices (corresponding to different time steps in variational-based DA).

As stated in Eq. (), the speedup of the MS method is approximately the inverse of nashn. So far there are no statistical data on the value of nashn. Considering the problem of volcanic ash transport, there is one emission point (at the volcano); all the ashes in atmospheres are transported by the directional wind drive from the same source point. Thus volcanic ash cloud is actually transported in a shape of a plume, which in general does not cover the full but only a small part of the 3-D domain. At the start phase of a volcanic ash eruption, nashn is much smaller than 1.0 (started from 0). During transport over a long time (1.5 months for this case study), nashn increases to approximately 1/3. Therefore, the speedup of MS in ensemble-based volcanic ash DA will be significant.

Based on the formulation of MS, one may think it can be taken as a localization approach . There is indeed a similarity between MS and the localization approach, in a sense that when computing Aa=AfX, both get rid of a large number of cells and only do computations related to the selected grids. These two algorithms are however functionally different. This is because the localization approach is meant for reducing spurious covariances outside a local region which is built up around the measurement; thus, the results with and without localization approaches are different, while MS is developed for the acceleration purpose. The masked region is discontinuous and independent of locations of measurement, but dependent on the model domain. Thus, there is no difference in the assimilation results between using MS and without using it. Therefore, based on the functional difference, MS cannot be taken as a localization approach.

In this study, we do not employ the localization strategy in the analysis step, because we use a rather large ensemble size of 100 to guarantee the accuracy, as introduced in Sect. . But for some applications (e.g., ozone, CO2, sulfur dioxide), especially when assimilating satellite data, localization is a necessary approach and has been widely used in reducing spurious covariances . In these cases, because the localization approach forces the analysis only to update the state within a localization region, one may think that localization could replace MS and that there would be no significance in employing MS. Actually this is not correct. We explain the reason as follows.

The localization approach is usually realized in Eq. () by employing a Schur product of a localization matrix and the forecast error covariance matrix given by K(k)=(f∘Pf(k))H(k)T[H(k)(f∘Pf(k))H(k)T+R]-1. The Schur product f∘Pf in Eq. () is defined by the element-wise multiplication of the covariance matrix Pf and a localization matrix f. f is defined based on the distance between two locations; thus, it is dependent on the domain and needs information on the full ensemble state locations. In this way, f∘Pf can contain more zeros than Pf, but the dimensions are not changed, so that the computations related to f∘Pf are actually not reduced. Therefore, we can understand the localization approach in the analysis step as that the states within and outside a local region are both updated with increments, but just the increments outside the region are zero (which seems like not updating). This is also the reason why the localization approach is not meant for acceleration, but only for reducing spurious covariances. Now it is clear that localization cannot replace MS. Actually both can be performed together in dealing with the time-consuming part Aa=AfX. The localization approach can first transfer Af to a localized matrix with more zero rows. Then MS can be used to accelerate the multiplication of the localized matrix and X. In this way, MS is expected to accelerate Aa=AfX with a high speedup rate, because the computational cost of more zero rows in the localized ensemble state matrix is avoided.

Motivated by the model's physics, the implementation MS currently is for the serial case. This implementation has reduced the computation time to an acceptable time (i.e., the simulation time is less than the period of forecast in real-world time). It is however interesting to discuss the potential of parallelization of the dense–dense matrix multiplication (Ãnash×Na=Ãnash×NfXN×N) in step (iv) of the algorithm (see Sect.  and Table ). The related matrix multiplication can be easily parallelized on multiple processors. Optimization and evaluation on the parallelized MS will be considered in future. For the current case study, the computational time (3.13 h; see Table ) for an “ideal” reduction by parallelization of MS is not much larger than the acceleration (already) gained by MS (2.26 h, subtraction between 3.13 h and 0.87 h; see Table ). Therefore, from the application's perspective, further acceleration by parallelization is not required.

Alternatively, one may also consider to (1) directly parallelize the expensive matrix multiplication of An×Na=An×NfXN×N, without first performing MS, or (2) implement CSR-based SDMM (see Sect. ) with parallelization. Both are possible alternative approaches to accelerate the expensive matrix multiplication. The first approach can be implemented by a user's own designed parallelization, or by utilizing scaLAPACK (https://www.netlib.org/scalapack/, where the main function is “pdgemm”). The second approach can be realized by using some general parallel sparse–dense matrix multiplication methods (e.g., sending each column of X and three CSR arrays of Af to each processor to calculate each column of Aa) or using a good parallel algebra library like PeTSC (https://www.mcs.anl.gov/petsc/) which allows users to specify own orderings and comes with machine optimized parallel matrix–matrix multiplication operations. However, given the fact that MS can also be parallelized using similar ways or the same libraries, it is fair to not consider parallelization for all cases (i.e., using MS, not using MS, using CSR-based SDMM). Actually, the parallelization in MS could be performed much more easily than other approaches in dealing with Aa=AfX, because the dense–dense matrix multiplication (parallelization in step (iv) of MS) is easier to parallelize than the sparse–dense matrix multiplication (direct parallelization for Aa=AfX or parallelized CSR-based SDMM).

In this paper, for the current usage, we keep the possibility of parallelization open, because a serial MS has been efficient already.