Vector quantities, e.g., vector winds, play an extremely important role in climate systems. The energy and water exchanges between different regions are strongly dominated by wind, which in turn shapes the regional climate. Thus, how well climate models can simulate vector fields directly affects model performance in reproducing the nature of a regional climate. This paper devises a new diagram, termed the vector field evaluation (VFE) diagram, which is a generalized Taylor diagram and able to provide a concise evaluation of model performance in simulating vector fields. The diagram can measure how well two vector fields match each other in terms of three statistical variables, i.e., the vector similarity coefficient, root mean square length (RMSL), and root mean square vector difference (RMSVD). Similar to the Taylor diagram, the VFE diagram is especially useful for evaluating climate models. The pattern similarity of two vector fields is measured by a vector similarity coefficient (VSC) that is defined by the arithmetic mean of the inner product of normalized vector pairs. Examples are provided, showing that VSC can identify how close one vector field resembles another. Note that VSC can only describe the pattern similarity, and it does not reflect the systematic difference in the mean vector length between two vector fields. To measure the vector length, RMSL is included in the diagram. The third variable, RMSVD, is used to identify the magnitude of the overall difference between two vector fields. Examples show that the VFE diagram can clearly illustrate the extent to which the overall RMSVD is attributed to the systematic difference in RMSL and how much is due to the poor pattern similarity.

Vector quantities play a very important role in climate systems. It is well known that atmospheric circulation transfers mass, energy, and water vapor between different parts of the world, which is an extremely crucial factor for shaping regional climates. The monsoon climate is a typical example of one that is strongly dominated by atmospheric circulation. A strong Asian summer monsoon circulation usually brings more precipitation, and vice versa. Therefore, the simulated precipitation is strongly determined by how well climate models can simulate atmospheric circulation (Twardosz et al., 2011; Sperber et al., 2013; Zhou et al., 2016; Wei et al., 2016). Ocean surface wind stress is another important vector quantity that reflects the momentum flux between the ocean and atmosphere, serving as one of the major factors for oceanic circulation (Lee et al., 2012). The wind stress errors can cause large uncertainties in ocean circulation in the subtropical and subpolar regions (Chaudhuri et al., 2013). Thus, the evaluation of vector fields, e.g., vector winds and wind stress, would also help in understanding the causes of model errors.

The Taylor diagram (Taylor, 2001) is very useful in evaluating climate
models, and it has been widely used in model intercomparison and evaluation
studies over the past several years (e.g., Hellström and Chen, 2003;
Martin et al., 2011; Giorgi and Gutowski, 2015; Jiang et al., 2015; Katragkou
et al., 2015). However, the Taylor diagram was constructed for evaluating
scalar quantities, such as temperature and precipitation. The statistical
variables used in the Taylor diagram, i.e., the Pearson correlation
coefficient, standard deviation, and centered root mean square error (RMSE),
do not apply to vector quantities. No such diagram is yet available for
evaluating vector quantities such as vector winds, wind stress, temperature
gradients, and vorticity. Previous studies have usually assessed model
performance in reproducing a vector field by evaluating its

To construct the VFE diagram, one crucial issue is quantifying the pattern
similarity of two vector fields. Over the past several decades, many vector
correlation coefficients have been developed by different approaches. For
example, some vector correlation coefficients are constructed by combining
Pearson's correlation coefficient of the

To measure how well the patterns of two vector fields resemble each other, a vector similarity coefficient (VSC) is introduced in Sect. 2 and interpreted in Sect. 3. Section 4 constructs the VFE diagram with three statistical variables to evaluate multiple aspects of simulated vector fields. Section 5 illustrates the use of the diagram in evaluating climate model performance. Methods for indicating observational uncertainty are suggested in Sect. 6. A discussion and conclusion are provided in Sect. 7.

Schematic illustration of two vector sequences. Panel

Consider two vector fields

Each vector sequence is composed of

We define a normalized vector as follows:

With the aid of Eqs. (2) and (3), Eq. (1) can be rewritten as

For all

If

If

If

If

If

In this section, we present three cases to explain why VSC can reasonably
measure the pattern similarity of two vector fields. To facilitate the
interpretation, we define the mean difference of angles (MDA) between paired
vectors as follows:

Given Eq. (6) and the Cauchy–Schwarz inequality,

For all

For all

MSD measures how close the paired vector lengths of two normalized vector
fields are to each other. Based on the definition of

VSC can be written as follows:

To examine the relationship of VSC with MSD, we assume each corresponding
angle between paired vectors

Thus,

In the previous section, the interpretation of VSC is based on the assumption
that the paired vectors have a constant included angle. In this section, we
will examine how VSC is affected by the difference of included angles in a
more general case. Firstly, we construct a reference vector sequence,

Scatterplot between the vector similarity coefficient
(

In this section, we compute the

Climatological mean 850 hPa vector wind in

To measure the differences in two vector fields, a root mean square vector
difference (RMSVD) is defined following Shukla and Saha (1974) with a minor
modification:

The geometric relationship between RMSVD,

Geometric relationship among the vector similarity
coefficient

With the above definitions and relationships, we can construct a diagram
that statistically quantifies how close two vector fields are to each other
in terms of the

Diagram for displaying pattern statistics. The vector similarity coefficient between vector fields is given by the azimuthal position of the test field. The radial distance from the origin is proportional to the RMS length. The RMSVD between the test and reference field is proportional to their distance apart (dashed contours in the same units as the RMS length).

Summary of the difference between the Taylor diagram and the VFE diagram.

A common application of the VFE diagram is to compare multimodel
simulations against observations in terms of the patterns of vector fields.
As an example, we assess the pattern statistics of climatological mean
850 hPa vector winds derived from the historical experiments by 19 CMIP5
models (Taylor et al., 2012) compared with the NCEP-DOE Reanalysis 2 data
during the period from 1979 to 2005. The evaluation was based on the monthly
mean datasets from the first ensemble run of CMIP5 historical simulations
and all datasets were regridded to a common grid of 2.5

Normalized pattern statistics of climatological mean 850 hPa vector winds in the Asian–Australian monsoon region
(10

Climatological mean 850 hPa vector winds in summer and winter for the NCEP Reanalysis 2 data and the results of historical
simulations obtained from three CMIP5 models during the period from 1979 to 2005. The vector similarity coefficient (

To illustrate the performance of the VFE diagram in model evaluation, Fig. 7 shows the spatial patterns of the climatological mean 850 hPa vector winds
over the Asian–Australian monsoon region derived from the NCEP2 reanalysis
and three climate models. Models 1 and 4 show a spatial pattern of vector
winds that is very similar to the reanalysis data in summer, and

Similar to the Taylor diagram (Taylor, 2001), the VFE diagram can be applied to the following aspects.

To summarize the changes in the performance of a model, the points on the VFE diagram can be linked with arrows. For example, similar to Fig. 5 in Taylor (2001) the tails of the arrows represent the statistics for the older version, and the arrowheads point to the statistic for the newer version of the model. By doing so, the multiple statistical changes from the old version to the new version of the model can be clearly shown in the VFE diagram. The VFE diagram can also be combined with the Taylor diagram to show the statistics for both scalar and vector variables in one diagram by plotting double coordinates because both diagrams are constructed based on the law of cosine.

Normalized pattern statistics for climatological mean 850 hPa vector winds over the Asian–Australian monsoon region
(10

As presented in Taylor (2001), one can qualitatively assess whether or not there are apparent differences in model performance by comparing ensemble simulations obtained from different models. The performance of two models can have a significant difference if the statistics from two groups of ensemble simulations are clearly separated from each other, and vice versa. As an illustration of this point, Fig. 8 shows the normalized pattern statistics of the climatological mean 850 hPa vector winds derived from CMIP5 historical experiments during the period from 1979 to 2005. Models 12, 13, and 14 include 5, 6, and 9 ensemble runs, respectively. For a given model, all ensemble members of historical runs are forced in the same way, but each is initiated from a different point in the preindustrial control run (Taylor et al., 2012). Thus, the differences between ensemble runs from the same model result from the sampling variability. In contrast, the differences between ensemble runs from different models are caused by both the sampling variability and model formulation differences. In Fig. 8, the symbols representing the same model show a close clustering, signifying that the sampling variability has less impact on the statistics of climatological mean vector winds. On the other hand, the symbols representing different models are clearly separated from each other. This suggests that the differences between models are much larger than the random sampling variability of individual models. Thus, the differences between models 12, 13, and 14 are likely to be significant. Models 12 and 13 are different versions of the same model. Compared with model 12, model 13 shows a similar RMSL but higher VSCs and smaller RMSVDs, which suggests that the improvement of model 13 beyond 12 is primarily due to the improvement of the spatial pattern of vector winds (Fig. 8). The ensemble member involved here is less than 10 and the statistics between models 12 and 13 are separated from each other by only a small distance on the VFE diagram, which may not be sufficient to conclude a significant difference between models 12 and 13. This is a shortcoming of this method, i.e., lacking quantitative evaluation on the significance of difference in model performance, and warrants further study. Specifically, it may hard to determine the significance when the pattern statistics of two groups of simulations are not clearly separated from each other.

Normalized pattern statistics of climatological mean 850 hPa vector winds in the Asian–Australian monsoon region
(10

The VFE diagram provides a concise evaluation of model performance. However,
it should be noted that, for a given VSC at relatively low value, the RMSVD
does not decrease monotonically when the RMSL approaches the observed value (Fig. 5). Thus, a smaller RMSVD may not necessarily indicate a better model skill.
To measure model skill in simulating vector fields, we developed two skill
scores following the definition of model skill scores in
Taylor (2001):

Normalized pattern statistics of climatological mean 850 hPa vector winds in the Asian–Australian monsoon region
(10

It is known that observation data are uncertain due to many reasons, such as instrumental or sampling errors. Thus, it is necessary to evaluate the impact of observational uncertainties on the result of model evaluation. Taylor (2001) presented a good approach to measure the observational uncertainty by showing the statistics of models relative to various observations on the Taylor diagram. Such an approach can also be applied here to assess the impact of observational uncertainty on the evaluation of simulated vector fields. For example, we can compute the normalized pattern statistics describing the climatological mean 850 hPa vector winds derived from CMIP5 models compared with six reanalysis datasets, respectively (Fig. 9). We assumed six reanalysis datasets, i.e., NECP/NCAR Reanalysis 1 (Kalnay et al., 1996), NCEP-DOE Reanalysis 2 (Kanamitsu et al., 2002), ERA-40 (Uppala et al., 2005), ERA-Interim (Dee et al., 2011), JRA25 (Onogi et al., 2007), and JRA55 (Kobayashi et al., 2015; Harada et al., 2016) are observational data here. The modeled pattern statistics against various reanalysis datasets are similar to each other, indicating that the observational uncertainty in vector winds has a minor impact on the evaluation of simulated climatological mean 850 hPa vector winds.

Note that the pattern statistics are less discriminable in Fig. 9 due to the overlapping of many symbols, although we use different symbols and colors to distinguish them from each other. To make the pattern statistics more clear, we propose an alternative way to show the observational uncertainty by comparing each model and observation with the mean of multiple observational estimates. If we assume various observational estimates are obtained independently and contain random noises, these noises can contaminate the observational estimate. The random noises in various observational estimates could cancel out each other to a certain degree. Thus, the mean of multiple observational estimates may be closer to the true value than the individual observational estimate. We therefore take the ensemble mean of six reanalysis datasets as reference data and compute the pattern statistics of various models compared with the reference data to assess the model performance. Likewise, we can also measure the observational uncertainty by computing the pattern statistics of individual observational estimates relative to the reference data. The pattern statistics derived from models and individual observations can be shown on the VFE diagram with different symbols (Fig. 10). By doing so, one can roughly estimate the impacts of observational uncertainty on the evaluation of model performance. For example, the six reanalysis datasets show very close pattern statistics in summer characterized by high VSCs (0.986–0.994) and almost same RMSLs (0.986–1.021) as the reference data, which indicate a small observational uncertainty. Consequently, the observational uncertainty should have less impact on the evaluation of model performance. This is further supported by the comparison of Fig. 10 with Fig. 6. For example, the pattern statistics of CMIP5 models only show some minor changes when we replace the referenced NCEP-DOE Reanalysis 2 datasets with the ensemble mean of six reanalysis datasets (Figs. 6, 10).

In this study, we devised a vector field evaluation (VFE) diagram based on the geometric relationship between three scalar variables, i.e., the vector similarity coefficient (VSC), root mean square length (RMSL), and root mean square vector difference (RMSVD). The three statistical variables in the VFE diagram are meaningful and easy to compute. VSC is defined by the arithmetic mean of the inner product of normalized vector pairs to measure the pattern similarity between two vector fields. Our results suggest that VSC can describe the pattern similarity of two vector fields well. RMSL measures the mean and variance of vector lengths (Eq. A1). RMSVD measures the overall difference between two vector fields. The VFE diagram can clearly illustrate how much the overall RMSVD is attributed to the systematic difference in vector length vs. how much is due to poor pattern similarity.

As discussed in Appendix A, three statistical variables can be computed with full vector fields (including both the mean and anomaly) or vector anomaly fields. One can compute three statistical variables using full vector fields if the statistics in both the mean state and anomaly need to be evaluated (Figs. 6, 8). Alternatively, one can compute three statistical variables using vector anomaly fields if the statistics in the anomaly are the primary concern. Under certain circumstance, e.g., if the pattern of vector fields is highly homogeneous, the statistics of full vector fields could largely be dominated by the mean vector fields with a minor contribution from the anomaly fields (Eqs. A1–A4). Consequently, the statistics derived from different models may be very similar and difficult to separate from each other. In this case, one may want to assess the mean and anomaly fields, respectively. By doing so, the model performance in simulating vector anomaly fields can be better identified on the VFE diagram. The VFE diagram is devised to compare the statistics between two vector fields, e.g., vector winds usually comprise two- or three-dimensional vectors. One-dimensional vector fields can be regarded as scalar fields. In terms of the one-dimensional case, the VSC, RMSL, and RMSVD computed by anomaly fields become the correlation coefficient, standard deviation, and centered RMSE, respectively, and they are the statistical variables in the Taylor diagram. Thus, the Taylor diagram is a specific case of the VFE diagram. The Taylor diagram compares the statistics of scalar anomaly fields. The VFE diagram is a generalized Taylor diagram that can compare the statistics of full vector fields or vector anomaly fields. In practice, one may want to take latitudinal weight into account in the evaluation of spatial patterns of vector fields. This can be easily done by weighting the modeled and observed vector fields before computing VSC, RMSL, and RMSVD. Note that weighting should not be used during the computations of VSC, RMSL, and RMSVD to maintain their cosine relationship (Eq. 13). The VFE diagram can also be easily applied to the evaluation of three-dimensional vectors; however, we only considered two-dimensional vectors in this paper. If the vertical scale of a three-dimensional vector variable is much smaller than its horizontal scale, e.g., vector winds, one may consider multiplying the vertical component by 50 or 100 to accentuate its importance. In addition, as with the Taylor diagram, the VFE diagram can also be applied to track changes in model performance, indicate the significance of the differences in model performance, and evaluate model skills. More applications of the VFE diagram could be developed based on different research aims in the future.

The code used in the production of Figs. 2 and 6a is available in the Supplement to the article.

The standard deviation of the

Given

Similarly, we have

The VSC between vector fields

The

Based on Eqs. (A1), (A2), and (A4), we can conclude that the

The vector fields

Z. Xu and Z. Hou are the first coauthors. Z. Xu constructed the diagram and led the study. Z. Hou and Z. Xu performed the analysis. Z. Xu and Y. Han wrote the paper. All of the authors discussed the results and commented on the manuscript.

We acknowledge the World Climate Research Programme's Working Group on
Coupled Modelling, which is responsible for CMIP, and we thank the climate
modeling groups for producing and making their model output available.
NCEP/NCAR Reanalysis 1 and NCEP-DOE Reanalysis 2 datasets were provided by
the NOAA/OAR/ESRL PSD, Boulder, Colorado, USA, through their website at