Six stations with long-term Doppler lidar wind observations were selected from around the globe (see Fig. ) for comparison with the operational IFS model winds. These stations have different surface characteristics; these were marine, rural, urban, and coastal urban. Four of the stations were operated by the Atmospheric Radiation Measurement programme ARM; of the U.S. Department of Energy, one by the Finnish Meteorological Institute (FMI), and one by the Andalusian Institute for Earth System Research (IISTA-CEAMA). The six stations selected were as follows:

SGP, the Southern Great Plains, United States, operated by ARM. The station is located in the central United States, near the border of Oklahoma and Kansas. The area is flat and rural, consisting mainly of cattle pasture and wheat fields. The SGP atmospheric observatory is the world's largest climate research facility with an extensive history of atmospheric measurements.

Measurement periods and nominal data coverage for each station are listed in Table .

The observed wind profiles were obtained from Halo Photonics Streamline and Streamline XR Doppler lidars, which are commercially available heterodyne pulsed systems capable of full hemispheric scanning (instrument specifications given in Table ). These instruments have a nominal maximum range of 9 km or more and were operated at temporal resolutions of 1–5 s. Data from the first 60–90 m in range suffer with contamination from the outgoing pulse and are discarded. Scan schedules for each Doppler lidar comprised sequences of conical scans at constant elevation (velocity azimuth display, VAD) interspersed with extended periods of vertical stare and other scan types. For this study, we used the vertical profiles of horizontal winds derived from the VAD scans, which were at elevation angles from the horizontal of 60, 70, or 75∘, depending on the station.

The standard ARM Doppler lidar scan schedule provides one VAD scan at 60∘ in elevation from the horizontal every 15 min, with each VAD scan comprising eight beams equally spaced in azimuth and each beam integrating 30 000 pulses. For Granada, the Doppler lidar scan schedule provides one VAD scan at 75∘ in elevation from the horizontal every 10 min, with each VAD scan comprising six (later 12) beams equally spaced in azimuth with each beam integrating 30 000 (later 45 000) pulses. For Kumpula, the scan schedule provides one VAD scan at 70∘ elevation from the horizontal every 15 min, with each VAD scan comprising 24 beams equally spaced in azimuth with each beam integrating 45 000 pulses.

For the ARM stations, we used the ARM Doppler lidar wind value-added-product DLPROFWIND4NEWS,, and for Granada and Kumpula, the winds were calculated using the method of .

We use the Doppler-lidar-retrieved wind profiles to evaluate wind forecasts generated by the European Centre for Medium-Range Weather Forecasts (ECMWF) from their operational Integrated Forecast System (IFS). The IFS is a global numerical weather prediction (NWP) system, which assimilates a wide range of observations and generates a range of forecast products. The forecast products include medium-range to seasonal predictions and both deterministic and ensemble forecasts. In this study we only consider the high-resolution deterministic medium-range forecast (referred to as HRES), which currently has a horizontal resolution of approximately 9 km and 137 levels in the vertical. The vertical grid spacing is non-uniform and below 15 km varies from 20 to 300 m, with higher resolution closer to the ground. There are about 20 model levels in the lowest 1 km, with the vertical resolution ranging from 20 to 100 m. Forecasts up to 10 d in length are initialised every 12 h, and the temporal resolution of the model output is 1 h.

The assimilation system ingests wind observations from surface-based instrumentation over land (10 m observations from synoptic stations) and sea (ships and buoys), aircraft, radiosondes, radar wind profilers, weather radars, and satellites, with wind speed and direction observations being transformed into wind components (u and v). Satellite observations include ocean scatterometer data, processed using dedicated observation operators and assumed to be equivalent to winds at 10 m above the surface . Atmospheric motion vectors (AMVs) derived from satellite-observed cloud advection are assimilated as single-level wind observations default operation, .

It should be noted that the IFS is under constant development, with a new version typically becoming operational every 6–12 months. We do not aim to investigate how changes to the IFS affect the wind forecasts, but there have been some potentially significant model updates during the evaluation period; at the end of June 2013, the number of vertical levels was increased from 91 to 137 (from 12 to 20 model levels below 1 km), and, in March 2016, the horizontal grid was changed from a cubic-reduced Gaussian grid to an octahedral-reduced Gaussian grid, resulting in an increase in horizontal resolution from 16 to 9 km. A full description of the IFS, observations that are assimilated, and the model updates over time can be found in ECMWF documentation: https://www.ecmwf.int/en/publications/ifs-documentation.

For this study we were provided with the model winds for the nearest model grid point to the station from day-ahead forecasts, which were initialised at 12:00 UTC the previous day and correspond to forecast hours t+12 to t+35.

Except for very close to surface, the IFS has a coarser vertical resolution than the Doppler lidar. Therefore, prior to the evaluation, the IFS winds were interpolated to match the vertical resolution of the Doppler lidar. To ensure that any differences observed between the data sets were due to the performance of the IFS only, the Doppler lidar data were filtered rather strictly to ensure that only data with low uncertainty were included. First, a signal-to-noise ratio (SNR) threshold of -20 dB was applied . Second, we applied a column-wise threshold filter, which identified the first point with SNR below the -20 dB threshold, and then removed all data above. Note that these two steps could be combined into one filter, but separating the steps gave the opportunity to identify where there was no longer continuity in the wind profile. The effect of the column-wise threshold filter was comparatively small when applied after the SNR filter.

Third, all data points having a wind fit residual greater than 1 m s-1 were discarded. The value of the residual is a good indicator of turbulence, and as the wind retrieval assumes horizontal homogeneity , wind retrievals with a high residual value cannot be considered reliable.

In some cases, precipitation can adversely affect the Doppler lidar wind retrievals, for example, when the precipitation intensity is not spatially homogeneous. While most of these cases were captured by the filters presented above, there were a few precipitation periods which were not adequately filtered. These were identified using a simple rain filter, which detected whether there were downdrafts exceeding 2 m s-1 in the lowest 200 m of the Doppler lidar vertical velocity profile. If so, the entire profile was discarded. This is obviously not an accurate method for identifying the presence of precipitation but was sufficient for detecting cases where precipitation was likely to compromise the wind retrieval and had not been removed by the other filters.

The order of the filters only matters for the column-wise threshold filter, which must be applied after the SNR filter.

While the filtering of strong turbulence and precipitation is necessary to achieve consistently high data quality, it also removes the impact of these regimes from the results. Therefore, while we are able to evaluate the model in low-turbulent, non-precipitating cases, it is important to remember that we are not able to evaluate the IFS performance in strongly turbulent or precipitating situations.

Figure displays an example of the impact of applying these filters in terms of a diurnal composite, showing the average fraction of data that would be removed by each filter at Granada. The residual filter and SNR filter each remove the largest fraction of data, with the patterns mostly resembling each other. This is because the increasing uncertainty in radial winds at low SNR would also result in a large residual after attempting the wind retrieval. There are some differences between these two filters; the SNR filter for this instrument and location becomes stronger more rapidly above 1 km than the residual filter, but the residual filter is more effective during the daytime boundary layer at altitudes below 1 km. As the residual filter produces similar results to the SNR filter, with the additional benefit of identifying retrievals suffering from strong turbulence, some wind retrieval methods use a residual filter instead of a SNR threshold . Here we use both filters in tandem in order to ensure good data quality. The precipitation filter only removes relatively few profiles, as was intended, but these are the profiles that are expected to have the largest errors. The column-wise threshold filter, which is applied after the SNR filter, mostly removes some data points above the boundary layer and generally removes an extra 3 %–6 % of data in this region. Figure a–d show the fraction of data at the nominal scan temporal resolution removed by each filter.

After filtering, the Doppler lidar wind profiles (resolution 10 to 15 min) were averaged to match the 1 h temporal resolution of the wind profiles from IFS. Averaging was performed separately for the u and v components, from which wind speed and direction were then calculated. Figure e shows the resulting impact of all filters in terms of the 1 h average being compared to IFS.

The fraction of filtered data remaining varies from station to station and is dependent on, among others, the dynamics of the boundary layer, aerosol concentration, and the specific Doppler lidar configuration . The dynamics of the boundary layer determine both the altitude to which sufficient aerosol is lofted and also the variation in the strength of turbulence. The SNR is determined by the aerosol concentration and the Doppler lidar configuration.

The four ARM stations launch radiosondes operationally, which provided an ideal opportunity to validate the Doppler lidar wind retrievals. To compare the winds from radiosonde and Doppler lidar, we performed the filtering of the Doppler lidar as described in Sect.  and then linearly interpolated the high-resolution radiosonde wind profile (about 10 m vertical resolution) to match the vertical resolution of the Doppler lidar wind profile (26 m at these locations). We chose four altitudes for the comparison: 117, 247, 507, and 1000 m above ground level (a.g.l.).

Figure shows that, for all locations, there are slightly larger differences between the two instruments at the lowest altitude selected (117 m a.g.l.) than above. This was expected as the radiosonde launch may still be impacting the balloon track, as the balloon is still accelerating and swinging, and any difference between Doppler lidar location and radiosonde launch location would have the largest impact near the surface. The distance from Doppler lidar to radiosonde launch location was only 40 m at two stations, Cape Cod and Graciosa, increasing to 120 m at Darwin and 250 m at SGP. The observed winds agree well everywhere except for close to the surface at Darwin. At 117 m a.g.l. in Darwin, the radiosonde reports consistently higher wind speeds than the Doppler lidar. This is unlikely to be due to distance only, as SGP still compares well, even though the distance between the Doppler lidar and radiosonde launch location is further. At Darwin, radiosonde winds were obtained by radar tracking rather than GPS during the period involved in this study, and it is thought that a mismatch between the stated and actual launch location is partly responsible for the discrepancy close to the surface.

Except for 117 m a.g.l. in Darwin, the mean absolute difference in wind speed is <1 m s-1. The mean absolute difference in wind direction was <10∘ for every site except Darwin. We attribute this to the large proportion of low wind speeds observed at Darwin; Fig.  shows that if low wind speeds <3 m s-1 were ignored, the agreement would also be much better at Darwin. Thus, we are confident in the filtered winds retrieved from Doppler lidar.

Kumpula and Granada lack co-located operational radio soundings, so it was not possible to perform the validation of Doppler lidar winds in the same manner. However, the Doppler lidars at these stations were of similar type and configuration; therefore we are also confident in the Doppler lidar winds retrieved from these stations.

For model evaluation, we calculated four different metrics: wind speed bias B (observation minus model forecast), mean absolute wind speed error MAES, mean absolute wind direction error MAED, and mean absolute wind vector error MAEV. The possible ranges for each of the metrics are listed in Table . Equations for the metrics are listed in Appendix . All metrics are calculated with respect to the ground level at the Doppler lidar location. Note that the observation and model surface altitude above mean sea level may not agree in locations with locally varying topography. For the calculation of MAED, data with wind speed less than 2 m s-1 were ignored, as the wind direction uncertainty increases rapidly as the wind speed tends towards 0 m s-1 e.g..

Two stations, SGP and Graciosa, had sufficiently long observation time series to investigate the seasonal variability of the absolute wind vector error. Figure shows a time series of the 30 d moving averages of the absolute wind vector error over SGP at different altitudes. The mean is consistently higher than the median, which is a result of the absolute wind vector errors following a Weibull distribution, but otherwise the mean and median show a similar pattern.

A seasonal cycle for the absolute wind vector error can be seen, particularly at the lower three altitudes selected, with larger errors seen during summer and lower errors seen during winter. However, there is significant variation from year to year. There is no clear signal in absolute wind vector error that could be attributed to the upgrade in the resolution of the IFS.

Figure a shows the monthly MAEV from SGP, which clarifies the seasonal cycle suggested by Fig. . Again, the winter–summer contrast is obvious in the three lowest altitudes selected. At 1 km a.g.l., the monthly MAEV follows the same pattern from April to September but displays elevated MAEV in winter, in contrast to the lower altitudes.

The time series of the absolute wind vector error from Graciosa (see Fig. ) also displays a seasonal cycle, although weaker than at SGP. This cycle can be seen in Fig. b, where the monthly MAEV is largest during winter and lower during summer at all altitudes selected.

The vertical and temporal resolution of Doppler lidar wind profiles enable the generation of diurnal composites of MAEV and B for each station, as shown in Figs.  and . These metrics were only calculated for time–height intervals containing at least 50 valid data points. The composites are displayed in approximate local solar time (see Table ).

The degradation of the 12:00 Z t+12 to t+35 wind forecast over time can be seen in MAEV at almost all sites, with the improvement seen for the new forecast coming at 00:00 Z (approximate local solar time: Graciosa, 23; Kumpula, 2; Cape Cod, 19; SGP, 17; Darwin, 9; Granada, 0). The forecast degradation with time is also visible in MAES and MAEV (see Figs.  and ), but notably, no degradation in terms of B with respect to the forecast time is seen.

The marine and coastal locations, Graciosa, Kumpula, and Cape Cod, display the lowest values of MAEV at all altitudes, with values generally <3 m s-1. For these locations, the impact of the surface on the winds is relatively small, and satellite scatterometer observations over the ocean or sea basin have been assimilated.

Graciosa shows little diurnal variation in B; for 90 % of the composite, -0.3>B>0.2 m s-1. There is more diurnal influence in B for Kumpula, with negative B extending gradually above the surface during the daytime, indicative of the influence of a growing boundary layer. Kumpula also displays a negative bias of about -1 m s-1 close to the surface, which we attribute to the impact of the urban surface not being captured adequately within the model grid box. Cape Cod also shows some diurnal variation; at low altitudes (200–400 m a.g.l.), there is a variation in B, with positive values during the night. As seen for Kumpula, there are negative values of B during the day extending from the surface upwards, indicating the influence of a growing boundary layer. Kumpula and Cape Cod are coastal locations and so can experience both shallow marine boundary layers and the deeper boundary layers generated over land.

SGP shows a distinct diurnal pattern where MAEV<3 m s-1 within the daytime turbulent boundary layer and can exceed 4 m s-1 during the night. MAEV>4 m s-1 is also seen above the boundary layer. The magnitude of B is very small above and within the upper daytime turbulent boundary layer, even when MAEV is large. The lowest 300 m of the daytime boundary layer shows a positive bias reaching 0.7 m s-1. A large positive bias above 1 m s-1 is seen in the nocturnal boundary layer, which has a maximum at altitudes around 300 m a.g.l. and has values exceeding 1.5 m s-1. The nocturnal pattern of larger MAEV and B suggests the presence of a low-level jet (LLJ), which is explored in more detail in Sect. .

Darwin exhibits a weak diurnal pattern in MAEV, with lower values during the morning and the largest values at night. However, there is a distinct feature in B that is present throughout the day – a layer of B <-1 m s-1 in the lowest 150 m a.g.l. with another layer of B >0.5 m s-1 above this. This bias is likely due to the differences in the local surface experienced by the two measurement types and the surface representation in the model.

For Granada, MAEV>3 m s-1 almost everywhere and can exceed 4 m s-1 above 1 km. Low values of MAEV are only seen in the morning boundary layer, where the wind speeds themselves are also generally low (median wind speed around 2 m s-1; see Fig. ). If IFS also produces low wind speeds, then MAEV will also be small, even if the wind direction is not correct. Hence, these low wind speeds explain the magnitude of B also being relatively low close to the surface. Above the surface, B becomes more and more negative with increasing altitude, reaching -1.5 m s-1 above 1 km a.g.l. During the afternoon, B is positive close to the surface, 0.5 m s-1, but still decreases with altitude, with B becoming negative above 1 km. This station also shows the challenges of wind verification in weak wind conditions, where the wind direction is much more difficult to predict (see Figs.  and ).

In Granada, the presence of the Sierra Nevada mountains and generally low wind speeds create conditions suitable for the frequent development of nocturnal mountain breezes katabatic winds; . Above the Doppler lidar station in Granada, the mountain breeze can be observed as a shallow layer of weak easterly and southeasterly winds near the surface with the wind direction matching that of the valleys. There are several valleys leading from the mountains to the city and the surrounding areas, and the resulting topography (see Fig. ) is too complex to be resolved by the IFS model. The smoothed IFS orography would not be expected to capture mountain breezes arriving from valley directions that are not resolved. However, it is important to know how large the local model forecast errors may be. Even at low wind speeds, katabatic flows can still impact issues such as the formation of fog and air quality .

To understand the impact of the mountain breeze on errors in the IFS wind profiles, we separated the evaluation into two classes based on the Doppler lidar wind profiles: cases with a mountain breeze and cases without. To identify mountain breeze cases in the Doppler lidar winds, the mountain breeze must satisfy four conditions: the mountain breeze must be present between 2 h before sunset and 2 h after sunrise, although it may persist beyond this time; the wind direction should be between 55 and 145∘ close to the surface; the mountain breeze should be at least 120 m deep; the mountain breeze should have a duration of at least 1 h.

Figure  presents the diurnal composites of MAED for when the mountain breeze is present and when it is not present. At night, MAED ranges from 60 to 85∘ near the surface when the mountain breeze is present and from 45 to 70∘ when the mountain breeze is not present. The daytime MAED is around 40∘. While MAED is larger when the mountain breeze is present, the complex topography and the generally low wind speeds make forecasting the wind direction in Granada difficult under any circumstances.

IFS forecasts the most common nocturnal wind direction near the surface to be from the south, which matches surface observations at Armilla Airbase, located 4 km southwest of the Doppler lidar station . However, in the city of Granada, where the Doppler lidar is located, south is the least prevalent wind direction near the surface.

Low-level jets occur frequently at SGP . Reliable LLJ prediction is highly relevant for the wind energy sector in terms of energy production and, consequently, profits . Additionally, nocturnal precipitation from thunderstorms generated by elevated convection initiated by LLJs is common in the Great Plains region .

To examine the ability of IFS to capture LLJs at SGP, we separated the evaluation into two different classes based on whether an LLJ was present in the Doppler lidar observations. LLJ detection was performed using the method described by , which identifies continuous local wind speed maxima in the vertical wind speed profile being both at least 2 m s-1 stronger and at least 25 % stronger than local wind speed minima above and below. Composites of wind speed bias for LLJ cases and non-LLJ cases are presented in Fig. . LLJ cases exhibit a bias of more than 2 m s-1 up to about 500 m during the night, in contrast to non-LLJ cases, where the bias rarely exceeds 1 m s-1.

Composites of mean wind speeds for the Doppler lidar and IFS during LLJ cases are shown in Fig. . While IFS can clearly produce LLJs at approximately the correct height, the mean wind speed is roughly 2 m s-1 lower than observed by the Doppler lidar. This is likely due to the effective vertical resolution of IFS smoothing the wind speed profile and reducing both the peak wind speed value and the amount of shear below the jet compared to what is seen in the observations .

Almost all of the largest absolute wind vector errors between IFS and Doppler lidar were due to incorrectly timed frontal passages and storms. While the timing of these fronts and storms may have been out by up to 4 h, IFS was usually able to produce structures that closely resembled the Doppler lidar observations. Therefore, although the absolute wind vector error could be greater than 20 m s-1, the IFS may still be considered to be performing rather well. We are limited here to observations from a single Doppler lidar, and we can only detect displacement errors as a temporal error. Given a network of such observations with suitable spatial coverage, it might be possible to evaluate displacement errors in the forecast.

An example of large momentary forecast error due to incorrect timing is shown in Fig. . The u and v components of IFS and Doppler lidar winds show similar features but with IFS displaying an approximately 4 h delay. Whether this incorrect timing is significant or not is dependent on the intended use of the forecasts .