Virtual Integration of Satellite and In-situ Observation Networks (VISION) v1.0: In-Situ Observations Simulator (ISO_simulator)

This work presents the first step in the development of the VISION toolkit, a set of Python tools that allows easy, efficient, and more meaningful comparison between global atmospheric models and observational data. Whilst observational data and modelling capabilities are expanding in parallel, there are still barriers preventing these two data sources from being used in synergy. This arises from differences in spatial and temporal sampling between models and observational platforms: observational data from a research aircraft, for example, are sampled on specified flight trajectories at very high temporal resolution. Proper comparison with model data requires generating, storing, and handling a large number of highly temporally resolved model files, resulting in a process which is data-, labour-, and time-intensive. In this paper we focus on comparison between model data and in situ observations (from aircraft, ships, buoys, sondes, etc.). A standalone code, In-Situ Observations Simulator, or ISO_simulator for short, is described here: this software reads modelled variables and observational data files and outputs model data interpolated in space and time to match observations. These model data are then written to NetCDF files that can be efficiently archived due to their small sizes and directly compared to observations. This method achieves a large reduction in the size of model data being produced for comparison with flight and other in situ data. By interpolating global gridded hourly files onto observation locations, we reduce data output for a typical climate resolution run, from ∼3 Gb per model variable per month to ∼15 Mb per model variable per month (a 200-times reduction in data volume). The VISION toolkit is relatively fast to run and can be automated to process large volumes of data at once, allowing efficient data analysis over a large number of years. Although this code was initially tested within the Unified Model (UM) framework, which is shared by the UK Earth System Model (UKESM), it was written as a flexible tool and it can be extended to work with other models.

1 Introduction

The importance of atmospheric observations from both in situ and remote sensing platforms has been growing in the last few decades, with data archives, such as the new Natural Environment Research Council (NERC) Environmental Data Service (EDS; https://eds.ukri.org, last access: 8 January 2025), becoming a key infrastructure for the storage, exchange, and exploitation of data. The strategic importance of in situ measurements was also highlighted by the recent GBP 49 million NERC funding to maintain and re-equip the BAe-146 research aircraft of the FAAM Airborne Laboratory out to 2040 (https://www.faam.ac.uk/mid-life-upgrade/, last access: 8 January 2025).

Advances in geophysical model developments and exascale computing have similarly led to an increase in the complexity of models used for climate projections in international modelling projects, such as CMIP6; chemistry and aerosol components are now routinely being included in a number of climate model simulations (Stevenson et al., 2020; Thornhill et al., 2021; Griffiths et al., 2021). Comparing all these models with observations is vital to increase our confidence in their ability to reproduce historical observations, to understand existing biases, and ultimately to improve their representation of the atmosphere.

A wide variety of observational datasets can be used for model evaluation; what makes such comparisons with model data inherently difficult is the difference between the orderly model data, defined on the model grid at regular time intervals, and the unstructured observational data, with variable coverage in space and time. A large computational effort is required for the handling and processing of gridded model data files into a format suitable for direct comparison with observations, especially when the measurement location varies with time (e.g. aircraft, ships, sondes). In order to compare this to observational data with varying coordinates, model output must include hourly (or higher-frequency) variables over a large atmospheric domain. As well as being data-intensive, extracting hourly data from a tape archive is also time-intensive. This leads to orders of magnitude more data being stored and processed than is actually required, and a significant number of labour and computer resources are spent to extract, read, and interpolate model data in space and time onto desired observation coordinates. Because of these issues, previous studies of comparison between models and in situ observations from aircraft are generally restricted to case studies over a limited number of campaigns (e.g. Kim et al., 2015; Anderson et al., 2021), or they compare model data with observed data independently of the time or location of measurement (e.g. Wang et al., 2020).

A previous attempt at producing Unified Model data on flight tracks was made several years ago, by embedding a flight_track routine (using Fortran programming language) within the UM-UKCA source code (Telford et al., 2013). However, using this approach added a computational burden to the running of the UM-UKCA model, and it was mainly intended for the output of chemical fields (model diagnostics related to some UM dynamical fields were not available within the UM-UKCA subroutines). As a result, the flight_track routine used in Telford et al. (2013) was never ported to further versions of UM-UKCA.

In this paper we describe the first tool in the VISION toolkit: ISO_simulator.py. This code can be embedded into the model workflow, or it can optionally be used as a standalone code with existing model data (e.g. to process variables from existing simulations). When ISO_simulator is embedded in the model workflow, it produces much smaller data files which can easily be archived and are ready to be used for direct comparison with observations.

This new tool allows the routine production of model data interpolated at the time and location of in situ observational data. This can enable the exploitation of large observational datasets, potentially spanning decades, to be used for large-scale model evaluation. Another possible application of the VISION toolkit is for improving model comparison with observations when conducting Observing System Simulation Experiments (OSSEs) (Zeng et al., 2020). These experiments are typically performed using models with a high spatial and time resolution; integrating the VISION tools into the workflow of such high-resolution nature runs (NRs) would allow us to efficiently sample data at the model time step with much-reduced data storage requirements. Whilst the UM/UKESM has been used as a test model and the current version of the tool is designed to work with UM output, the next version of the VISION toolkit (currently under development) will be model-independent. Since VISION is designed to work with CF-compliant data, including CMIP CMORised output, it could provide a valuable tool for supporting expanded diagnostics in upcoming CMIP7 experiments.

In Sect. 2 we describe the ISO_simulator code, including command line arguments, input–output files, and code optimisation. In Sect. 3 we show how ISO_simulator is embedded within the Unified Model workflow. In Sect. 4 we provide some example plots showing comparisons of UM–UKESM-modelled ozone to measurements from the Cape Verde Atmospheric Observatory (CVAO) (Carpenter et al., 2024), ozone measurements from the Tropospheric Ozone Assessment Report (TOAR) ocean surface database (including data from cruise ships and buoys, similar to Lelieveld et al., 2004, and Kanaya et al., 2019), and ozone measurements from the FAAM Airborne Laboratory (Smith et al., 2024) and the NASA DC-8 Atmospheric Tomography (ATom) mission (Thompson et al., 2022).

2 Description of ISO_simulator.py

In order to run ISO_simulator.py v1.0, the user will need access to Python 3.8 or higher, including CIS v1.7.4 (Watson-Parris et al., 2016), cf-python v3.13.0 (Hassell and Bartholomew, 2020), and Iris v3.1.0 (Hattersley et al., 2023) APIs. Iris libraries are used in some CIS functions to read gridded model data.

ISO_simulator.py performs the following steps:

1. Reading time and coordinates from observational files using CIS Python libraries,

2. Reading all model variables from hourly files using cf-python libraries,

3. Co-locating model variables in space and time to the same time/location as the observations using CIS Python libraries,

4. Writing monthly NetCDF files (Rew et al., 1989) containing model variables co-located onto flight tracks.

2.1 Input arguments

ISO_simulator.py requires a number of command line arguments which are shown in Table 1. The current version of ISO_simulator was developed for use within the UM modelling framework; therefore some of the current command line arguments are UM-specific. However, when interfacing ISO_simulator to different models, these command line arguments can be changed to reflect output data that are specific to each model.

A subparser argument, “jobtype”, is used to indicate whether the code is running within a model runtime workflow (if “batch” is selected) or as a standalone postprocessing tool, e.g. on existing model data (if “postprocessing” is selected). These subparser arguments also unlock specific conditional arguments: –archive_hourly can be used only if “batch” is selected, and –select_stash can only be used if “postprocessing” is selected. By default, when running in batch mode, all fields present in the output file being processed will be co-located to the observational locations.

Table 1 Description of command line arguments used to run ISO_simulator.py.

2.2 Required input files

Model input files can be supplied in NetCDF, UM pp, and UM fieldsfile formats and must have a date tag in the filename (YYYYMMDD) to identify the date in the file. The ability to read different formats of model input files gives extra flexibility to the code, as it allows one to read both other model data and UM data.

The interpolation code can use either air_pressure or altitude as the vertical coordinate for interpolation. If this is not specified, it will use altitude by default. When using air_pressure as a vertical coordinate, model variables are output on selected pressure levels. Since the UM has a terrain-following, hybrid-height vertical coordinate system, we additionally need to output a Heaviside function that accounts for missing model data where a pressure level near the surface falls below the surface height for that grid box. Where data are valid, the Heaviside function has a value of 1, and it has a value of 0 otherwise. By dividing the model field on pressure levels by this Heaviside function, the model data are correctly masked and missing data are assigned to invalid grid points.

When using altitude as a vertical coordinate, because of the UM hybrid height coordinates, model variables are defined at specified heights above the model surface; therefore, the model orography field has to be provided to correctly convert the model hybrid height to altitude above sea level. The name and path of the orography file can be defined using the -e command line argument.

For observational data which are defined at the surface (ground measurements or ship/buoy data), a vertical coordinate is generally not provided. In this case, ISO_simulator will use the model lowest level and interpolate in time, latitude, and longitude only.

Along with model files, input files containing information on the observational data coordinates are also required. These input files should be in NetCDF format (Rew et al., 1989), all data should be organised into daily files, and each file must have a date tag in the filename (YYYYMMDD) to identify the date of the measurement. Along with time and positional coordinates, an optional string variable can be added to the observational input files to identify data belonging to a specific dataset or campaign; this can help during analysis to subset-relevant data, which is useful when comparing to several datasets/campaigns over a number of decades. Existing observational data might require a degree of pre-processing to ensure files are in the right format to be used by ISO_simulator; the extent and type of processing will vary depending on the format and structure of each observational dataset.

2.3 Output files

The model data, co-located to the observational coordinates, are generated in NetCDF-format files. ISO_simulator produces one file per model variable per month. The size of these output files depends on the number and size of the observational files on which the model data are co-located and can therefore vary each month.

2.4 Code optimisation

There are several Python libraries that can deal with reading and writing of large gridded data files. The choice to use CIS Python libraries in ISO_simulator.py stems from their ability to handle ungridded data (such as data from ships and aircraft) and the ease of performing co-location from gridded to ungridded data. Initial timing tests using the VISION toolkit identified reading the model data as the single most time-consuming step compared to reading the observational data, extracting the values along a trajectory, and writing the output. Therefore, the time required to read model data using different Python libraries was investigated. However, these tests showed that reading UM model input files using CIS was significantly slower than reading the same file with Iris or cf-python; Table 2 shows reading times for loading files in UM fieldsfile, UM pp, and NetCDF format using different libraries. Note that, unlike the current release of ISO_simulator, which uses older versions of CIS, Iris, and cf-python, the tests in Table 2 were performed using the latest versions of CIS (v1.7.9), Iris (v3.10.0), and cf-python (v3.16.2).

Table 2 Comparison of file-reading times using CIS library (version: 1.7.9), Iris library (version: 3.10.0), and cf-python library (version: 3.16.2). The “structured UM loading” (https://scitools-iris.readthedocs.io/en/stable/generated/api/iris.fileformats.um.html#iris.fileformats.um.structured_um_loading, last access: 8 January 2025) method in Iris is a context manager which enables an alternative loading mechanism for “structured” UM files, providing much faster load times. The times in the table include reading the file and accessing the data NumPy array (via a simple print statement) to avoid lazy loading. The numbers in the table are averages, plus or minus the standard error, for reading each file 10 times on a local cluster (2×32 core Intel Xeon Gold 6338 2.00 GHz and 384 GB of memory).

* Note that we were unable to read 36 fields from the UM pp file using CIS; therefore the numbers in parentheses show the time required to read only 9 fields from the file. Additionally, CIS does not support the reading of UM fieldsfiles.

Given that potentially many such large files would need to be read in each model month, cf-python was chosen to read the model data. In practical tests, when run over a large number of years, ISO_simulator takes around 2–3 min to process one variable for 1 model year. However, CIS and cf-python use very different data structures for the gridded variables they read. In order to overcome this problem, a Python function was developed to convert the cf-python gridded data structure to the CIS gridded data structure.

Since reading model data is the slowest step in ISO_simulator, we further optimised the code by only reading model output files for days for which an observational input file exists.

3 Embedding ISO_simulator within the UM model workflow

This section describes how our code is interfaced within the UM framework. The UM uses Rose configuration editor (Shin et al., 2018) and the Cylc workflow engine (Oliver et al., 2018), respectively, as a graphical user interface (GUI) and to control the model simulation workflow. Rose is a system for creating, editing, and running application configurations, and it is used as the GUI for the UM to configure input namelists. Cylc is a workflow engine that is used to schedule the various tasks needed to run an instance of the UM in the correct sequence: for example, atmos_main runs the main UM code, postproc deals with data formatting and archiving, and housekeeping deletes unnecessary files from the user workspace.

A new Rose application, VISION_iso, was created and inserted into the Cylc workflow between the model integration step (atmos_main) and the postproc step (see Fig. 1). This new application includes an input namelist and calls ISO_simulator.py; the NetCDF output files, containing model data co-located to the observations, are then sent to the MASS tape archive during the postproc step.

Figure 1 A sketch of the UM runtime workflow, showing where the VISION toolkit is included.

Since this software can be embedded into the UM runtime workflow and operates on UM output files (rather than being part of the UM source code), it has the following advantages compared to the approach in Telford et al. (2013):

1. Model data interpolated to the measurement times and locations is output using the internationally recognised NetCDF format, thus providing any required metadata information and making handling and analysis quicker and easier for users.

2. The code runs in parallel to the atmosphere model and does not affect the model run time.

3. The code can easily be customised to process any model data (not just UM data), therefore making it useful to the wider atmospheric science community.

Model data interpolated to the measurement times and locations can then be archived for long-term storage. When embedded into the UM workflow, data can either be transferred to the MASS tape archiving system or to the JASMIN data analysis facility (Lawrence et al., 2012). Further savings in data storage can be made by optionally deleting the hourly model output files used by the VISION toolkit.

Figure 2 Probability distribution of ozone concentration at CVAO between 2007 and 2022. The black line shows observed ozone, and the red line is model data co-located using ISO_simulator.

Figure 3 Map plot difference between UKESM ozone and observed ozone for the period January 1986 to May 2022. Model data are co-located using ISO_simulator.

4 Test cases

Model simulations of UKESM were performed with a horizontal grid of 1.875° × 1.25° and 85 vertical levels with a model top at 85 km, and ERA reanalysis data were used to constrain the model meteorology (Telford et al., 2008) to allow better comparison with observations. For more information on the model configuration and details of the simulations, the reader is referred to the model description in Russo et al. (2023) and Archibald et al. (2024).

The aim of the plots in this section is not to answer specific science questions but to illustrate the way ISO_simulator can be used to co-locate model data onto different sets of in situ observations, namely ground-based stations, ships/buoys, sondes/flights, and unmanned aerial vehicles (UAVs).

All model data are output hourly from UKESM at a horizontal resolution of ∼150 km and then co-located, using ISO_simulator, to the same time and geographical coordinates as the observational data; the resulting data have the same time and spatial resolution as the observational data, making model data directly comparable to observational data. Furthermore, since the model and observational datasets can be compared over a long period of time, spanning several years, it is possible to sample seasonal and interannual variability, with better statistical sampling of extreme values. This type of comparison can greatly help to identify and improve model biases and to use models and observations in synergy to better understand atmospheric processes.

4.1 Cape Verde Atmospheric Observatory

The Cape Verde Atmospheric Observatory (CVAO) provides long-term ground-based observations in the tropical North Atlantic Ocean region (16°51^′49^′′ N, 24°52^′02^′′ W). The CVAO is a Global Atmospheric Watch (GAW) station of the World Meteorological Organisation (WMO); measurements from CVAO are available in the UK Centre for Environmental Data Analysis (CEDA) data archives (http://catalogue.ceda.ac.uk/uuid/81693aad69409100b1b9a247b9ae75d5, last access: 8 January 2025; Carpenter et al., 2024). The University of York provides the CVAO trace gas measurements, supported by the Natural Environment Research Council (NERC) through the National Centre for Atmospheric Science (NCAS) Atmospheric Measurement and Observation Facility (AMOF). Data from CVAO were chosen as an example of surface station data because the ozone measurements are provided at a higher temporal resolution than the hourly model output; ISO_simulator can therefore be useful to interpolate model data in time to match the time of the observations.

Figure 4 FAAM and UKESM ozone concentrations (a, b); UKESM carbon monoxide and temperature (c, d). UKESM data are co-located in space and time to match the data collected during all FAAM flights in August 2019.

Figure 5 Difference between UKESM and FAAM ozone concentrations (ppbv) in the lower troposphere (0–6 km) for all FAAM flights which measured ozone between 2010 and 2020.

Figure 6 ATom and UKESM ozone concentrations (ppbv) for the ATom flight on 3 February 2017 from Hawaii to Fiji.

4.2 Ships and buoys dataset

The Tropospheric Ozone Assessment Report (TOAR; https://igacproject.org/activities/TOAR/TOAR-II, last access: 8 January 2025) is an international activity under the International Global Atmospheric Chemistry project. It aims to assess the global distribution and trends of tropospheric ozone and to provide data that are useful for the analysis of ozone impacts on health, vegetation, and climate. A novel dataset has been produced by the TOAR “Ozone over the Oceans” working group. This dataset is an extension of previous similar datasets (Lelieveld et al., 2004; Kanaya et al., 2019), and it combines ship and buoy data from the 1970s to the present day. This dataset will be released later this year as part of the TOAR-II assessment; given the large temporal span, this dataset constitutes a great example of using ISO_simulator to compare model and observational data over a large number of years.

UKESM data were co-located to the same times and locations as observations. The plot in Fig. 3 shows the difference between modelled and observed ozone.

4.3 Aircraft data: comparison to FAAM and ATom

The FAAM Airborne Laboratory is a state-of-the-art research facility dedicated to the advancement of atmospheric science.

It operates a specially adapted BAe 146-301 research aircraft and is based at Cranfield University. The FAAM Airborne Laboratory is funded by the Natural Environment Research Council and managed through the National Centre for Atmospheric Science (NCAS).

FAAM data from 2010 to 2020 were processed to extract ozone, time, latitude, longitude, air pressure, and altitude and to ensure variable names were consistent throughout this time period (Russo et al., 2024). Figures 4 and 5 show comparisons of modelled data and ozone observed by the FAAM aircraft. Figure 4 shows FAAM ozone from all individual flights from a specific campaign occurring in August 2019 and a number of model variables interpolated on the FAAM flight tracks (ozone, carbon monoxide, and temperature). Figure 5 shows the difference between modelled and observed ozone for all flight points between the surface and ∼6 km and for all FAAM flights between 2010 and 2020.

The NASA Atmospheric Tomography (ATom) mission was a global-scale airborne campaign, funded through the NASA Earth Venture Suborbital-2 (EVS-2) programme, to study the impact of human-produced air pollution on greenhouse gases and on chemically reactive gases in the atmosphere. ATom utilised the fully instrumented NASA DC-8 research aircraft to measure a wide range of chemical and meteorological parameters in the remote troposphere (Thompson et al., 2022). Data from the ATom mission are available in the NASA data archive (https://doi.org/10.3334/ORNLDAAC/1925, Wofsy et al., 2021).

Figure 6 shows ozone concentrations from ATom and UKESM, as a function of time and altitude, for a specific flight on 3 February 2017.

5 Conclusions

The ability to sample atmospheric model output at the same time and location as in situ observations allows better synergy between model and observational data, resulting in better understanding of atmospheric processes and more effective model evaluation. However, doing this usually requires the processing of large volumes of high-frequency gridded model data. By interfacing with the CIS and cf-python libraries, we are able to efficiently automate this step, greatly reducing manual post-processing time and the volume of data that needs to be saved following a model simulation. This method is also transferable to many different atmospheric models, and the code is provided on GitHub under an open-source licence.

The use of the cf-python library to read-in the UM-format files significantly decreases the time taken to read these files when compared to the Iris or CIS libraries. This further reduces the time required for the HPC batch system to post-process the files from the global model to the times and locations of the in situ observations. An extension to this work is currently being carried out to be able to output model data on satellite swaths for better comparison between atmospheric models and satellite data.