Exploring Deep Learning for Air Pollutant Emission Estimation

. The inaccuracy of anthropogenic emission inventory on a high-resolution scale due to insufficient basic data is one of the major reasons for the deviation between air quality model and observation results. A bottom-up approach, as a typical 15 emission inventory estimation approach, requires a lot of human labor and material resources, and a top-down approach focuses on individual pollutants that can be measured directly and relies heavily on traditional numerical modelling. Lately, deep neural network has achieved rapid development due to its high efficiency and non-linear expression ability. In this study, we proposed a novel method to model the dual relationship between emission inventory and pollution concentration for emission inventory estimation. Specifically, we utilized a neural network based comprehensive chemical transport model (NN-CTM) to learn the 20 complex correlation between emission and air pollution. We further updated the emission inventory based on backpropagating the gradient of the loss function measuring the deviation between NN-CTM and observations from surface monitors. We first mimicked the CTM model with neural networks (NN) and achieved a relatively good representation of CTM with similarity reaching 95%. To reduce the gap between CTM and observations, the NN model would suggest an updated emission of NO x , NH 3 , SO 2 , VOC and primary PM 2.5 which


Introduction
The clean air policies have been implemented by China government since 2010 which has been effectively reducing the pollutant concentrations such as sulfur dioxide (SO2), nitrogen oxides (NOx) (Zheng et al., 2018). Nevertheless, China still faces challenges in addressing O3 and PM2.5 pollutions. Particularly, the level of ozone (O3) in China has increased by 1.3% from 2013 to 2017 (Li, 2019); moreover, concentrations of PM2.5 (particulate matter with an aerodynamic diameter less than 35 2.5 μm) in most Chinese cities still far exceed the World Health Organization (WHO) recommended values (<10 μgm −3 ), leading to frequent heavy pollution events (Guo et al., 2014;Richter et al., 2005;Vesilind et al., 1988). Such high pollutant concentration may substantially affect human health given air pollution has being ranked fifth in global risk factors for mortality (Institute, 2019).
A prerequisite of effectively controlling air pollution lies in accurate knowledge of the related emission sources. A well-40 established emission inventory should summarize the amount of pollutants emitted into atmosphere from all sources in a specific region and time span (Institute, 2019). A typical bottom-up approach is adopted to develop the emission inventory through investigation of emission sources in Air Benefit and Cost and Attainment Assessment System Emission Inventory (ABaCAS-EI) (Zheng et al., 2019) and Multi-resolution Emission Inventory (MEIC) (He, 2012) developed by Tsinghua University, wherein the activity rate of each source is multiplied with emission factor (Vallero and Daniel, 2018). Such 45 technology-oriented bottom-up emission inventory can reflect the types of technology operated in China but has limitation in actual application because of its need for labor power and material resources, especially in cities where it is hard to support thorough investigation (Xing et al., 2020b). What's more, varied assumptions for activity rate and emission factor from different studies result in large uncertainties (Aardenne and Pulles, 2002). Therefore, the development of a method for efficient, low-cost, and sufficiently accurate grid-emission information is being considered. 50 The top-down method, as another typical emission inventory estimation approach, can be used to constrain emission estimation by combining observation results from surface monitors and satellite retrievals. Brioude et al. (2012) has estimated emissions of anthropogenic CO, NOx and CO2 in the Los Angeles Basin using the FLEXPART Lagrangian particle dispersion model based on the top-down method. Recently, Yang et al. (2021) linked the bottom-up MAPLE model with the top-down CGE model to evaluate deep decarbonization pathways' (DDP) comprehensive impacts in China. However, most of previous studies 55 merely focused on individual pollutants that can be measured directly (Brioude et al., 2012;Xing et al., 2020a;Yang et al., 2021) and relied on traditional numerical modelling.
On the contrary, neural networks (NN), as a more efficient tool, can also model complex nonlinear relations in the atmospheric system thus converting precursor emissions into ambient concentrations. Due to its ability of end-to-end learning, NN can automatically extract key features of input data and capture the behaviour of target data, thus has been widely used in 60 atmospheric science recently (Fan et al., 2017;Tao et al., 2019;Wen et al., 2019;Xing et al., 2020a;Xing et al., 2020c). For example, many studies (Fan et al., 2017;Tao et al., 2019;Wen et al., 2019) combined recurrent NN (RNN) and convolutional NN (CNN) to capture spatial and temporal features in air pollution related questions since RNN has a strong capability in mining temporal patterns from time series data Chung et al., 2014;Hochreiter and Schmidhuber, 1997) with certain ability to handle missing values efficiently (Fan et al., 2017) and CNN exhibits potentials in leveraging spatial 65 dependencies, e.g., in meteorological prediction (Krizhevsky et al., 2012). Furthermore, Xing et al. (2020d) applied NN to the surface response model (RSM), thus significantly enhancing the computational efficiency, demonstrating the utility of deep learning approaches for capturing the nonlinearity of atmospheric chemistry and physics. The application of deep learning improves the efficiency of air quality simulation and can quickly provide data support for the formulation of emission control policies, so as to adapt to the dynamic pollution situation and international situation. But the use of deep NN to estimate 70 emission inventory is more complex compared to traditional machine learning problems, because there is no precise emission observation that can be used as supervision for model training.
To address all these issues, we proposed a novel method based on dual learning (He et al., 2016), which leverages the primaldual structure of artificial intelligence (AI) tasks to obtain informative feedbacks and regularization signals, thus enhancing both the learning and inference process. In terms of emission inventory estimation, if we have a precise relationship from 75 emission inventory to pollution concentrations, we can use the pollution concentrations as a constraint to get accurate emission inventory. In particular, we proposed to employ a neural network based chemical transport model (NN-CTM) with a delicately designed architecture, which is efficient and differentiable compared to the chemical transport model (CTM). Furthermore, when a well-trained NN-CTM can accurately reflect the direct and indirect physical and chemical reactions between emission inventory and pollutant concentrations, the emission inventory can be updated by backpropagating the gradient of the error 80 between observed and NN-CTM predicted pollutant concentrations. Figure 1 shows the framework of this study.
The method used for this study is described in Section 2. Section 3 takes China emission inventory estimation as an example to demonstrate the superiority of our method. In section 4, we make a conclusion and discuss some possible future work.

Main Framework
The task of emission inventory estimation can be naturally formalized into a typical dual learning framework. Concretely, we 90 denote as the data of emission volumes and meteorological conditions and as the corresponding pollutant concentration at time t. In addition, we denote the mapping function from emission to pollutant concentration as f and that from pollutant concentration to emission as . Since the transformation from emission to pollutant concentration is a continuous process in time, approximately, we have the following equations: where [ : ] is defined as { , +1 , … , } for convenience, so is [ : ].
The formulas above are based on two assumptions:

1.
The pollutant concentration is only dependent on the emission and meteorological conditions in the past time steps, e.g., hours or days. 100

2.
There is a bijection relationship between emission and pollutant concentration. This is a necessary prerequisite for the existence of function .
The first assumption will hold true as long as setting a sufficiently large . The second assumption may not be true unless we introduce more external constraints on emission inventory, since there exists information loss in the process from emission to pollutant concentration. 105  The input data of our NN-CTM are similar to that of CTM, including emission inventory, meteorology and geographical data.
The first two are time-dependent data, while the last one, denoted as , is static data. In the Eulerian grid based CTM system, 140 for each time step , the dynamic input data is a matrix with dimension × ℎ × . The concentration is simulated continuously in a continuous time sequence. Unlike CTM, the NN-CTM cannot deal with too long data sequences. Thus, we just use the data from past time steps (i.e, [ − + 1: ]) as input for the pollutant concentration estimation . At the same time, we add − as supplementary input data into the network. Same as CTM, the output of NN-CTM is a matrix with dimension × ℎ × , where is the number of concerned pollutant species. 145 The NN-CTM consists of three branches: two CNN branches for − and , and one long short term memory (LSTM) (Hochreiter andSchmidhuber, 1997) with U-Net (Ronneberger et al., 2015) branch. The CNN branches are used to extract features for − and geographical information. We employ parametric rectified linear unit (P-RELU) (He et al., 2015b) as the non-linear activation function in these branches to improve model fitting with nearly zero extra computational cost and little overfitting risk. We adopt the architecture of combining LSTM and U-Net based on the understanding of temporal-spatial 150 relationship in emission inventory. In temporal dimension, pollutants are the accumulation of historical emissions. In spatial 删除的内容: sequence 删除的内容: a dimension, adjacent grids will affect each other because of meteorological and diffusion factors. The LSTM layer is used to aggregate information from time series data [ − + 1: ]. The aggregated sequence of hidden states h − +1 , … , h will be 155 concatenated and entered into U-Net block. U-Net is a widely adopted pixel-to-pixel model which can effectively utilize neighbour information. In U-Net, the stacked of convolution can get the neighbour information with a bigger receptive field (e.g. stacking 5×5 convolution and 5×5 convolution can get a 9×9 convolution), the non-linear function (P-RELU) is employed to improve model fitting with nearly zero extra computational cost and little overfitting risk, and the batch normalization and dropout are employed to enhance the robustness of the model. We calculate that the receptive field of our 160 model is 38×38 grid. In other words, the predicted pollutant concentration is related to its surrounding 38 ×38 grid's information, which represents the transmission between different grids. Meanwhile, the closer the distance, the greater the contribution. We employ 2-layers U-Net as shown in Figure 4 to capture the spatial information between grids. In the training process, we take ( , ) as training dataset, where is the input data of the CTM system while 170 is the corresponding output. Since relative changes in pollutant concentrations are the metric often used by policymakers, we adopt an objective function that measures the relative loss between NN-CTM predicted and CTM-simulated pollutant concentrations. We denote the output of NN-CTM as ̂, and have:

Emission Inventory Estimation Based On NN-CTM 185
Given a well-trained NN-CTM whose approximation accuracy is high enough for predicting pollutant concentrations, the emission inventory can be updated based on the error between observed and NN-CTM predicted pollutant concentrations. The observation data will help update the surrounding grids' emission inventory within the receptive field. However, in extreme circumstances, if we have no observation data, our method will not work as we have no more information to adjust the emission inventory. If the observation data is denser, the emission inventory estimation is more accurate as it can consider more 190 observation data.
In particular, we make the relationship between emission and pollutant concentration more robust by fixing the trained LSTM-U-Net model parameter. Then by training NN-CTM parameter, we adjust the input emission inventory to minimize the loss between NN-CTM output and observation. Such loss can be formally defined as: where * represents observed pollutant concentration (we use average value in case of multiple observation stations in a grid), , is a binary indicator variable indicating whether or not there is a site monitoring equipment in grid ( , ). The emission inventory will be updated by backpropagating the gradient . The stochastic gradient descent (SGD) method (Bottou, 2010) is used as optimizer. 200 Meanwhile, aiming at ensuring the reasonableness and effectiveness of estimated emission inventory, we set two constrains: 1) The update rate of emission inventory to be a maximum of 200% compared with the prior emission for each grid. There exist biases in meteorological conditions and chemical mechanism, which determines that we cannot attribute all the errors to the emission inventory. If the update ratio is very large, the NN-CTM cannot well reflect the correlation of the unseen data. 删除的内容: will Furthermore, the prior emission is accurate to a certain extent in terms of the spatial and temporal dimensions. 2)The updated emission inventory must be positive.

Experiments and Results Analysis
In this section, we apply our proposed method for emission inventory estimation in China in 2015. In the following, we will first describe the data and CTM configuration. After that, we will show experimental results in terms of the accuracy of NN-210 CTM. Then, we conduct further analysis on the prior emission inventory and our emission inventory estimation results.

Data and CTM Configuration
The prior emission inventory ABaCAS-EI with high spatial and temporal resolution is based on the bottom-up method, including primary pollutants such as NOx, ammonia (NH3), SO2, volatile organic compounds (VOC) and primary PM2.5.
ABaCAS-EI is a grid-unit-based emission inventory including sources of power, cement, steel industries, and mobile sources. 215 It also takes into consideration of technical progress and more stringent emission standards (Zheng et al., 2019). The prior emission inventory is initially used for NN-CTM training and then updated as per the proposed method of dual learning.
The simulation domain covers mainland China and portions of surrounding countries with a 27km × 27km horizontal resolution (with ℎ = 182 and = 232) and 14 vertical layers from ground to 100 hPa. Simulations are performed in January, 235 April, July, and October 2015 to represent winter, spring, summer, and autumn, respectively. A 5-day simulation spin-up was performed to minimize the effects of initial conditions. Pollutant concentrations are analysed at monthly averages.

NN-CTM Learning and Evaluation
Training parameters. The parameter of NN-CTM was optimized using Adam optimizer with a mini-batch size of 8. A learning 240 rate of 0.001 was used. To reduce the risk of over-fitting, we applied weight regularization on all trainable parameters during training and fine-tuning. The NN-CTM was trained for 30000 epochs.
Evaluation. We examined the performance of NN-CTM to check whether it has learnt the relationship between emission and pollutant concentration.
We trained NN-CTM on the data of first 22 days in January, April, July and October 2015 and tested it on the remaining successive 8 days of each month. As listed in Table 1, NN-CTM (with LSTM-U-Net) can well reproduce the spatial and 250 temporal relation with a small MAE of 0.27, 0.17, 1.39 ppbv and 1.46 μg m −3 for NO2, SO2, O3 and PM2.5, respectively, on average of four months. Results suggest that the NN-CTM can well reproduce the CTM within an acceptable bias, thus can be used for emission adjustment. Such bias (<4%) is much smaller than that of simulation compared to observations which are normally more than 10% even 20%.

Emission Inventory Updating and Analysis 265
A well trained NN-CTM is used to update the emission inventory through back propagation using stochastic gradient descent SGD (Bottou, 2010) optimizer with a mini-batch size of 2. The learning rate is 0.1. The optimization of emissions is achieved after 10000 epochs. For convenience, we denote the emission inventory from ABaCAS-EI as prior emissions (P-Emis) and the updated emission inventory as NN-emission (N-Emis), which is constrained by station observations. Compared with P-Emis, N-Emis has adjusted emission rates of NOx, NH3, SO2, VOC and primary PM2.5 as per the difference between simulated concentrations 275 and the observed values of pollutants in each grid, as shown in Figure 5. Average emission rates of NH3, SO2 and VOC in most grids tend to decrease, while that of primary PM2.5 tend to increase except for in the Yangtze River Basin, which may be related to the non-included dust emission. Changes in emission rate of NOx vary a lot by region, and such changes are concentrated in urban areas. The distribution of N-Emis for each grid is consistent with P-Emis, indicating that the deep 删除的内容: regions learning method in this study can identify the distribution of emission sources and focus on the calibration in high-emission areas.
Annual anthropogenic emissions in China for NOx, NH3, SO2, VOC and primary PM2.5 in P-Emis are 20.44, 10.39, 14.40, 23.05 and 7.19 Mt, respectively (Liu et al., 2020), while in N-Emis changed by -1.34%, -2.65%, -11.66%, -19.19% and 3.51%, respectively. 285 The sensitivity of change ratios to different seasons varies. Table 2 lists the change ratios of N-Emis compared to P-Emis in four months. As for N-Emis, NOx increases in January and October by about 3.5~4.0%, while it decreases by more than 10% 290 in July. Emission of NH3 increases in January while decreases in other three months with the highest decrease registered in October. Emission of SO2 tends to decrease in all four months with ratios around 10%. Emission of VOC also tends to decrease but with a larger magnitude of about 20% compared to SO2, which may be related to the overestimation of O3. Emission of primary PM2.5 tends to increase by less than 5% in four months.
Such changes in emissions are based on mathematical algorithms and thus cannot be explained by physical and chemical 295 processes. The NN method tries to give a solution to make simulation results of all pollutant species closer to observations by compensating the errors in emission inventory. For example, concentrations of PM2.5 obtained using P-Emis are generally lower than the observed level, so the emission of primary PM2.5 will be increased during the adjustment. SO2 tends to be overestimated using P-Emis, so the adjustment tends to decrease. However, because sulfate is an important component of PM2.5, the adjustment of SO2 will be restricted by the underestimation of PM2.5. Concentrations of O3 obtained using P-Emis 300 are generally higher than the observed level, so it tends to reduce the emissions of NOx and VOC, which are precursors of O3, during the adjustment. It is worth noting that the adjustment range of NOx is much lower than of VOC, because only the observed concentration of NO2 is used as a constraint. Such results are consistent with our previous study (Xing et al., 2020a).
In order to further analyse the change of emissions at a regional level, we calculated the four-month average emissions of P-Emis and change ratios of N-Emis for five emission species in the Beijing-Tianjin-Hebei region (BTH), the Yangtze River 305 Delta (YRD), the Pearl River Delta (PRD), the Sichuan Basin (SCH) and northwest China (NWC), as highlighted in Figure 6.
The first four areas were selected because they are the main population clusters, and NWC was selected because there are so few observation sites in this area that the constraints are relatively insufficient.

315
The adjustment of emission varies greatly by seasons and regions. Seasonal details are listed in Table 3. The four-month average changes of N-Emis in BTH are highest for SO2 and VOC emissions reaching about -20% while that of NOx, NH3 and primary PM2.5 vary by less than 5%. In YRD, NOx and VOC emissions record the highest extent of changes with -14.73% for NOx and 12.54% for VOC. The range of changes in other emission species is less than 5% (all decrease). Emission of primary PM2.5 in PRD increases by about 7%, which is the largest change ratio among four urban regions. Emission of NH3 in PRD 320 changes the least compared with other regions. In SCH, emissions of SO2 and VOC decrease the most (change ratio) compared with other emission species (>20%). Emission of primary PM2.5 in SCH, which decreases by 5.95%, shows an opposite trend to that in PRD. As for in NWC, emissions of NH3 and VOC show a small decrease (<5%), while emissions of NOx and primary PM2.5 have a large percentage increase compared with other regions (10%), thus particularly indicating the large inaccuracy in emission inventory in NWC. 325

Accuracy Improvements of CTM Simulation for pollutants with N-Emis
We use the CTM to evaluate the accuracy of P-Emis and N-Emis. The configuration of CTM keeps constant. 330 Generally, simulations using P-Emis tend to underestimate the PM2.5 concentrations and overestimate the O3 concentrations on average of four months in China, which are consistent with our previous studies Liu et al., 2019). The underestimation of PM2.5 using P-Emis usually appears in Northern and South eastern China, and sometimes occurs in some provinces of the Yangtze River Basin. The simulations of O3 using P-Emis are generally overestimated at observation sites.
Such errors can be narrowed when using N-Emis. We calculated the MAE for each simulation to compare the performances 335 considering all observation sites. After using adjusted emissions (i.e., N-Emis), the MAE for NO2, SO2, O3 and PM2.5 concentrations reduced significantly from 7.39 to 5.91 (20.03%), 3.64 to 3.22 (11.54%), 14.33 to 11.56 (19.33%) ppbv and 18.94 to 16.67 (11.99%) μg m −3 , respectively, average for total 612 observation stations, as shown in Figure 7. Such improvements prove the advantages of using N-Emis compared with P-Emis. Spatial distributions of comparison between simulations and observations in 612 sites can be found in Figure 8. The model performance of most stations has been improved, 340 and a small number of stations with worsen performance show the link between compound pollutants. For example, stations with larger deviations between PM2.5 simulation results and observations tend to have greatly improved O3 performance, and vice versa.

350
Difference of monthly simulations using N-emis & P-Emis as input can be used to estimate the seasonal impacts of emission changes. Concentrations of O3 and PM2.5 tend to increase in July while decrease in other months on average for China.
Concentrations of NO2 and SO2 tend to decrease in four months, which are consistent with the direct trend of emission adjustments.
We also calculated the average concentrations of four pollutants in five typical regions to quantify the degree of improvement 355 in pollutant concentrations after adjusting the emission inventory, as listed in Table 4. Changes in NO2 and SO2 concentrations are consistent with adjustments in emissions but are more sensitive, i.e. a small change (~10%) in emission results in a larger proportional change (~20%) in concentration. The reduced SO2 emission is an important reason for the improvement of PM2.5 overestimations in the Yangtze River Basin. PM2.5 concentrations in NWC show the highest increase (15%) compared with other regions. As the emission inventory in NWC has great potential for improvement (subject to production methods and the 360 acquisition of basic data), the qualitative changes in PM2.5 concentrations brought about using NN method seem meaningful.
The increase and decrease of NOx and VOC emissions directly control the variance in O3 concentration. Effect of using N-Emis on O3 concentration is not obvious, with a change range of less than 5% in typical regions. Although the adjustment ratio of emissions of O3 precursors is considerable, the O3 concentration doesn't change by much. The same can be linked to the complex relationship of precursor emissions of NOx and VOC which might not change simultaneously and in the same 365 direction (e.g. increase NOx and decrease in VOC or vice-versa) thus resulting in only a slight change in O3 concentration.

Conclusion and Discussion
In this study, we pioneer the use of machine learning to re-formulate the problem of emission inventory estimation. It creates a new perspective that the data-driven approach can be applied to automatically improve the quality of the emission inventory, 375 avoiding manual intervention and empirical error. We proposed a differential neural network based chemical transport model (NN-CTM), which achieves a relatively good representation of CTM. And then, we employed backpropagation algorithm to update the emission inventory based on the deviation between observed and NN-CTM predicted pollutant concentrations. In terms of method, we have proposed a novel emission inventory estimation method based on dual learning which consists of dual-loop of emission-to-pollution and pollution-to-emission. Results indicate that our NN based method with adjusted 380 emission inventory performed better than using prior emissions.
Compared with previous studies, our framework employs dual learning mechanism where in the simulated concentrations are compared to ground observation and the gradient is back propagated to update the emission inventory in each epoch. Results show that new emissions after the adjustment can improve the model performance in simulating the concentrations close to observations. The mean absolute error for NO2, SO2, O3 and PM2.5 concentrations reduced significantly by 10% to 20%. This 385 application uses a constant biogenic emission inventory, so the potential errors in biogenic emissions are also included in the learning of anthropogenic emissions.
Our method can be naturally extended to other fundamental problems, such as CO2 and other greenhouse gas emission inventory estimation, and has broad application prospects, such as building a real-time emission monitoring system based on real-time pollutant observation data. 390