Abstract

The need for government regulators and industrialized sources to determine the level of air pollution is essential. Air dispersion models are often used to determine the concentration of a pollutant. However changing conditions and several assumptions made by the models limit their accuracy at various times. This research was performed by combining four different air dispersion models (Gaussian Plume, Variable K Theory, Box, and AFTOX) into a superensemble. Since the superensemble is typically more accurate than its member models, the calculated result should be a more accurate prediction under any condition. One of the key parameters in the formation of the superensemble is whether the superensemble calculations for that range are to be fixed or continuous throughout. In the interest of evaluating performance, the change in accuracy for each member model and superensemble was measured through determining RMSE.

1. Introduction

One of the drawbacks of using any air dispersion model is the amount of error in the estimation of the pollutant concentration that gets produced. This error mostly arises from the inherent difficulty of using a relatively simple modeling framework to determine the concentration profile which is influenced by very complex and variable stimuli ¹. In the process of calculating the concentration of a pollutant there are a multitude of opportunities for error to become a factor. Some of the more prominent sources of error for a particular model include: the development of mathematical formula used by the model, the quality of the initial data parameters used in the calculation, the number of pollutant sources, and the complexity of the atmosphere ².

Another prominent source of error is commonly found during the gathering and measurement of air samples. The concentration data obtained from air samples obtained at the same testing site but during different time periods could vary widely. The sampled data could differ by 10 or even 100 times as much due to changes in the wind direction, atmospheric stability, emission level, reactivity of the pollutant, or the pollutant existing in two or more phases ³.

However, there are also many sources of error that end up being too costly or impractical to obtain either full or partial estimations. The error sources include: missing some or all of the data necessary for calculating the magnitude of imprecision from the measurements, having the available data be insufficient, or not knowing how exactly it is related to the uncertainty calculation ⁴. One example of this type of error may stem from the concentration of nitrogen. The highest being at the fuel source inside the smokestack. Since the fuel source typically is not fully heterogeneously mixed, the concentrations will often vary at various times ⁵. These nitrogen concentrations may also be influenced through the various manufacturing processes being run. Human error and unpredicted events are also sources of unknown error. The main corrective solution to minimize the error is to create adjustments in the model calculations.

2. Superensemble Algorithm

By calibrating the model against known concentrations, it is hoped that the error may be minimized. However, it is well known that some bias may be introduced within the process ⁵. A possible workaround is to use several models with the objective that some of the residual error will cancel itself out ⁶. The superensemble has the capability to provide significant error reduction through its design. This is accomplished by using a training phase where the forecasts of the various models can be analytically reconstituted through applying multiple linear regressions ⁷. This means that the error, bias, and skill are implicitly included and optimized into the superensemble forecast in the prediction phase.

Each air model has a weakness that hinders their accuracy under some condition or time of day. Thus, it is theorized that it may be beneficial to combine the various air dispersion models. This research study attempted to do this using a superensemble. The superensemble uses the first part of each model calculation and compares it to the actual conditions. This helps it to choose which model result to give the most weight in the calculation of the superensemble result ⁸. It is hoped that through reducing the influence of the less accurate models and giving more weight to the best models, the final prediction will be more accurate.

The superensemble was created due to the need for enhancement of the capabilities in ensemble mean forecasting ⁹. An ensemble mean is essentially just a straight average of all its member models. All ensemble mean members are given the same weight of one. The superensemble is an extension of bias-corrected ensemble mean forecasting which removes the bias of each member model. The superensemble assigns higher and lower weights to account for biases and better or worse model performances ¹⁰. The weights used are found through a multiple linear regression method which regresses the member model forecasts against the observed state. This creates a reward system which gives the better performing models a higher weight ¹¹. The only drawback to this method is that a model’s previous forecasts do not always predict how accurate the model will be in the current period especially if the data has changed ¹².

Because not all models are equally reliable at different points in space and time, it can be expected that the probability associated with the resulting deterministic forecast would not be the same for both the superensemble and the multi-model ensemble ¹³. The probability of a forecasted event from an ensemble system is based on the fraction of ensemble members predicting the event. After finding the probability using the superensemble model, a comparison of the two forecasts should be made ¹⁴.

The superensemble was first developed to improve the Atlantic tropical cyclone forecasts ¹⁵. It was designed to use previous forecasts from a pool of member models to correct the biases in their forecasts and rank each model’s relative strength. By using this method, the new tropical cyclone predictions were more accurate than any of the models in the pool ¹⁶. The superensemble method was next applied to climate forecasting and global precipitation forecasting ¹⁷. The Briar Skills Score was introduced to the superensemble process to assist in the process of increasing forecast accuracy ¹⁸.

Up to this point, the superensemble had been used to predict hurricanes in the Atlantic. Other applications began to emerge, starting with the adaptation for the prediction of the surface temperature in several US regions ¹⁹. The superensemble was also adapted to predict tropical storms and typhoons in regions such as the Pacific ²⁰, Korea ²¹, Southeast Asia ²², South America ²³, and China ²⁴. Other applications developed for the superensemble application include prediction of precipitation ^{25, 26}, and ²⁷; evaluating planetary boundary schemes ²⁸; determining the ocean surface temperature ²⁹; and modeling the diural cycle ³⁰ and ³¹.

³² first came up with the idea of using several of the available numerical weather prediction (NWP) models in an ensemble to predict air quality measurements. That was followed by the combination of both NWP and air quality models ³³. ³⁴ developed an ensemble of several forecasts which were input into a Lagrangian air dispersion model. The next breakthrough appeared with the idea to create a mean ensemble to predict the ozone concentration ^{35, 36}, and ³⁷. In 2007, ³⁸ developed Polyphemus, an ensemble consisting of two Gaussian and two Eulerian models. To date there has been no published research that combines the superensemble method with air dispersion modeling.

The superensemble is composed of two parts: a training phase and a forecast phase. The training phase entails the previous forecasts from the member models and the corresponding observed state ³⁹. The parameters consistent with all the models in the pool are regressed against the observed state using a linear regression technique. This linear regression technique is performed through a minimization function which acts to limit the spread between the parameters and the observed state ⁴⁰. The minimization function is given by this equation:

(1)

where J is the error term which needs to be minimized for obtaining the statistical weights, length is the length of the training period dataset, S(t) is the superensemble prediction, and O(t) is the observed state ⁴¹. The weights are then sent to the forecast phase where the superensemble forecast is created. The weights are based solely on each model’s performance in the training phase and can be negative if that model’s result is negatively correlated with the result of the other models in the superensemble ⁴² and ⁴³. The weights can also be modified in consideration of the regional variation of skills of each model. These varied weights make the forecast more accurate compared to those from the biased-removed ensemble forecasts ⁴⁶. The length of the training period is an important factor in obtaining accurate forecasts. Prior superensemble research has found that between 50 and 75 cases are needed to obtain an acceptable forecast ⁴⁷. One of the limitations of the training period is the changing of member models. The amount of data sets which can be used is limited by the last change of each model ⁴⁸.

The prediction phase of the superensemble is found through the following equation:

(2)

where is the mean of the observed value in the training phase, N is the number of models in the superensemble, is the regression coefficient or weight of each model, is the forecast made by each model I, and is the mean of all forecasts over the training period.

3. Methodology

The superensemble algorithm used in this research was developed through the renovation of the algorithm used in the Amorphe project ³⁸. Their superensemble was adapted from one which interfaced with rainfall precipitation models to one which could be applied to the air dispersion models discussed previously. Since the air dispersion models were run under steady state conditions, further alterations were performed to change the measurement variable for the training period. The effect of the alteration shifted the paradigm from one of elapsed time (days) to one of distance traveled (meters). This is how a distance based superensemble was created.

In this research four different air dispersion models (Gaussian Plume, Variable K Theory, Box, and AFTOX) were used individually and then were combined into a superensemble. The models were discussed in greater detail in ^{49, 50, 51, 52}, and ⁵³.

Two different sets of pollutant concentration profiles were used in the creation and evaluation of the superensemble algorithm. The first data set was used in the training phase as the input data needed for the superensemble creation ⁵⁴ and ⁵⁵. The remaining data set was used in the prediction phase for evaluation purposes. Unless otherwise specified, the intervals for both the superensemble training and prediction phases were set at 500m. This meant that over the course of the emitted distance several superensemble calculations were made. The data output from the superensemble will give the calculated pollutant concentration and the weights given to each model ⁵⁶. The accuracy of the superensemble and each model can then be obtained by comparing their results to the measured profile.

This research was applied to a case study which used common procedures. These procedures were uniformly performed and included superensemble formation and performance evaluation using both the Root Mean Square Error (RMSE) ^{57, 58}, and ⁵⁹ and the Brier Skill Score (BSS) ⁶⁰ and ⁶¹. ⁴⁷ and ⁶² provided the guidelines for applying these methods for air dispersion models.

4. Superensemble Formation

The creation of the superensemble was performed through the execution of a multi-step process. The first step was to run each member model using a given set of initial conditions unique for each data set. The concentration profile was calculated for each distance away from the point source found within the data set. The concentrations obtained by all the models in the pool were then regressed against the measurements of the data set using a linear regression technique ⁶³. This linear regression technique was performed through applying a non-linear minimization program written using LINGO 13.0 to solve the minimization function. This function acts to limit the spread between the parameters and the observed state and is commonly solved through both matrices and the Gauss-Jordan Elimination Method ⁶⁴. The minimization function used is similar to Equation 1 with the adaptation of using downwind distance as opposed to time elapsed due to steady state conditions and is given below:

(3)

where J is the error term that needs to be minimized to obtain the statistical weights, x is the length of the downwind distance away from the point source, S(x) is the superensemble prediction of the pollutant concentration, and O(x) is the observed value of the pollutant concentration. The need to find the necessary weights for each model made it necessary for Equation 3, the training phase equation, to be combined with Equation 4 the prediction phase equation. This new equation is given below.

(4)

In the prediction equation is the mean of the observed value in the training phase, N is the number of models in the superensemble, is the regression coefficient or weight of each model, is the forecast made by each model i, and is the mean of all forecasts over the training period. When combined these two equations form Equation 5.

(5)

The value of J was then minimized through the previously mentioned LINGO program designed to simulate the Gauss-Jordan elimination method others have successfully used ^{65, 66, 67}, and ⁶⁸.

Once the model weights were calculated for each model, the superensemble was then created. The next step was the completion of the training phase. After that, the prediction phase occurred. This meant that the superensemble and the member models were applied to predict the pollutant concentration values from a completely different data set. The member models and superensemble were run using the new initial conditions relevant to the new data set to obtain the predicted values ⁶⁹. Once the calculated values are obtained, the next logical step is to evaluate how well the member models and superensemble were able to perform. The purpose of this case study is to determine the effectiveness of the superensemble for emission prediction.

5. Baldwin Case Study

In this case study the Westvaco data set was used for the superensemble training phase and the Baldwin data set was used for the prediction phase ⁴⁷. The initial parameters required for the operation of each model were obtained from the initial conditions given in each data set. These initial values are given in Table 1. The concentration data that had been collected from both the Westvaco site and Baldwin site was organized into three separate files. These files contained the concentration measurements (WVA01, WVA03, and WVA24) and (BAL01, BAL03, and BAL24) originating from slightly different time periods. Concentration profiles of the three Westvaco measurement runs are given in Figure 1 and the three Baldwin measurement runs are given in Figure 2. The data from all three of these Westvaco measurement runs as well as their initial conditions was then averaged out to simulate a single collection obtained using steady state conditions ⁷⁰.

However, during the Baldwin measurement runs, the data from the BAL01 run was found to be significantly different from the other two runs. The best solution to this was to treat BAL01 as an outlier. This meant that there were only two measurement runs. Their initial conditions were averaged to simulate a single collection under steady state conditions.

Figure 3 displays the profile of the calculated concentrations from the models as well as the concentrations that were measured during the training phase. The models were calibrated during this period to minimize the difference with the measured concentration. Once the weights were obtained, the superensemble was then created and utilized along with the member models. The purpose of this was to predict the concentrations that occur at the Baldwin location.

The initial range for the superensemble calculation was arbitrarily set at a value of 500m. Because the total distance of the concentration profile was just under 2500m, five total superensemble calculations were necessary. Table 2 shows the calculated superensemble weights for that were given for each model over all five distance ranges. From the information shown in Table 2, it becomes apparent that the weights for each model vary significantly from range to range.

The next portion of the study began to explore the effects of the superensemble range. A superensemble was created from the calculated superensemble weights of Table 2. This superensemble will now be called a Fixed Superensemble. One of the observed drawbacks was that the weights for each model may vary at different distances. This means that the superensemble may not have the optimum weight values at any given point within the distance range. A second Superensemble was created to observe the effects. This new superensemble used a continuous range of 500m and calculated the model weights at every 1m point. This created superensemble will now be called a Continuous Superensemble. Figure 4 shows the calculated concentrations for each member model and both Fixed and Continuous superensembles for the Baldwin data set. Over the course of the entire concentration range, the graph gets very busy. Figure 5 shows a smaller snapshot of the calculated concentrations over an 100m interval.

A further analysis between the superensembles is undergone. Figure 6 shows the calculated concentrations for both Fixed and Continuous superensembles over the course of the entire concentration range. Figure 7 clearly demonstrates the large advantage of determining model weights at each point as it is magnified over an 50m interval. Being able to adjust higher or lower at every point makes such a huge difference in accuracy and precision.

6. Error Determination

Root Mean Square Error (RMSE) was used as the method to evaluate error. Figure 8 shows the calculated RMSE for the concentration profile for all member models and both superensembles. As one can see, the highest error generally occurred in the area around the peak. The one clear exception was the AFTOX model which significantly overpredicted concentrations after the peak.

The difference between the different superensembles really becomes apparent as shown in Figure 9. While the Fixed Superensemble is slightly better in some stretches of the concentration profile, the Continuous Superensemble is very accurate around the peak where it has very little error overall. This is due to it being able to constantly readjust relative to the smaller training period size. Thus, it can get closer to calculating the measured value. Most of the stretches where the Fixed Superensemble had a lower RMSE value were characterized by very low measured concentration values so the difference between the two was minimal overall.

7. Conclusion

One of the key limiting factors in the effectiveness of the superensemble is the reliance of using the member models to obtain the weights during the training phase. Because the models are often missing the peak on the concentration profile it makes the RMSE values higher there than at other points of the concentration profile. This also affects the superensemble due to their relationship. In many cases it can be indicated that the superensemble is limited by the performance of the best model. However, by using very low or negative weights on the worst models, slightly better performance is theoretically possible. It is advantageous to start out with very accurate and consistent models. But a perfectly predicting model would negate the need for a superensemble in the first place.

Where the superensemble gains its competitive edge is through the inconsistency of the models. An individual model tends to have regions of above average accuracy as well as regions with poor predictions. By having the superensemble continually near the best performing model, it constantly beats all the member models. This is due to the models losing their effectiveness in certain regions of the concentration profile. This is the reason why the range size is so important towards superensemble accuracy. Further study of the effects of decreasing the range size is planned as well as exploring the full tradeoff of implementing continuous superensemble calculations at each measurement point with the amount of calculation time and computing power that would be required.

Another point of contention is the method of evaluation for the models and the superensemble. One of the issues with using RMSE is the variability of the calculations as one travels away from the source. One model may have the lowest error at one distance and then have one of the highest values further downwind. Using the mean RMSE can give a gage of accuracy. However, there is the chance of applying the model or superensemble with the lowest mean RMSE and having the highest error at the property line.

Future research will look into incorporating the Briar score, which is dependent on the threshold concentration being exceeded to trigger an event. Thus, it would able to determine accuracy around the threshold concentration level. The Brier score operates on the assumption that the difference in the number of events will lead to a better prediction of accuracy. Other methods of error evaluation will need to be explored.

References