Abstract

This paper presents the procedure of building a dynamic predictive model using an artificial neural network to perform an iterative forecast. An algorithm is proposed and named as “Artificial Neural Network Approach for Dynamic Iterative Forecasting”. The development of this algorithm focused on feature selection, identification of best network architecture for the model, moving window selection and finally the iterative prediction. This proposed algorithm was deployed to forecast next day’s hourly total demand in Sri Lanka as an illustration. Inclusion of a clustering effect that were based on the specialty of the day, as an input was investigated through this application, from which improved accuracies were shown.

1. Introduction

In the era of data science, prominence has been given to data driven research where static models become obsolete as new records are added to the database continuously over time. In such scenarios, most of the traditional statistical techniques become less capable to handle complex real world situations in a dynamic way. With the advancement of artificial intelligence approaches, these barriers can be overcome effectively.

In recent years, neural networks have gained much attention in the fields of pattern identification, classification and forecasting due to its capabilities of handling both linear and non-linear relationships efficiently. Thus, the use of neural networks in data driven research have gained much attention in the recent past.

2. Literature Review

When considering the literature regarding forecasting time series data, statistical models are static in nature so that a fitted model can only be used to forecast during a limited time frame ^{1, 2, 3, 4}. When the forecasts have to be performed for a longer time span, the model parameters should be updated to represent the more recent situation and hence static models become obsolete over time. In this regard, artificial intelligence approaches plays a key role in implementing dynamic models using the concept of moving windows to update the models with respect to most recent data ^{5, 6, 7}. In almost all of the published work regarding dynamic forecasting, neural networks play a main role solely or in combination with another technique as a hybrid model.

Guan and Luh (2005) have combined fuzzy logic, neural networks, and autoregressive models-for very short-term power system load forecasting due to the high accuracy levels ⁵. To forecast sunspot activates Xie et. al (2006) have used a hybrid neural network model, which integrated characteristics decomposition units, and a dynamic spline interpolation unit into the multiple adaptive time-delay neural networks ⁶. An accurate and timely forecasting of traffic flow model has been implemented by Jiang and Adeli (2005) using a novel nonparametric dynamic time-delay recurrent wavelet neural network ⁷.

When the forecast has to be performed for many steps ahead, the researches have recommended iterative predictions than multi-step ahead predictions with experimental evidence ^{8, 9}. Thus, this algorithm also proposes an iterative neural network model to increase the accuracy levels of forecasts which are several steps ahead.

3. Proposed Algorithm: Artificial Neural Network Approach for Dynamic Iterative Forecasting

When considering data driven research, many applications can be for forecasting future values. This paper proposes an algorithm that can be used to forecast time series data with many associative variables. The iterative forecast mechanism used in this algorithm enables the use of the recently forecasted values when forecasting the next immediate forecast, and is said to be superior to forecasting the series multi steps ahead.

4. An illustration of Artificial Neural Network Approach for Dynamic Iterative Forecasting

When considering power and energy industry, Short Term Load Forecasting (STLF) plays a critical role as the decision regarding the forecast should be taken within a very limited time span. Even though a vast range of research have been carried out on STLF, the challenging aspect is to increase the accuracy of the forecasts. Literature on STLF reveals that, using artificial intelligence based approaches for predictive models claimed to generate high accuracies compared to other conventional statistical techniques ^{10, 11, 12, 13, 14}.

Thus, to illustrate this algorithm, a dynamic model was implemented to forecast next day’s hourly electricity demand in Sri Lanka. The dataset consisted of hourly electricity demands form 01^st January 2008 to 31^st December 2012. Data in the first four years were used for training purposes and the data of the final year was used for testing. Figure 1 displays how the hourly electricity demands vary over the five years.

Prior to designing the network, a feature selection procedure was conducted to identify the most relevant inputs for the model, as inclusion of irrelevant inputs can distort the forecasting performances, increase the training time and increase network complexity ¹⁵.

A filter based approach, namely, mutual information based feature selection procedure was used to choose the lagged demand terms as the inputs to the neural network. Because of the strong daily and weekly patterns in the data series, a candidate feature set consisting of 24*7=168 variables were used to select the most appropriate features for the model. It is important to note that Rana et. al (2014) have justified a sliding window of one week is sufficient for this choice of variables, as it captures all seasonal patterns across a week ¹⁴. Figure 2 displays mutual information estimates sorted in descending order and the first 52 lagged terms were selected as the inputs to the neural network, due to no substantial changes with the mutual information estimates thereafter. The selected lagged variables were X_t-1 to X_t-14; X_t-16 to X_t-26; X_t-28 to X_t-31; X_t-35 to X_t-37; X_t-47 to X_t-49; X_t-71 to X_t-73; X_t-95 to X_t-97; Xt_-119 to X_t-121; X_t-143 to X_t-145; X_t-164 to X_t-168.

The exploratory analysis revealed that day of the week, month, specialty of the day and temperature plays a main role in STLF ^{16, 17} and were used along with the lag variables as inputs in the model. In addition, results of the hourly data clustering ^{16, 17} and curve clustering, were also considered as inputs to improve neural network performances.

Designing the neural network

Ÿ To implement a dynamic forecasting model, the neural network should be efficient, adjustable and less complex. Among the relevant literature, back propagation neural networks are widely used for STLF ^{15, 18, 19, 20, 21} Thus an error back propagated neural network was selected to implement the forecasting model.

Ÿ As the requirement was to forecast next day’s hourly demand dynamically, an iterative forecast was chosen as opposed to a multi-step ahead forecast. This choice was supported through literature in general, and has discussed extensively by ²².

Ÿ Given consistent data and enough neurons in its hidden layer, a two-layered feed-forward network with sigmoid hidden neurons and linear output neurons, fit multi-dimensional mapping problems arbitrarily well. The inputs as well as outputs were scaled in the range of [-1,1] ²³.

Ÿ Due to the less training time, the Levenberg-Marquardt back propagation algorithm was used to train the network. As the size of the required memory capacity is manageable the choice of this training algorithm can be justified.

Ÿ This neural network consisted of three layers as input layer, hidden layer and an output layer. Even though researchers argue that one hidden layer is capable of modelling well avoiding overfitting, for the completeness of the experiment, all the test cases were done considering two hidden layers with same number of neurons in each layer.

- The number of inputs in the input layer was varied with respect to different combinations of the input variables. The experiments are displayed in Table 1.

- Only one neuron was in the output layer as one step ahead prediction will be done.

- The number of neurons in the hidden layer was varied and the best combination was identified accordingly.

Selecting input variable combination and number of neurons in the hidden layer

Ÿ Sixteen separate neural networks were trained with different combinations of inputs.

Ÿ Each network was trained 50 times and the number of neurons in the hidden layer was chosen based on the minimum average MSE (Mean Squared Error).

- The median MSE was also considered for this selection as the average is subjected to be influenced by extreme points.

- For a particular network, when the minimum mean value and the minimum median value relates to the same number of neurons in the hidden layer, the choice was made on that number of neurons. When minimum average and minimum median acquired from two distinct sets of hidden neurons, both neuron sizes were observed and the one with the lower variations of the MSE was chosen.

Ÿ Each combination of inputs were considered for neural networks with 2 hidden layers too, but they generated worse errors. Thus it was decided to have only one hidden layer in the network.

A summary of results is displayed in Table 1. The categories of each input variable are listed below where the parenthesis gives the number of inputs under each category. Note that the categorical variables were used in the form of several binary variables.

Ÿ Lags (52) -> 52 variables with lagged observations

Ÿ Temp (1)-> Estimated hourly temperature measurement

Ÿ Day Type (7) -> Sunday, Monday, Tuesday, … , Saturday (7 indicator variables)

Ÿ Specialty (5) -> None, poya day, PBM (Public, Bank, Mercantile) holiday, one day after New Year, two days after New Year (5 indicator variables)

Ÿ Month (12) -> January, February, … , December (12 indicator variables)

Ÿ Cluster(6) -> Clusters identified based on hourly demand.

- Cluster 1 – Demand during early morning

- Cluster 2 – Demand during 5 a.m., 7 a.m., 8 a.m. and 11 a.m.

- Cluster 3 – Demand during working hours on Sundays (in 2008, Saturdays were added in this cluster based on the cluster analysis results)

- Cluster 4 – Demand during working hours on Monday – Saturday where specialty is none

- Cluster 5 – Demand during working hours on Monday – Saturday where specialty is holiday

- Cluster 6 – Demand during peak time

Ÿ CurveCluster(2) -> Clusters identified based the shape of the daily load curves (2 indicator variables)

- Cluster 1 – No specialty (This includes PB, working day after holiday, etc.)

- Cluster 2 – Holidays (PBM, poya day, Saturday after holiday)

Based on the experiments, two key findings were observed.

1. When exploring different input combinations as displayed in Table 1, with the inclusion of the cluster information as inputs, the Mean Squared Error of the validation set has reduced considerably.

2. With the inclusion of cluster information, the training times have also decreased.

It is therefore evident that, with the inclusion of clustering effect, the training process of the neural network becomes more efficient. Moreover, by using clusters as inputs, the network became less complex due to the reduction of dimensionality of a set of variables as inputs.

In conclusion, the minimum validation set error was obtained for the inputs, lags, cluster results based on hourly data, curve cluster results based on daily data, and temperature estimate. The results suggested that with the usage of clustering results, the performance of the neural network was increased.

Size of the moving window

Ÿ For the selected combination of inputs, the neural networks were trained by varying the moving window size. Figure 3 displays how the average MSE of the validation set varies with the size of the moving window. It can be clearly observed that, a moving window of 2 years resulted in the minimum average MSE when training the neural network. Thus, for the final prediction, the neural networks were trained using most recent data spanning 2 years. Considering the minimum MSE for the two year moving window period, 10 neurons were chosen for the hidden layer.

The average MAPE for the 2012 forecasted values was 3.96%. Some selected descriptive statistics of the MAPE for the 2012 are shown in Table 2. The RMSE of the final dynamic prediction is shown in Figure 4. It can be clearly observed that for a majority of days the RMSE of the forecasted values were considerably low, except for a few days. Except the New Year’s Day, all other days with very high errors were generated on days with no specialty.

It is learnt from Table 2, which 50% of the MAPE for the year 2012 were between 0.026 and 0.0368. Note that the prediction was made for an entire year to validate the suitability of this dynamic model for a longer time. The results have proven that the predictions are reasonably accurate even for this long period of one year. Thus, this model may be suitable even after a one year but, the applicability should be tested prior to using the implemented forecasting model.

In order to identify whether the observed errors can be grouped in some way, the RMSE values were clustered where the maximum Silhouette was obtained for 2 clusters. Figure 5 displays the distribution of the RMSE for the two clusters named as low error cluster (1) and high error cluster (2).

Table 3 shows the assignments with respect to the specialty of the day. It can be clearly seen that the majority of the PBM holidays and poya days were assigned to the group having low RMSE. Also it is to be noted that nearly 35% of the days having no specialty, have been assigned to the group with high errors.

Ÿ When exploring errors based on the day of the week, it was identified that Saturdays have high errors compared to other days of the week. It is anticipated that this may be as a result of the unclear assignment of Saturdays to a particular cluster when clustering hourly electricity demands ^{16, 17}.

Ÿ To address this issue, another neural network was trained with the same set of inputs, but replaced the six dummy variables with seven, to represent hourly demand clustering results. The seventh variable indicated whether a certain demand is during a working hour on a Saturday or not. As such, the working hours on weekdays were thus separated from the demand on Saturday working hours.

Ÿ When there is a holiday (PBM holiday or poya day) the demands during working hours were assigned to the cluster including working hours of holidays during weekdays.

After this adjustment for Saturdays, the process was repeated to dynamically forecast electricity demand for the year 2012. The model resulted in substantially low errors, for both Saturdays and weekdays when compared to the previous model. However, the new model, when compared to the previous neural network, was not able to improve accuracies for certain holidays. The new errors were compared with the previous forecasting errors as shown in Figure 6.

Ÿ It can be clearly seen that some of the high errors resulted from the previous model have been reduced with the new Saturday adjusted model.

Ÿ The new model is also incapable of accurately predicting the demands of one day after the New Year’s day.

Ÿ Out of the 366 days of the year 2012, 66% of the days resulted with low MAPE for the new model when compared to the previously model.

Ÿ The average MAPE for the year 2012 decreased from 3.969% to 3.522% (decreased by 11.26%) after addressing the effect of Saturday working hour demands.

Ÿ Thus the recalibrated model was selected to be used in the final forecasting process.

Figure 7 displays examples for accurately, moderately and poorly forecasted days using the Saturday adjusted model. For accurate forecasts, MAPE values were less than 2% in general. Based on the overall deviation for the entire day, the examples were classified into moderately or poorly forecasted days.

5. Synopsis

In this paper, an algorithm named “Artificial Neural Network Approach for Dynamic Iterative Forecasting” was formulated to iteratively forecast time series data using artificial neural networks in a dynamic way. As an illustration, hourly electricity demand in Sri Lanka were forecasted. It was identified that with the inclusion of clustering effects as inputs of the neural network, the training process became more efficient and accurate. It is important to emphasize that, a majority of holidays were well forecasted through this NN approach, except for a few specific holidays.

Acknowledgements

We acknowledge University of Colombo for the financial assistance provided for this research.

References