Goal
To use explainability methods to compare two popular machine learning approaches in the forecasting of near-term electricity demand.
Preamble
The ultimate aim of forecasting electricity usage is to provide just enough energy to supply demand, which limits resource wastage and therefore costs, as well as reducing unnecessary environmental impacts. Those producers who have not been able to respond quickly to changing demand profiles have tended to be driven out of the market.
The literature is replete with studies investigating various methods to predict near-term electricity demand. Very few of them, however, make effective use of model explainability techniques, which would not only facilitate a better understanding of how the models work, but also allow for more rapid improvements in model creation, while minimizing the associated maintenance costs.
This project focused on improving electricity demand forecasting models using model explainability techniques like SHAP and Model Approximation. Two advanced models, eXtreme Gradient Boosting (XGBoost) and Long Short-Term Memory (LSTM) networks, were explored - mainly because they were the most relied-upon models cited in the recent literature.
Note: This project was initially completed as a group Masters project, but ended up forming the basis to a commercial project I worked on.
Data
The data was selected from multiple sources, based on guidance derived from other projects. All data related to New South Wales - Australia, as this was the focus of the project. The data included:
Total Electricity Demand data for NSW from the Market Management System (MMS) database
Air temperature data from the Australian Data Archive for Meteorology (ADAM)
Price and demand data from the Australian Energy Market Operator (AEMO)
Location-specific weather data from the Time and Date website
Solar Panel Installations data from the Clean Energy Regulator website (Australian Government)
Solar exposure, minimum and maximum temperature datasets from the Australian Bureau of Meteorology (BoM)
Public holidays from the Australian Government website
Although much more data was available, it was decided to use 3 years of data, with the first two years acting as the training data set, and the final year as the testing data set.
Method
The data was first federated, cleaned and standardized.
XGBoost:
A number of lagged variables at various lag distances were generated, and a model was constructed to predict the next-day electricity demand. Using GridSearchCV the hyperparameters were optimized and the top-performing model discovered.
LSTM:
A base model was initially built, after which a series of steps were performed to gradually improve it. These included varying model complexity by increasing the number of hidden layers, utilizing activation functions such as tanh and relu, adding dropout layers and modifying the optimizer to enhance model convergence and performance. Using Bayesian Optimization, the hyperparameters were optimized and the top-performing model discovered.
Results
Both models were able to effectively forecast electricity demand but differed in the data features they relied on. They both primarily relied on past electricity demand, but other factors like calendar effects, weather conditions, solar radiation, and solar panel output also played roles. The importance of these features varied between models. For XGBoost, priority features included average temperature, solar radiation, day, month, and retail price, while for LSTM, calendar effects and solar panel output were important factors.
SHAP analysis was used on the XGBoost model to derive these feature importances. However, since SHAP doesn’t work directly on LSTM models, an approximation approach was needed. In this case a separate Random Forest Regressor model was developed to predict the electricity demand based on the result of the LSTM model. This random forest model was then put through SHAP analysis as a kind of reverse engineering approach, and although it was likely to miss some of the sophistication of the LSTM model, it was thought to still give a near-enough approximation of the importance each of the features contributed to the model.
As a follow-up to this project, it is anticipated that further refining of the models can be achieved by further leveraging the power of model explainability, especially since the accuracy of both models was already high.