Enhancing deep learning for demand forecasting to address large data gaps

The COVID-19 pandemic, with its unprecedented challenges and disruptions, triggered a profound transformation in the retail industry. Health and safety regulations including periodic lockdowns, supply chain disruptions, and economic uncertainty affected the way businesses operate and led to a drastic change in consumer behaviour. This triggered the modification of traditional sales trends, which in turn impacted the accuracy of existing demand forecasting methods, and will affect their future performance. Moreover, the reliance of machine learning algorithms on historical sales data for training made them ill-equipped to adapt to these abrupt sales pattern shifts. Therefore, innovative solutions to address the complexities arising from this new landscape are needed. This paper introduces a framework aimed at enhancing demand forecasting accuracy in the post-pandemic period. Central to this framework is a feature engineering approach involving the creation of a predictor variable that encapsulates the level of restrictions imposed during the pandemic across seven distinct categories. This granular approach not only accounts for lockdowns and store closures but also considers indirect factors influencing retail sales, such as remote work arrangements and school closures. An extensive empirical evaluation of the proposed approach was conducted on a real-world retail dataset obtained from Charles Clinkard, a UK-based footwear retailer, demonstrating consistent improvements in forecasting accuracy across four deep probabilistic models and three levels of product aggregation in a real-life setting. Further validation utilising the Retail Sales Index dataset, which reflects monthly sales across various retail sectors in Great Britain, was also undertaken using six forecasting models, corroborating our initial findings. Overall, we show that leveraging historical sales data spanning the pandemic period – or, in general, any period where the data has inherent bias – is still viable for training machine learning models to forecast the demand, provided that an effective feature engineering approach is implemented.