This study proposes an ensemble deep learning approach that integrates Bagging Ridge (BR) regression with Bi-directional Long Short-Term Memory (Bi-LSTM) neural networks used as base regressors to become a Bi-LSTM BR approach. Bi-LSTM BR was used to predict the exchange rates of 21 currencies against the USD during the pre-COVID-19 and COVID-19 periods. To demonstrate the effectiveness of our proposed model, we compared the prediction performance with several more traditional machine learning algorithms, such as the regression tree, support vector regression, and random forest regression, and deep learning-based algorithms such as LSTM and Bi-LSTM. Our proposed ensemble deep learning approach outperformed the compared models in forecasting exchange rates in terms of prediction error. However, the performance of the model significantly varied during non-COVID-19 and COVID-19 periods across currencies, indicating the essential role of prediction models in periods of highly volatile foreign currency markets. By providing an improved prediction performance and identifying the most seriously affected currencies, this study is beneficial for foreign exchange traders and other stakeholders in that it offers opportunities for potential trading profitability and for reducing the impact of increased currency risk during the pandemic.