This chapter introduces recurrent neural networks (RNNs) — the architecture behind most sequence and time-series modeling — then applies them to two problems: generating Bach-style chorales and forecasting streamflow from a hydrological time series.
Sequential data, recurrent neurons, vanishing/exploding gradients, and where RNNs and NLP show up in environmental science.
Memory cells, LSTM and GRU gating, training RNNs through time, and the attention mechanism and transformer architecture.
Training a WaveNet-style dilated convolutional model, and recurrent architectures, to generate new Bach chorales one note at a time.
Forecasting streamflow from temperature and precipitation records with an LSTM, evaluated using the Nash-Sutcliffe efficiency coefficient.
Resources¶
Textbooks and papers this chapter’s exercises adapt¶
Hands-On Machine Learning with Scikit-Learn and PyTorch — Aurélien Géron; chapter 13 (processing sequences using RNNs and CNNs) is the direct source of the Bach chorale/WaveNet exercise. (7.2, 7.3)
Mishra, S., Bordin, C., Taharaguchi, K., & Palu, I. “Comparison of deep learning models for multivariate prediction of time series wind power generation and temperature.” Energy Reports 6 (2020): 273-286 — the source of the RNN/LSTM schematic figure. (7.2)
Rassem, A., El-Beltagy, M., & Saleh, M. “Cross-country skiing gears classification using deep learning.” arXiv preprint arXiv:1706.08924 (2017). (7.2)
Yang, Y., et al. “A study on water quality prediction by a hybrid CNN-LSTM model with attention mechanism.” Environmental Science and Pollution Research 28.39 (2021): 55129-55139. (7.2)
Alerskans, E., et al. “A transformer neural network for predicting near-surface temperature.” Meteorological Applications 29.5 (2022): e2098. (7.2)
PyTorch¶
torch.nn.LSTMandtorch.nn.GRU— recurrent layers. (7.4)torch.nn.Conv1dandtorch.nn.Embedding— the building blocks of the WaveNet architecture. (7.3)torch.utils.data.Dataset— building a custom windowed dataset. (7.3)
Datasets¶
JSB Chorales — 382 Bach chorales, from Géron’s own public dataset repository. (7.3)
Streamflow, temperature, and precipitation records for a gauged catchment. (7.4)