Gradient descent, backpropagation, loss functions and regularisation, with a recurrent network for Bitcoin prices
|
Deep Learning (pdf)
|
3.15 M |
A neural network is a function whose weights are fitted by mini-batch gradient descent, with gradients obtained by backpropagation. Differentiation can be symbolic, automatic with dual numbers, or numerical, with different costs and rounding errors. Fully connected, convolutional and recurrent networks are three architectures with different connection patterns between neurons. Loss functions are chosen by task, mean squared error for regression and cross-entropy for classification, with sigmoid or softmax output layers for probabilities. Overfitting is reduced by penalty terms, dropout, early stopping and data augmentation. In a worked example in Python, simple RNN and LSTM models are fitted to Bitcoin prices, a financial time series, with a time-ordered train and test split, scaling, grid search with cross-validation and a white noise test of the residuals.