Part VIII — Neural Networks and Deep Learning

Booklet: Abridged Math Foundations for Signal Processing, Machine Learning, and Artificial Intelligence, Lessons 52–62 (PDF).

Feedforward networks, activations, and what depth buys; logits, softmax, and the losses maximum likelihood forces; logistic regression and kernel methods; the matrix and tensor calculus behind the derivations — outer products, Kronecker products, traces, and index notation; Jacobians, the chain rule, and backpropagation; vanishing and exploding gradients, initialization, normalization, and residual connections; convolutional networks — equivariance, receptive fields, and parameter sharing; recurrence, gating, and attention; autoencoders, variational autoencoders, and adversarial networks; risk, the bias–variance decomposition, and cross-validation; Monte Carlo, Metropolis–Hastings, and Gibbs sampling.

44 exercises across Lessons 52–62 — 22 [Hand], 22 [Proof].

Exercises

Worked sets are linked below as they are completed.

  • Lesson 52 — Feedforward Networks: Layers, Activations, and What Depth Buys — 4 exercises (2 [Hand], 2 [Proof])
  • Lesson 53 — Logits, Softmax, and the Losses That Come From Likelihood — 4 exercises (2 [Hand], 2 [Proof])
  • Lesson 54 — Logistic Regression and Kernel Methods: The Two Classical Routes to Nonlinearity — 4 exercises (2 [Hand], 2 [Proof])
  • Lesson 55 — The Matrix and Tensor Toolkit: Outer Products, Kronecker Products, and Index Notation — 4 exercises (2 [Hand], 2 [Proof])
  • Lesson 56 — Jacobians, the Chain Rule, and Backpropagation — 4 exercises (2 [Hand], 2 [Proof])
  • Lesson 57 — Training Dynamics: Vanishing Gradients, Initialization, and Normalization — 4 exercises (2 [Hand], 2 [Proof])
  • Lesson 58 — Convolutional Networks: Weight Sharing, Equivariance, and Receptive Fields — 4 exercises (2 [Hand], 2 [Proof])
  • Lesson 59 — Sequences: Recurrence, Gating, and Attention — 4 exercises (2 [Hand], 2 [Proof])
  • Lesson 60 — Learned Representations: Autoencoders, Variational Autoencoders, and Adversarial Networks — 4 exercises (2 [Hand], 2 [Proof])
  • Lesson 61 — Generalization: Risk, Validation, and Cross-Validation — 4 exercises (2 [Hand], 2 [Proof])
  • Lesson 62 — Sampling: Monte Carlo, Markov Chains, and Gibbs — 4 exercises (2 [Hand], 2 [Proof])