VIDEO SEGMENTS BY TOPIC

Here is the map of machine learning

Index of Topics

-- Use your browser's "Find" to look for keywords below --

Aggregation

Overview of ensemble learning (boosting, blending, before and after the fact)

Bayesian Learning

Validity of the Bayesian approach (prior, posterior, unknown versus probabilistic)

Bias-Variance Tradeoff

Basic derivation (overfit and underfit, approximation-generalization tradeoff)
Example (sinusoidal target function)
Noisy case (Bias-variance-noise decomposition)

Bin Model

Hoeffding Inequality (law of large numbers, sample, PAC)
Relation to learning (from bin to hypothesis, training data)
Multiple bins (finite hypothesis set, learning: search for green sample)
Union Bound (uniform inequality, M factor)

Data Snooping

Definition and analysis (data contamination, model selection)

Error Measures

User-specified error function (pointwise error, CIA, supermarket)

Gradient Descent

Basic method (Batch GD) (first-order optimization)
Discussion (initialization, termination, local minima, second-order methods)
Stochastic Gradient Descent (the algorithm, SGD in action)
Initialization - Neural Networks (random weights, perfect symmetry)

Learning Curves

Definition and illustration (complex models versus simple models)
Linear Regression example (learning curves for noisy linear target)

Learning Diagram

Components of learning (target function, hypothesis set, learning algorithm)
Input probability distribution (unknown distribution, bin, Hoeffding)
Error measure (role in learning algorithm)
Noisy targets (target distribution)
Where the VC analysis fits (affected blocks in learning diagram)

Learning Paradigms

Types of learning (supervised, reinforcement, unsupervised, clustering)
Other paradigms (review, active learning, online learning)

Linear Classification

The Perceptron (linearly separable data, PLA)
Pocket algorithm (non-separable data, comparison with PLA)

Linear Regression

The algorithm (real-valued function, mean-squared error, pseudo-inverse)
Generalization behavior (learning curves for linear regression)

Logistic Regression

The model (soft threshold, sigmoid, probability estimation)
Cross entropy error (maximum likelihood)
The algorithm (gradient descent)

Netflix Competition

Movie rating (singular value decomposition, essence of machine learning)
Applying SGD (stochastic gradient descent, SVD factors)

Neural Networks

Biological inspiration (limits of inspiration)
Multilayer perceptrons (the model and its power and limitations)
Neural Network model (feedforward layers, soft threshold)
Backpropagation algorithm (SGD, delta rule)
Hidden layers (interpretation)
Regularization (weight decay, weight elimination, early stopping)

Nonlinear Transformation

Basic method (linearity in the parameters, Z space)
Illustration (non-separable data, quadratic transform)
Generalization behavior (VC dimension of a nonlinear transform)

Occam's Razor

Definition and analysis (definition of complexity, why simpler is better)

Overfitting

The phenomenon (fitting the noise)
A detailed experiment (Legendre polynomials, types of noise)
Deterministic noise (target complexity, stochastic noise)

Radial Basis Functions

Basic RBF model (exact interpolation, nearest neighbor)
K Centers (Lloyd's algorithm, unsupervised learning, pseudo-inverse)
RBF network (neural networks, local versus global, EM algorithm)
Relation to other techniques (SVM kernel, regularization)

Regularization

Introduction (putting the brakes, function approximation)
Formal derivation (Legendre polynomials, soft-order constraint, augmented error)
Weight decay (Tikhonov, smoothness, neural networks)
Augmented error (proxy for out-of-sample error, choosing a regularizer)
Regularization parameter (deterministic noise, stochastic noise)

Sampling Bias

Definition and analysis (Truman versus Dewey, matching the distributions)

Support Vector Machines

SVM basic model (hard margin, constrained optimization)
The solution (KKT conditions, Lagrange, dual problem, quadratic programming)
Soft margin (non-separable data, slack variables)
Nonlinear transform (Z space, support vector pre-images)
Kernel methods (generalized inner product, Mercer's condition, RBF kernel)

Validation

Introduction (validation versus regularization, optimistic bias)
Model selection (data contamination, validation set versus test set)
Cross Validation (leave-one-out, 10-fold cross validation)

VC Dimension

Growth function (dichotomies, Hoeffding Inequality)
Examples (growth function for simple hypothesis sets)
Break points (polynomial growth functions)
Bounding the growth function (mathematical induction, polynomial bound)
Definition of VC Dimension (shattering, distribution-free, Vapnik-Chervonenkis)
VC Dimension of Perceptrons (number of parameters, lower and upper bounds)
Interpreting the VC Dimension (degrees of freedom, Number of examples)