Essential terms for understanding deep learning research papers, tutorials and textbooks.
Term Description
Jacobian matrix The matrix containing all partial derivatives of a function whose input and output are both vectors
Hessian matrix Similar to Jacobian matrix but it contains the second derivatives collected in a matrix
First-order optimization algorithms Optimization algorithms that use only a gradient such as gradient descent
Second-order optimization algorithms Optimization algorithms that use Hessian matrix like Newton’s method
Constrained Optimization Find the maximal or minimal value of f(x)
Karush-Kuhn-Tucker (KKT) Approach General solution to constrained optimization making use of generalized Lagrange function (Lagrangian)
KKT Conditions Simple set of properties that describe the optimal points of constrained optimization problems
Hyperparameters Machine algorithms’ settings that must be determined external to the learning algorithm itself
Accuracy Proportion of examples for which the model produces the correct output
Error rate Proportion of examples for which the model produces an incorrect output
Design matrix Matrix containing a different example in each row
Underfitting Model cannot obtain sufficiently low error value
Overfitting Large gap between training and test error
Capacity Model’s ability to fit functions
Hypothesis space Set of functions learning algorithm s allowed to select as being the solution
Representational capacity The model specifies which family of functions the learning algorithm can choose from when varying the parameters to reduce training objective
Occam’s razor Among competing hypotheses, one should choose the simplest one
Vapnik-Chervonenkis (VC) dimension Measures the capacity of a binary classifier
Nearest neighbor Regression Non-parametric model minimizing the L2 norm of the point and the surrounding points
Parametric Models Models that learn a function described by a finite-sized parameter vector such as Linear Regression. And if it has less than optimal capacity, it will asymptote with an error value more than the Bayes error
Non-parametric Models No limitation on parameters such as nearest neighbour regression. And more data yields better generalization
Nearest neighbour regression It simply stores X and y and when given x it looks up for the nearest entry and returns the label
Bayes error The error incurred by an oracle, knowing the true probability distribution that generates the data, making predictions from the true distribution p(x, y)
Generalization error It can never increase with more training examples
No Free Lunch Theorem Averaged over all possible data generating distributions, every classification algorithm has the same error rate when classifying previously unobserved points
Weight decay Large (underfitting), medium (just right), small (overfitting)
Regularization We can regularize a model simply by adding a penalty to the cost function called a Regularizer. There are other ways too and a more generic definition is: regularization is any modification we make to the algorithm that is intended to reduce the generalization error not the training error
Hyperparameter Settings that we can use to control the behaviour of the learning algorithm. The setting must be a hyperparameter because it is not appropriate to learn that hyperparameter for the training set such as hyperparameter controlling model capacity where it would always choose to maximize the model capacity for the training set that results in overftting
Validation set Examples that the training algorithm does not observe. This is not the test set. It is used to guide the selection of our hyperparameters. Since it is used to “train” the hyperparameters, the validation set error will underestimate the generalization error though typically by a smaller amount than the training error
Test set This is the set we use to estimate our generalization error after all our hyperparameter optimization is complete. If the test set is small, it can be problematic as this implies statistical uncertainty around the estimated test error
K-fold Cross Validation This is computationally expensive. We partition the data into k non-overlapping subsets and the test set can be estimated by taking the average test error across k-trials
Point estimator or statistic Point estimation is the attempt to provide the single “best” prediction of some quantity of interest. Any function of the data that is drawn i.i.d. Since the data is drawn i.i.d. any function of the data is random and therefore the point estimator is a random variable
Function estimator This can also be called point estimator. But a function estimator is the estimation of the relationship between input and output variables
Estimator is ubiased if