ReviewA tutorial on kernel methods for categorization
Section snippets
Inner products
So what is a kernel? Kernels can be regarded as a non-linear generalization of inner products. We will take a little detour before explaining kernels and discuss the relationship between inner products, perceptrons and prototypes. This will set the stage on which kernels appear naturally to solve non-linear classification problems.
Kernels
The next section will introduce the kernel trick that makes it possible to work with high dimensional (even infinite dimensional) and flexible linearization spaces.
Regularization
By using the exemplar network with as many free parameters as stimuli it is always possible to find weights such that the network can classify all training stimuli perfectly. The price for this flexibility is the danger of overfitting. A network may learn to categorize all training stimuli perfectly but only because it has learned the stimuli by heart. Any regularity in the data is overlooked in this way and therefore the network will not be able to generalize. An example for overfitting is
Conclusions
We have introduced kernel methods as they are used in machine learning. The most important results here are the kernel trick and its link to reproducing kernel Hilbert spaces. On the way we have hinted to parallels with psychological theories. First, kernel methods can be implemented as a one-layer neural network. Second, the Gaussian kernel can be interpreted as a similarity measure and representation of the stimuli in a RKHS can be seen as representing the stimuli via their similarity to all
Acknowledgments
We would like to thank Jakob Macke, Jan Eichhorn, Florian Steinke, and Bruce Henning for comments on an earlier draft of this work.
References (43)
- et al.
Categorization as probability density estimation
Journal of Mathematical Psychology
(1995) - et al.
Relations between prototype, exemplar, and decision bound models of categorization
Journal of Mathematical Psychology
(1993) - et al.
Evaluating an adaptive network model of human learning
Journal of Memory and Language
(1988) - et al.
Multilayer feedforward networks are universal approximators
Neural Networks
(1989) Relations between exemplar-similarity and likelihood models of classification
Journal of Mathematical Psychology
(1990)Pattern recognition and categorization
Cognitive Psychology
(1972)- et al.
Theoretical foundations of the potential function method in pattern recognition learning
Automation and Remote Control
(1964) - et al.
Decision rules in the perception and categorization of multidimensional stimuli
Journal of Experimental Psychology: Learning, Memory and Cognition
(1988) - et al.
Foundations of multidimensional scaling
Psychological Review
(1968) Neural networks for pattern recognition
(1995)
An introduction to support vector machines and other kernel-based learning methods
Color similarity represented as a metric of color space
Representation is representation of similarities
Behavioral and Brain Sciences
Induction of category distributions: A framework for classification learning
Journal of Experimental Psychology: Learning, Memory, and Cognition
Classification of faces in man and machine
Neural Computation
ALCOVE: An exemplar-based connectionist model of category learning
Psychological Review
Linear separability in classification learning
Journal of Experimental Psychology: Human Learning and Memory
Learning machines
Cited by (56)
Development of a smart multiphase system for disperse flows using machine learning
2024, International Journal of Multiphase FlowA boom damage prediction framework of wheeled cranes combining hybrid features of acceleration and Gaussian process regression
2023, Measurement: Journal of the International Measurement ConfederationMachine learning in multiscale modeling of spatially tailored materials with microstructure uncertainties
2021, Computers and StructuresA machine learning field calibration method for improving the performance of low-cost particle sensors
2021, Building and Environment