Abstract
We investigate the use of the so-called variably scaled kernels (VSKs) for learning tasks, with a particular focus on support vector machine (SVM) classifiers and kernel regression networks (KRNs). Concerning the kernels used to train the models, under appropriate assumptions, the VSKs turn out to be more expressive and more stable than the standard ones. Numerical experiments and applications to breast cancer and coronavirus disease 2019 (COVID-19) data support our claims. For the practical implementation of the VSK setting, we need to select a suitable scaling function. To this aim, we propose different choices, including for SVMs a probabilistic approach based on the naive Bayes (NB) classifier. For the classification task, we also numerically show that the VSKs inspire an alternative scheme to the sometimes computationally demanding feature extraction procedures.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Aggarwal, C.C.: Data Classification: Algorithms and Applications, Boca Raton, FL, USA CRC Press (2014)
Aminian Shahrokhabadi, M., Neisy, A., Perracchione, E., Polato, M.: Learning with subsampled kernel-based methods: Environmental and financial applications. Dolomites Res. Notes Approx. 12, 17–27 (2019)
Ball, K.: Eigenvalues of Euclidean distance matrices. J. Approx. Theory 68, 74–82 (1992)
Bartlett, P.L., Mendelson, S.: Rademacher and Gaussian complexities: risk bounds and structural results. J. Mach. Learn. Res. 3, 463–482 (2002)
Bhatia, R.: Matrix Analysis. Springer-Verlag, New York (1997)
Bozzini, M., Lenarduzzi, L., Rossini, M., Schaback, R.: Interpolation with variably scaled kernels. IMA J. Numer. Anal. 35, 199–219 (2015)
Bui, D.T., Pradhan, B., Lofman, O., Revhaug, I.: Landslide susceptibility assessment in Vietnam using support vector machines, decision tree and Naïve Bayes models, Math. Probl. Eng. 1–26
Campagna, R., Conti, C., Cuomo, S.: Smoothing exponential-polynomial splines for multiexponential decay data. Dolomites Res. Notes Approx. 12, 86–100 (2019)
Daumé, H.: Frustratingly easy domain adaptation. In: Association for computational linguistics (ACL) (2007)
De Marchi, S., Erb, W., Marchetti, F., Perracchione, E., Rossini, M.: Shape-Driven Interpolation with Discontinuous Kernels: Error Analysis, Edge Extraction and Applications in MPI. SIAM J. Sci. Comput. 42, B472–B491 (2020)
De Marchi, S., Marchetti, F., Perracchione, E.: Jumping with variably scaled discontinuous kernels (VSDKs). BIT Num. Math. 60, 441–463 (2020)
Diederichs, B., Iske, A.: Improved estimates for condition numbers of radial basis function interpolation matrices. J. Approx. Theory 238, 38–51 (2019)
Donini, M., Aiolli, F.: Learning deep kernels in the space of dot product polynomials. Mach. Learn. 106, 1245–1269 (2017)
El Karoui, N.: The spectrum of kernel random matrices. Ann. Statist. 38, 1–50 (2010)
Fasshauer, G.E.: Meshfree Approximations Methods with Matlab. World Scientific, Singapore (2007)
Fasshauer, G.E., McCourt, M.J.: Kernel-based Approximation Methods Using Matlab. World Scientific, Singapore (2015)
Hoffmann, H.: Kernel PCA for novelty detection. Pattern Recogn. 40, 863–874 (2007)
Horn, R.A., Zhang, F.: Bounds on the spectral radius of a Hadamard product of nonnegative or positive semidefinite matrices. Electron J. Linear Algebra 20, 90–94 (2010)
Kim, K. I., Jung, K., Kim, H.J.: Face recognition using kernel principal component analysis. IEEE Signal Proc. Lett. 9, 40–42 (2002)
Larsson, E., Fornberg, B.: Theoretical and computational aspects of multivariate interpolation with increasingly flat radial basis functions. Comput Math. Appl. 49, 103–130 (2005)
Li, W., Duan, L., Xu, D., Tsang, I.W.: Learning with augmented features for supervised and semi-supervised heterogeneous domain adaptation. IEEE Trans. Pattern. Anal. Mach. Intell. 36, 1134–1148 (2014)
Mangasarian, O.L., Nick Street, W., Wolberg, W. H.: Wisconsin breast cancer database, UCI machine learning repository. http://archive.ics.uci.edu/ml University of Wisconsin (1991)
Mangasarian, O.L., Wolberg, W.H.: Cancer diagnosis via linear programming. SIAM News 106, 1–18 (1990)
Marchetti, F.: The extension of Rippa’s algorithm beyond LOOCV. Appl. Math. Lett. 120, 107262 (2021)
Maron, M.E.: Automatic indexing: An experimental inquiry. J ACM. 8, 404–417 (1961)
Mercer, J.: Functions of positive and negative type and their connection with the theory of integral equations. Phil. Trans. Royal Society 209, 415–446 (1909)
Muquet, B., Wang, Z., Giannakis, G.B., De Courville, M., Duhamel, P.: Cyclic prefixing or zero padding for wireless multicarrier transmissions? IEEE Trans. Commun. 50(12), 2136–2148 (2002)
Narcowich, F.J., Ward, J.F.: Norm estimates for the inverses of a general class of scattered-data radial-function interpolation matrices. J. Approx. Theory 69, 84–109 (1992)
Nocedal, J., Wright, S.J.: Numerical Optimization. Springer-Verlag, New York (1999)
Orr, M.J.L.: Introduction to radial basis function networks, Tech. rep., University of Edinburgh Centre for Cognitive Sciences (1996)
Pang, B., Lee, B., Vaithyanathan, S.: Thumbs up? Sentiment Classification Using Machine Learning Techniques. In: Proc. of EMNLP, pp. 79–86 (2002)
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: Machine learning in python. J. Mach. Learn. Res. 12, 2825–2830 (2011)
Ren, J., Lee, S.D., Chen, X., Kao, B., Cheng, R., Cheung, D.: Naive Bayes classification of uncertain data. In: Proc. 9th IEEE Int. Conf. Data Mining (ICDM), pp. 944–949 (2009)
Rippa, S.: An algorithm for selecting a good value for the parameter c in radial basis function interpolation. Adv. Comput. Math. 11, 193–210 (1999)
Romani, L., Rossini, M., Schenone, D.: Edge detection methods based on RBF interpolation. J. Comput. Appl. Math. 349, 532–547 (2019)
Rossini, M.: Interpolating functions with gradient discontinuities via variably scaled kernels. Dolom. Res. Notes Approx. 11, 3–14 (2018)
Schaback, R.: Error estimates and condition numbers for radial basis function interpolation. Adv. Comput. Math. 3, 251–264 (1995)
Schaback, R., et al.: Multivariate interpolation and approximation by translates of a basis function. In: Chui, C. (ed.) Approximation Theory VIII: Approximation and Interpolation, pp 491–514. World Scientific, Singapore (1995)
Schölkopf, B., Smola, A.J.: Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, Cambridge (2002)
Schur, J.: Bemerkungen zur Theorie der beschränkten Bilinearformen mit unendlich vielen Veränderlichen. J. Reine Angew. Math. 140, 1–28 (1911)
Stura, I., Gabriele, D., Guiot, C.: A simple PSA-based computational approach predicts the timing of cancer relapse in prostatectomized patients. Cancer Res. 76, 4941–4947 (2016)
Shawe-Taylor, J., Cristianini, N.: Kernel Methods for Pattern Analysis. Cambridge University Press, Cambridge (2004)
Styan, G.P.H.: Hadamard products and multivariate statistical analysis. Linear Algebra Appl. 6, 217–240 (1973)
Vapnik, V.N., Chervonenkis, A.Y.: On the uniform convergence of relative frequencies of events to their probabilities. Theory Probab. Appl. 16, 264–280 (1971)
Weston, J., Bengio, S., Usunier, N.: Large scale image annotation: Learning to rank with joint word-image embeddings. Mach. Learn. 81, 21–35 (2010)
Wirtz, D., Haasdonk, B.: A vectorial kernel orthogonal greedy algorithm. Dolomites Res. Notes Approx. 6, 83–100 (2013)
Zhang, D., Lee, W.S.: Question classification using support vector machines. In: Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, New York, NY, 26–32 (2003)
Acknowledgements
We sincerely thank the reviewers for helping us to significantly improve the manuscript. This research has been accomplished within Rete ITaliana di Approssimazione (RITA) and partially funded by GNCS-INδ AM, by the European Union’s Horizon 2020 research and innovation programme ERA-PLANET, grant agreement no. 689443, via the GEOEssential project and by the ASI - INAF grant “Artificial Intelligence for the analysis of solar FLARES data (AI-FLARES).”
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by: Robert Schaback
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Campi, C., Marchetti, F. & Perracchione, E. Learning via variably scaled kernels. Adv Comput Math 47, 51 (2021). https://doi.org/10.1007/s10444-021-09875-6
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10444-021-09875-6
Keywords
- Variably scaled kernels
- Kernel ill-conditioning
- Meshfree methods
- Binary classification
- Regression networks