The reliability analysis of complex structures is hindered by the implicit nature of the limit-state function. For their approximation use has been made of the Response Surface Method (RSM) and, more recently, of Neural Networks. From the statistical viewpoint this corresponds to a regression approach. In the structural reliability literature little attention has been paid, however, to the possibility of treating the problem as a classification task. This enlarges the list of methods that are eventually useful to the purpose at hand and justifies an overall examination of their distinguishing features. This task is performed in this paper from the point of view of the Theory of Statistical Learning, which provides a unified framework for all regression, classification and probability density estimation. The classification methods are grouped into three categories and it is shown that only one group is useful for structural reliability, according to some specific criteria. In this category are the Multi-Layer Perceptrons and the Support Vector Machines, which are the recommended methods because (a) they can estimate the function on the basis of a few samples, (b) they use flexible and adaptive models and (c) they can overcome the curse of dimensionality. The paper also includes an in-depth analysis of the RSM from the point of view of statistical learning. It is shown that the empirically found instability of this method is explained with statistical learning concepts.