In this paper we present an automatic facial expression recognition system that utilizes a semantic-based learning algorithm using the analytical hierarchy process (AHP). All the automatic facial expression recognition methods are similar in that they first extract some low-level features from the images or video, then these features are used as inputs into a classification system, and the outcome is one of the preselected emotion categories. Although the effectiveness of low-level features in automatic facial expression recognition systems has been widely studied, the success is shadowed by the innate discrepancy between the machine and human perception to the image. The gap between low-level visual features and high-level semantics should be bridged in a proper way in order to construct a seamless automatic facial expression system satisfying the user perception. For this purpose, we use the AHP to provide a systematical way to evaluate the fitness of a semantic description for interpreting the emotion of a face image. A semantic-based learning algorithm is also proposed to adapt the weights of low-level visual features for automatic facial expression recognition. The weights are chosen such that the discrepancy between the facial expression recognition results obtained in terms of low-level features and high-level semantic description is small. In the recognition phase, only the low-level features are used to classify the emotion of an input face image. The proposed semantic learning scheme provides a way to bridge the gap between the high-level semantic concept and the low-level features for automatic facial expression recognition. Experimental results show that the performance of the proposed method is excellent when it is compared with that of traditional facial expression recognition methods.