The ability to group visual stimuli into meaningful categories is a fundamental cognitive process. Some experiments are made to investigate the neural mechanism of visual categorization. Although experimental evidence is known that prefrontal cortex (PFC) and inferior temporal cortex (ITC) neurons sensitively respond in categorization task, little is known about the role of interaction between ITC and PFC in categorization task that gives the category boundaries in relation to behavioral consequences. To address this issue, we propose a functional model of visual system in which categorization task is achieved based on functional roles of ITC and PFC. The functional role of ITC is to represent features of object parts, based on different resolution maps in early visual system such as V1 and V4. In ITC, visual stimuli are categorized by the similarity based on the features of object parts. The PFC neurons combine the information about feature and location of object parts, and generate a working memory of the object information relevant to the categorization task. The synaptic connections between ITC and PFC are learned so as to achieve the categorization task. The feedback signals from PFC to ITC enhance the sensitivity of ITC neurons that respond to the features of object parts critical for the categorization task. In the present study, we present a neural network model, which makes categories of visual objects depending on categorization task. We investigated the neural mechanism of the categorization task of line drawings of faces used by Sigala and Logothetis (N. Sigala, N.K. Logothetis, Visual categorization shapes feature selectivity in the primate temporal cortex, Nature 415 (2002) 318–320). Using this model we show that ITC represents similarity of face images based on the information of the resolution maps in V1 and V4. We also show that PFC generates a working memory state, in which only the information of face features relevant to the categorization task are sustained. It is further shown that the feedback connections from PFC to ITC may play an important role in extracting the diagnostic features of visual images in the categorization task.