Fuzzy c-means (FCM) clustering algorithm has been widely used in many medical image segmentations. However, the conventionally standard FCM algorithm is noise sensitive because of not taking into account the spatial information. To overcome the above problem, a novel modified FCM algorithm (called FCM–AWA later) for image segmentation is presented in this paper. The algorithm is realized by modifying the objective function in the conventional FCM algorithm, i.e., by incorporating the spatial neighborhood information into the standard FCM algorithm. An adaptive weighted averaging (AWA) filter is given to indicate the spatial influence of the neighboring pixels on the central pixel. The parameters (weighting coefficients) of control template (neighboring widow) are automatically determined in the implementation of the weighted averaging image by a predefined nonlinear function. The presented algorithm is applied to both artificial synthesized image and real image. Furthermore, the quantifications of dental plaque using proposed algorithm-based segmentation were conducted. Experimental results show that the presented algorithm performs more robust to noise than the standard FCM algorithm and another FCM algorithm (proposed by Ahmed) do. Furthermore, the results of dental plaque quantification using proposed method indicate the FCM–AWA provides a quantitative, objective and efficient analysis of dental plaque, and possesses great promise.