Abstract: The status of grains, including impurity and breakage, is one of the key criteria for the assessment of combine harvester, whereas the on-line monitoring system for grain impurity and breakage is not yet well understood. In this paper a method of image capturing and processing for rice impurity and breakage based on machine vision was presented. The machine vision system designed was mainly composed of grain collection device, embedded processor, industrial camera, light source and display module. One hundred paddy images were collected when the combine harvester was working. The image resolution was set to 1 600×1 200 pixels, and the image format was jpg. There were lots of phenomena of grain stacking and adhesion in the rice images collected from this experiment. It was necessary to recognize MOG (material other than grain) and broken paddy grains from an image in complex background. HSV (hue, saturation, value) color space is closer to human perception of color and is more suitable for color expression based on machine vision than RGB (red, green, blue) color space. The interval ranges of hue channel and saturation channel of whole grains, impurities and broken grains were analyzed from 20 images sampled randomly from 100 images, and the boundaries between intact rice, impurities and broken grains were found. Among them, the saturation value of broken grain was between 0 and 60, while the saturation values of intact rice and impurity were mainly between 60 and 255. From the hue value as a whole, the hue value of intact rice was smaller than that of stem impurities, and the hue value of stem impurities was less than that of branch impurities. But mere color feature was not enough to recognize the accurate recognition, and the shape characteristics of impurities and grains such as length, width and area were also needed to eliminate the interference part. On the embedded processor, the computer vision library OpenCV was used to design the image processing algorithm. The multi-scale Retinex with color recovery algorithm was used to enhance the original image and then different thresholds were set up in the hue and saturation channel of the HSV color model for image segmentation respectively. Then the shape features of impurity and broken paddy grains were used to obtain the detection results. The comprehensive evaluation index F1-score was used on the detection results for the quantitative evaluation. F1 is a comprehensive consideration of precision and recall rate. It was showed that the F1 of detected stem of impurity reached 86.92%, the F1-score of detected small branch of impurity was 85.07%, the F1 of detected broken paddy grain was 84.74%, and the average time for the detection of an image was 3.24 s. The proposed technology can effectively recognize the impurities and broken paddy grains from the captured images, which provides a solid foundation for monitoring impurity and breakage rate of paddy grain during harvesting, and also provides a reference for future research on the identification methods of wheat images containing impurities and broken grains. Later, this recognition method can be combined with a variety of color models for image recognition, which can further improve the accuracy of recognition.