Our group is researching a support system for cultivating Shine-Muscat grapes. The ripeness of the grapes is determined by their color and is compared against a Shine-Muscat color chart. In this paper, we introduce a color estimation model that can determine the optimal time for harvesting by using a deep learning model to estimate the color of the grape bunch. We incorporate metric learning during model training and use the specific color space representation of images to achieve high accuracy in our color estimation model. In addition, we propose a method to incorporate an appropriate reference color into the color estimation model. Our evaluation experiment indicatesd that our proposed model correctly identified the optimal time to harvest for 85.9% of the grapes.