The present study aimed to clarify the types and situations of play which are difficult for referees to judge by reviewing the results of video challenge system introduced in volleyball competition. The data analyzed were all the challenge requested from 76 matches in volleyball competition in the 2016 Rio de Janeiro Olympics. The success rate of challenges was calculated from the number of successful challenges divided by the number of challenges for each type of play. As results, the following findings were obtained: 1) The total number of challenges was 400 times, the number of successful challenges (i.e., the number of errors in referee’s judgment) was 163 times, and the success rate of challenges was 40.8%. There was no difference in these variables between men’s and women’s competitions. 2) The number of successful challenges and the success rate of challenges for the blocker’s ball contact were higher than those for the other types of play, suggesting that it is one of the most difficult judgments for volleyball referees to determine the contact between the blocker and the ball. 3) Regarding the judgment on the ball contact of the blocker, the frequency of judgment corrected from no touch to touch was larger than that corrected from no touch to touch. Regarding the judgment on ball in/out, the frequency of judgment corrected from ball out to ball in was larger than that corrected from ball in to ball out. The present study revealed the types and situations of play which are likely to be erroneously judged in volleyball competition for the first time. These findings would be useful for improving the technique of referees from the viewpoint of using visual information and for the future application of video challenge system in volleyball.