Autonomous tractors can operate in open fields by using the RTK-GNSS (Real-time Kinematic-Global Navigation Satellite System). However, RTK-GNSS signal might not be available at all geographical locations, which could make autonomous driving unsafe. To avoid such situations, we sought to develop a vision-based agricultural road detection system that does not rely on RTK-GNSS. It takes the RGB images obtained from an HD onboard camera, removes camera distortion, filters the images, detects the road edges, and calculates the lateral error. Experimental runs showed that the lateral error was less than 0.2 m for unpaved roads and 0.4 m for paved roads. This shows that the system can detect agricultural roads and thus can be used in place of RTK-GNSS.