In this study, we developed 44 working motion detector templates and a behavioral recognition system with the motion templates to recognize worker behaviors, such as turning, squatting, and moving arms, performed by a worker along furrows in the cultivated field of peas. The 44 motion detector templates were designed to identify working motions such as sitting down, moving arms, turning counterclockwise, turning clockwise, up-and-down, side-to-side. The behavioral recognition system with the motion templates was designed to recognize the 12 behavior types and the three behavior category types of turning, squatting, and moving arms behaviors. In the experiments, the developed motion detector templates could consistently well respond to the working motions in the 12 types of behaviors performed by two workers. The behavioral recognition system with the motion templates could recognize the three behavior category types could be robustly determined with the total recognition rate of 80.5%. The results of the experiments demonstrated the applicability and robustness of the developed working motion templates and the behavioral recognition system with the motion templates to recognize the three behavior category types.