Fig. 1. Overview of the system.

Fig. 2. A representation of the complex event “converse.”

(A) Frame 74.

(B) Frame 75.

Fig. 3. Splitting of moving regions and noise.

Fig. 4. Projection of the bottom points of moving regions on to the ground plane.

Fig. 5. The “Stealing by Blocking” sequence. “A” approaches a reference object (a person standing in the middle with his belongings on the ground). “B” and “C” then approach and block the view of “A” and the reference person from their belongings. In the mean time, “D” comes and takes the belongings.

(A) Before.

(B) After.

Fig. 6. A graph representation of the possible tracks of object “D.” (A) Without using the ground plane knowledge, several hypotheses can be made about the possible tracks of the object. (B) After filtering, regions are merged or disregarded, decreasing the ambiguity.

(A) Object 2 is totally occluded by the tree.

(B) Discontinuing tracks between object 5 and 6.

Fig. 7. (A) Object 3 is considered a different object from object 2 due to the total occlusion. (B) Due to the reflection on a car surface, the moving region of object 5 is larger than that of object 6, causing the discontinuity of the trajectory.

Fig. 8. Trajectories of the objects tracked using the ground plane knowledge and K–S statistics.

Fig. 9. A finite-state automaton that represents the complex event “converse.”

Fig. 10. A graphical description of a multi-thread event.

Fig. 11. A detailed illustration of a naive Bayesian classifier that is used to infer “approach the reference person” in Fig. 2.

(A) Simple events.

(B) Complex event: “Converse.”

Fig. 12. Analysis of the single-thread events for object “A.”

(A) Sub-events of “Taking Object.”

(B) Complex event: “Taking Object.”

Fig. 13. Analysis of single-thread events for object “D.”

(A) Detection and tracking of moving regions for “contact1.”

(B) Recognition results of two competing activities.

Fig. 14. (A) Input sequence A shows a complex event “contact1.” Object 1 (at the top) approaches object 2 (at the bottom), makes contact (both objects have merged as they meet), turns around, and leaves. (B) “contact1” is recognized with P(MS*|O) = 0.7. On the other hand, “passing by” is recognized with lower probability (almost 0 at the end) since sub-event “leaving without turning around” is not established.

(A) Detection and tracking of moving regions for scenario “passing by.”

(B) Recognition results of two competing activities.

Fig. 15. (A) Input sequence B shows a complex event “passing by”: Object 1 walks past object 2. (B) “passing by” is recognized with higher confidence (P(MS*|O) = 0.6) than “contact1” (P(MS*|O) = 0.2).

Fig. 16. The likelihood of the complex event inferred by a probabilistic finite-state automaton. The event, at different times, may have different likely start times as illustrated by “X,” depending on the most likely transition timings between states considered up to the current frame.

(A) Recognition of single-thread events.

(B) Multi-thread events evaluated related to “blocking” and the corresponding assignment of actors.

(C) Two recognition results of “stealing,” each with different actor and temporal event combinations.

Fig. 17. Event analysis results of the sequence “stealing.”

(A) Results on perturbed sequences.

(B) Example of a perturbed sequence.

Fig. 18. (A) The recognition of “contact1” over 400 perturbed image sequences. (B) The recognition results of a perturbed “contact1” sequence with the loss of tracking on every three frames. The recognition of sub-events becomes zero every time the mobile object is lost. “contact1,” however, is still correctly recognized.

Fig. 19. ROC curves for a data set with various noise levels.

(A) Short “approach then leave.”

(B) Long “approach then leave.”

(C) Short “approach then stop.”

(D) Long “approach then stop.”

Fig. 20. Discriminating “approach then leave” and “approach then stop” in the test sequences of various lengths.

Fig. 21. A graphical description of the event “cooperative object transfer.”

Fig. 22. ROC curves for a data set with varying execution styles.

Description of the multiple-thread event “stealing”

Computation time of video sequences