Grammar-Based Event Detection from Video

Abstract

In these days, a vast amount of videos is recorded and archived every minute and demand on automatic video understanding increases consequently. As a step toward video understanding, in this thesis we focus on composite event detection from video. A composite event is composed of multiple primitive actions, which are arranged in a specific temporal-logical context to attain full meaning. Composite event detection aims to discover high-level interpretation of video through the context as well as to detect primitive actions accurately. The contextual structure of a composite event is called scenario, and we assume that scenario is described manually by domain experts.We first propose a new scenario description method. A scenario description method is important for video event detection since its expressive power determines the range of events to be detected. A set of temporal-logical predicates is defined to represent relationships between primitive actions more fluently. The proposed description method is in a form of regular grammar, which is based on the temporal-logical predicates instead of simple ordering of the original grammar. Consequently, the description method has more expressive power and is easy to describe complex composite events at the same time. More flexible scenarios are required to represent complicated composite events in real videos, but enlarge the search space prohibitively as well. We move to an inference algorithm to detect composite event efficiently and exactly even with the huge search space. To this end, we propose constraint flow, which is a combinatorial state transition machine and equivalent with scenario. Our inference algorithm is based on dynamic programming with the constraint flow. We show that the search space containing the globally optimal solution can be reduced significantly by constraint flow and the compact search space allows an on-line and efficient inference algorithm.Most event detection frameworks including the above assume that every agent in video participate in an event with known role. However, such assumption is invalid in real videos, where it is unknown which agent participates in an event with which role. We finally propose an efficient method to identify participants and their roles jointly. We observe that the role of an agent can be estimated by analyzing actions of the agent. Also, the agent-wise role analysis is much more efficient than event detection. Given the results of the agent-wise role analysis, the joint identification problem is solved efficiently by a two-step optimization. By applying event detector only to the identified participants, composite event detection in real videos could be done more efficiently and accurately than a naive approach that detects events from all possible agent combinations.