Human Activity Analysis

By: Ryan Wendel

It is an ongoing analysis in which videos are analyzed frame by frame

Most of the video recognition is pulled from 3-D graphic engines

What is the Human Activity Analysis?

“HAA” stands for Human Activity Analysis Surveillance systems Patient monitoring systems Human-computer interfaces

What HAA covers

We are going to take a look at methodologies that have been developed for simple human actions.

And high-level activities.

What we will cover

Gestures Actions Interactions Group activities

Basic Human Activities

Basic movements of a persons body parts. For example: Raising an arm Lifting a leg

Gestures

A Single persons activities which could entail multiple gestures.

For example: Walking Waving Shaking body

Actions

Interactions that involve two or more people / items.

For Example: Two people fighting

Interactions

Activities performed by multiple people. For example: A group running A group walking A group fighting

Group Activities

Can be separated into two sections◦ Single-layered approaches: An approach that

deals with recognizing human activities based on a video feed (frame by frame.)

◦ Hierarchical approaches: An approach aimed at describing the high level approach to HAA by showing high level activities in simpler terms.

Activity Recognition Methodologies

Main objective is to analyze simple sequences of movements of humans

Can be categorized into two different categories ◦ Space-time approach: takes an input video as a

3-D volume◦ Sequential approach: takes an input video and

interprets it as a sequence of observations

Single-layered approaches

Divided into three different subsections based on features◦ Space-time volume◦ Space-time Trajectories ◦ Space-time features

Space-time approach

Captures a group of human activities by analyzing volumes of a video (frame by frame.)

Also uses types of recognition using space-time volumes to measure similarities between two volumes

Space-Time Volume

Uses stick figure modeling to extract joint positions of a person at each frame by frame

Space-Time Trajectories

Does not extract features frame by frame Extracts features when there is a

appearance or shape change in 3-D Space-time volume

Space-Time features

Space-Time Volume◦ Hard to differentiate between multiple people in

the same scene. Space-Time Trajectories

◦ 3-D body-part detection and tracking is still an unsolved problem, and it requires a strong low-level component that can estimate 3-D join location.

Space-Time features◦ Not suitable for modeling complex activities

Disadvantages of Space-time approach

Divided into two different subsections based on features◦ Exemplar-based◦ State model-based

Sequential approach

Review◦ Sequential approach: takes an input video and

interprets it as a sequence of observations Exemplar-based

◦ Shows human activities with a set of sample sequences of action executions

Exemplar-based

Sequential set of sequences that represent a human activity as a model composed of a set of states.

State Model-Based

Exemplar-based is more flexible in terms of comparing multiple sample sequences

Where as State Model-based can handle a probabilistic analysis of an activity better.

Exemplar vs State Model

Sequential approach is able to handle and detect more complex activities performed

Whereas the Space-time approach handles simpler less complex activities.

Both methods are based off of some type of a sequences of images

Space-time vs Sequential approach

Allows the recognition of high-level activities based on the recognition results of other simpler activities

Advantages of the Hierarchical Approach◦ Has the ability to recognize high-level activities

with a more in depth structure◦ Amount of data required to recognize an activity

is significantly less then single-layered approach◦ Easier to incorporate human knowledge

Hierarchical Approaches

Statistical approach Syntactic approach Description-based approach

Three main subgroups of Hierarchical approach

Statistical approaches use the state-based models to recognize activities

If you use multiple layers of a state-based model you can use these separate models to recognize activities with sequential structures

Statistical approach

Human activities are recognized as a string of symbols

Human activities are shown as a set of production rules generating a string of actions

Syntactic approach

Human activities that use recognition with complex spatio-temporal structures◦ A spatio-temporal structure is a detector used for

recognizing human actions Uses Context-free grammars (CFGs) to

represent activities ◦ CFGs are used to recognize high-level activities◦ The detection extracts space-time points and

local periodic motions to obtain a sparse distribution of interest points in a video

Description-based approach

Probability theory Fuzzy logic Bayesian network:

◦ Used for recognition of an activity, based on the activities temporal structure representation

◦ Uses a large network with over 10,000 nodes

Image Understanding (IU)

A group of persons marching◦ The images are recognized as an overall motion

of an entire group A group of people fighting

◦ Multiple videos are used to recognize the activity that a “group is fighting”

Group Activities

Recognition of interactions between humans and objects requires multiple components involved.

A lot of human-object interaction ignores interaction between object recognition and motion estimation

You can also factor in object dependencies, motions, and human activities to determine activities involved

Interactions between humans and Objects

J.K. Aggarwal and M.S. Ryoo. 2011. Human activity analysis: A review. ACM Comput. Surv. 43, 3, Article 16 (April 2011), 43 pages. DOI=10.1145/1922649.1922653 http://doi.acm.org/10.1145/1922649.1922653

  Christopher O. Jaynes. 1996. Computer vision and artificial intelligence.

Crossroads 3, 1 (September 1996), 7-10. DOI=10.1145/332148.332152 http://doi.acm.org/10.1145/332148.332152

Zhu Li, Yun Fu, Thomas Huang, and Shuicheng Yan. 2008. Real-time human action recognition by luminance field trajectory analysis. In Proceedings of the 16th ACM international conference on Multimedia (MM '08). ACM, New York, NY, USA, 671-676. DOI=10.1145/1459359.1459456 http://doi.acm.org/10.1145/1459359.1459456

Paul Scovanner, Saad Ali, and Mubarak Shah. 2007. A 3-dimensional sift descriptor and its application to action recognition. In Proceedings of the 15th international conference on Multimedia (MULTIMEDIA '07). ACM, New York, NY, USA, 357-360. DOI=10.1145/1291233.1291311 http://doi.acm.org/10.1145/1291233.1291311

References

Questions?