research and development of multimedia data management...
TRANSCRIPT
ShuShu--ChingChing ChenChenProfessor
Distributed Multimedia Information Systems LaboratoryFlorida International University
(The State University of Florida at Miami)
Contact: [email protected]://www.cs.fiu.edu/~chens/
Research and Development of Multimedia Data Management System (MDBMS)
AgendaAgendaMotivation and ChallengesMotivation and ChallengesPrototype MDBMS SystemPrototype MDBMS SystemApplicationsApplications
Social NetworkingSocial NetworkingHighHigh--level Feature Extractionlevel Feature ExtractionStorm Surge Animation ModelStorm Surge Animation ModelFlorida Public Hurricane Loss ModelFlorida Public Hurricane Loss Model
Motivation and ChallengesMotivation and ChallengesWhy multimedia data popular?
Attractive InformativeCompactCheap memory makes storage easy
Why handling and representing multimedia data challenging?Huge size (a typical 10 sec MPEG video ~4M)Temporal and spatial informationSemantic meaningMultidimensional representation
Why a separate database management system for multimedia data?Traditional database incapable of accommodating above characteristics efficiently
Motivation and Challenges (Cont.)Motivation and Challenges (Cont.)Three Research issues in Multimedia Database Three Research issues in Multimedia Database Management System (1)Management System (1)A. Storage and Representation: query efficiency and accuracy depends on optimum storage and representation
Leads to investigation in feature selection and indexingConflicting Issues
Greater the feature space, more accurate the query resultGreater the feature space, greater the computation overhead
Thus, we needEfficient Indexing supporting varied multimedia retrieval Optimum feature selection to represent semantic, temporal and spatial information
Motivation and Challenges (Cont.)Motivation and Challenges (Cont.)Three Research issues in Multimedia Database Three Research issues in Multimedia Database Management System (2)Management System (2)B. Query and Retrieval: temporal, spatial and semantic content should be considered during query of multimedia data
Leads to investigation in Image and Video retrieval strategiesLeads to investigation in concept identification and event detection via multimedia data mining
Eg. Key word based query: SunsetSunset
KeokiSunsetBottled
SunsetView
Driving directions to Sunset
Query By ExampleSimilarity MeasurementContent InterpretationUser FeedbackEtc.
Query By ExampleSimilarity MeasurementContent InterpretationUser FeedbackEtc.
Motivation and Challenges (Cont.)Motivation and Challenges (Cont.)Three Research issues in Multimedia Database Three Research issues in Multimedia Database Management System (3)Management System (3)
C. Multimedia System: integrated and robust system should be developed to address the nuances of multimedia data
Leads to investigation in developing an unique prototype to integrate the complex characteristics of multimedia data
PresentationAuthoringUser FeedbackData SecurityWeb-Based Multimedia Search
AgendaAgendaMotivation and ChallengesMotivation and ChallengesPrototype MDBMS SystemPrototype MDBMS SystemApplicationsApplications
Social NetworkingSocial NetworkingHighHigh--level Feature Extractionlevel Feature ExtractionSoccer Event DetectionSoccer Event DetectionStorm Surge Animation ModelStorm Surge Animation ModelFlorida Public Hurricane Loss ModelFlorida Public Hurricane Loss Model
Prototype MDBMS SystemPrototype MDBMS SystemMotivation: The distributed multimedia management system design requires a comprehensive and integrated framework, which
Employs distributed architecture for the large-scale multimedia management systemOffers a database modeling solution to manage the complicated multimedia databasesFacilitates content based retrievalIncorporates the innovative techniques to further refine the retrieval performanceDelivers appealing experiences on multimedia presentations to all the usersEnsures the security assurance for multimedia data access
Prototype MDBMS Prototype MDBMS System (Cont.)System (Cont.)
Client SideClient Side
Security Management Module
Security Management Module
Multimedia Browsing and Retrieval Module
Multimedia Browsing and Retrieval Module
Presentation Authoring and Rendering Module
Presentation Authoring and Rendering Module
Image RetrievalImage
Retrieval
Video BrowsingVideo
BrowsingVideo
RetrievalVideo
Retrieval
Online Media Retrieval
Online Media Retrieval
Environment Role ManageEnvironment Role Manage
User Roles Manage
User Roles Manage
Object Role Manage
Object Role Manage
Security Rule Manage
Security Rule Manage
Java Media Frame PlayerJava Media Frame Player
MATN Model Design
MATN Model Design
SMIL GeneratorSMIL
Generator
Web based Player
Web based Player
User Log InUser Log InAdministratorAdministratorGeneral UserGeneral User
Security CheckingSecurity Checking
Request HandlerRequest HandlerRequest PackagerRequest Packager Request SenderRequest Sender Information ReceiverInformation Receiver
Network LayerNetwork Layer TCP/IPTCP/IP UDPUDP RTPRTP HTTPHTTP ……
Server SideServer Side
Proposed Framework
Prototype MDBMS Prototype MDBMS System (Cont.)System (Cont.)
Network LayerNetwork Layer
Server SideServer Side
PostgreSQL: Multimedia Database Management (Relational Object Database )PostgreSQL: Multimedia Database Management (Relational Object Database )
Security Policy Rules
Environmental Roles
Object Roles
User Roles
Permission Roles
Affinity Relationships
Visual/Audio Features
Multimedia Meta Data
Segments and Objects
Processed Data
Videos
Audios
Images
Textual Data
Source DataMMM & HMMMMMM & HMMM XMLXML
Multimedia Search EngineMultimedia Search Engine
SMARXO Security ControlSMARXO Security Control
Image RetrievalImage Retrieval Video RetrievalVideo Retrieval Online LearningOnline Learning Video ClusteringVideo Clustering
Security Policy Checker
Security Policy Checker
Multimedia Data Processor
Multimedia Data Processor
Multimedia Data ManagerMultimedia Data Manager
Multimedia Data Analyzer
Multimedia Data Analyzer
Multimedia Data Supplier
Multimedia Data Supplier
Request HandlerRequest HandlerReceiverReceiver
Data SenderData Sender
Multimedia Database Indexing Modeling Multimedia Data Relationships
Offline LearningOffline Learning
Prototype MDBMS Prototype MDBMS System (Cont.)System (Cont.)
Multimedia Database Management ‐ PostgreSQLMultimedia Database Management ‐ PostgreSQL
Multimedia Database Indexing (AH‐tree)
Modeling Multimedia Data Relationships
Source Data + Processed Data (MMM & HMMM)Source Data + Processed Data (MMM & HMMM)
Modified PostgreSQL’s index mechanism to include AH‐TreeEfficient multimedia data retrieval in a commercial DBMSAllow nearest neighbor queries (KNN)
Modeling multimedia data relationships (Social networking)
Prototype MDBMS Prototype MDBMS System (Cont.)System (Cont.)
Modification of PostgreSQL’s indexing mechanismPostgreSQL’s index algorithm based on GIST was changed to have the functionalities of AH‐Tree
Added index key of AH‐TreeModified insert algorithmModified search algorithm to allow KNN queriesAdjusted the insert and search algorithms of AH‐Tree to be robust to concurrency
Prototype MDBMS Prototype MDBMS System (Cont.)System (Cont.)
Functionalities:
Content‐based image retrieval
Video browsing and retrieval
Presentation and authoring
Prototype MDBMS Prototype MDBMS System (Cont.)System (Cont.)Content Based Image Retrieval
Prototype MDBMS Prototype MDBMS System (Cont.)System (Cont.)Video Browsing and Retrieval
Prototype MDBMS Prototype MDBMS System (Cont.)System (Cont.)Presentation and Authoring
AgendaAgendaMotivation and ChallengesMotivation and ChallengesPrototype MDBMS SystemPrototype MDBMS SystemApplicationsApplications
Social NetworkingSocial NetworkingHighHigh--level Feature Extractionlevel Feature ExtractionStorm Surge Animation ModelStorm Surge Animation ModelFlorida Public Hurricane Loss ModelFlorida Public Hurricane Loss Model
Social NetworkingSocial NetworkingSocial networks are huge and growing
~ 35 million
~ 200 million
~ 12 million
Visualizing Social Data1
1. http://overstated.net/2009/03/09/maintained‐relationships‐on‐facebook
Social Networking (Cont.)Social Networking (Cont.)Visualizing Large Social Networks
IssuesIssues Solution approachesSolution approaches
Social Networking (Cont.)Social Networking (Cont.)Existing Solutions
Social Networking (Cont.)Social Networking (Cont.)Discussions
Issues of Clustered Graph representation
Social Networking (Cont.)Social Networking (Cont.)Proposed Approach
Social Networking (Cont.)Social Networking (Cont.)Generated Representative Graph
The lines between images represent the similarity relationship between them.
AgendaAgendaMotivation and ChallengesMotivation and ChallengesPrototype MDBMS SystemPrototype MDBMS SystemApplicationsApplications
Social NetworkingSocial Networking
HighHigh--level Feature Extractionlevel Feature ExtractionStorm Surge Animation ModelStorm Surge Animation ModelFlorida Public Hurricane Loss ModelFlorida Public Hurricane Loss Model
TRECVID HighTRECVID High--level Feature Extractionlevel Feature Extraction
TREC conferenceResearch in information retrievalTRECVID: independent track created in 2001
TRECVIDResearch in automatic segmentation, indexing, and content‐based retrievalLarge test collection (videos) and uniform scoring procedures
TRECVID HighTRECVID High--level Feature Extraction (Cont.)level Feature Extraction (Cont.)
TRECVID TasksHigh‐level feature extraction
High‐level semantic concept (e.g., people, indoor/outdoor, speech, etc) detection
Search (interactive, manually‐assisted, and/or fully automatic)Surveillance event detectionContent‐based copy detection
TRECVID HighTRECVID High--level Feature Extraction (Cont.)level Feature Extraction (Cont.)
High‐level Feature Extraction Automatically identify the occurrence of high‐level concepts (Indoor/Outdoor, People, Speech, etc) in test video dataData
Model‐development data: 200 hours of videoTest data: 180 hours of video
Multimedia Framework Feature ExtractionNormalization, DiscretizationSubspace AnalysisClassification
TRECVID HighTRECVID High--level Feature Extraction (Cont.)level Feature Extraction (Cont.)
Next Step:The techniques of the High‐level Feature Extraction task serve to help video searchSearch Task (interactive, manually‐assisted, and/or fully automatic)Apply features of our multimedia management system, such as event detection
AgendaAgendaMotivation and ChallengesMotivation and ChallengesPrototype MDBMS SystemPrototype MDBMS SystemApplicationsApplications
Social NetworkingSocial NetworkingHighHigh--level Feature Extractionlevel Feature Extraction
Storm Surge Animation ModelStorm Surge Animation ModelFlorida Public Hurricane Loss ModelFlorida Public Hurricane Loss Model
Storm Surge Animation ModelStorm Surge Animation ModelStorm Surge Animation Environment
Storm Surge Animation Model (Cont.)Storm Surge Animation Model (Cont.)Virtual Terrain Project
SimulatorSimulator
33‐‐dimensionaldimensional
InteractiveInteractive
OpenOpen‐‐sourcesource
EnginesEngines
Storm Surge Animation Model (Cont.)Storm Surge Animation Model (Cont.)LIDAR
AirborneAirborneLight detectionLight detectionTopographic dataTopographic dataAccurateAccurateHigh resolutionHigh resolution
Morphological Morphological filteringfilteringManMan‐‐made structuresmade structuresFootprintsFootprintsProcedural Procedural constructionconstruction
Storm Surge Animation Model (Cont.)Storm Surge Animation Model (Cont.)Engines
SkySky Wind & RainWind & Rain MarineMarine
VegetationVegetation TrafficTraffic Storm SurgeStorm Surge
Storm Surge Animation Model (Cont.)Storm Surge Animation Model (Cont.)Storm Surge Simulator
Simulate surge flooding effect by manipulating water mesh plane.Water height are taken from NOAA hourly numerical surge estimation.Displaying a new height every 10 seconds.
…1,3,6,7,8,5,4,8…
NOAA hourly water height estimation
Storm surge engine
Adjusting water mesh height
3
Time Engine
Synchronized sunlight
AgendaAgendaMotivation and ChallengesMotivation and ChallengesPrototype MDBMS SystemPrototype MDBMS SystemApplicationsApplications
Social NetworkingSocial NetworkingHighHigh--level Feature Extractionlevel Feature ExtractionStorm Surge Animation ModelStorm Surge Animation Model
Florida Public Hurricane Loss Florida Public Hurricane Loss ModelModel
Florida Public Hurricane Loss ModelFlorida Public Hurricane Loss Model
Introduction Florida Public Hurricane Loss Model is funded for $4.3 million by Florida Office of Insurance Regulation.Approved by Florida Commission on Hurricane Loss Projection Methodology in 2007, 2008, and 2009. The objective of the project is to develop and maintain a computer model to assess hurricane risk and to project insured losses due to the occurrence of hurricanes in Atlantic Basin.
Florida Public Hurricane Loss Model (Cont.)Florida Public Hurricane Loss Model (Cont.)FPHLM Framework
Wind ModelWind Model
Vulnerability Model
Vulnerability Model
Insured Loss Model
Insured Loss Model
Output
Met Data
Bldg Stock Data
Engr Data
Insurance Data
Terrain Data
GIS Data
Exposure Data
Statistical Program
Statistical Program
Com
puter Platform
Com
puter Platform
C++C++
JavaJava
IDLIDL
MatlabMatlab
ArcGISArcGIS
Oracle 9iOracle 9i
FortranFortran
DMIS (Distributed Multimedia Information DMIS (Distributed Multimedia Information Systems) LaboratorySystems) Laboratory
Currently 9 Ph.D. students and several undergraduate students 4 Ph.D. students graduated. Two obtained tenure track assistant professor positions at reputed US schools (U. of Alabama at Birmingham and U. of Montana). One works at eBay (first job after graduation) and another one works at State Street (Assistant Vice President), USA18 MS students graduated with placements at Microsoft, IBM etc.Research areas:
Distributed Multimedia Database Systems Multimedia Data Mining GIS and Remote Sensing
DMIS (Distributed Multimedia Information DMIS (Distributed Multimedia Information Systems) Laboratory (Cont.)Systems) Laboratory (Cont.)
Dr. Chen is the Editor-in-Chief of International Journal of Multimedia Data Engineering and Management (IJMDEM)
Most Active SMC Technical Committee Award, IEEE Systems, Man, and Cybernetics Society, October 2006Outstanding Contribution Award, IEEE Systems, Man, and Cybernetics Society, August 2005The Inaugural Florida International University Excellence in Graduate Mentorship Award, 20062004 Florida International University Outstanding Faculty Research AwardSCIS Faculty Research/Service/Mentorship AwardsSIRI Fellow
Best Paper Award, IEEE International Symposium on Multimedia (ISM2006), December 11-13, 2006, San Diego, CA, USA.
Dr. Shu-Ching Chen
e-mail: [email protected]
Website: http://www.cs.fiu.edu/~chens
Research Lab: http://dmis.cs.fiu.edu/
Dr. Dr. ShuShu--ChingChing ChenChen
e-mail: [email protected]
Website: http://www.cs.fiu.edu/~chens
Research Lab: http://dmis.cs.fiu.edu/