cubrik at smila conference in berlin
DESCRIPTION
CUbRIK project extensions to basic SMILA Framework, presented in Berlin on May 15, 2012TRANSCRIPT
Human-enhanced Multimedia Processing
in CuBRIK with SMILA
Alessandro Bozzon, Ph.d.
Politecnico di Milano mail: [email protected]: aleboz
Human-enhanced Multimedia Processing
in CuBRIK with SMILA
Alessandro Bozzon, Ph.d.
Politecnico di Milano mail: [email protected]: aleboz
The CUbRIK project
36 month large-scale integrating project
partially funded by the European Commission’s 7th Framework ICT Programme for Research and Technological Development
www.cubrikproject.eu
04/13/2023 SMILA Themenkonferenz 3
Objectives
The technical goal of CUbRIK is to build an open search platform grounded on four objectives: Advance the architecture of multimedia search Place humans in the loop Open the search box Start up a search business ecosystem
04/13/2023 SMILA Themenkonferenz 4
Objective: Advance the architecture of multimedia search
Multimedia search: coordinated result of three main processes: Content processing: acquisition, analysis,
indexing and knowledge extraction from multimedia content
Query processing: derivation of an information need from a user and production of a sensible response
Feedback processing: quality feedback on the appropriateness of search results
04/13/2023 SMILA Themenkonferenz 5
Objective: Advance the architecture of multimedia search
Objective: Content processing, query processing and
feedback processing phases will be implemented by means of independent components
Components are organized in pipelines Each application defines ad-hoc pipelines that
provide unique multimedia search capabilities in that scenario
04/13/2023 SMILA Themenkonferenz 6
CUbRIK architecture
04/13/2023 SMILA Themenkonferenz 7
6 March 2012 The CUbRIK Project is .... 8
SMILA is the backbone of CUbRIK
CUbRIK makes use of SMILA framework as a start-up service engine for supporting workflow definition and execution
Provides architectural extensions to SMILA for enhanced services:
Extensible content, query and feedback processing search workflow Multimodality, Orchestration of human and machine computation
tasks in all search processes Time and Space Awareness Support for social and human computation Persistency and Caching of content and metadata Support of federated configurations across a distributed architecture Different styles of User Interface for queries and presentation of
search results Includes tools and methods for application design
Objective: Humans in the loop
Problem: the uncertainty of analysis algorithms leads to low confidence results and conflicting opinions on automatically extracted features
Solution: humans have superior capacity for understanding the content of audiovisual material
State of the art: humans replace automatic feature extraction processes (human annotations)
Our contribution: integration of human judgment and algorithms
Goal: improve the performance of multimedia content processing
904/13/2023 SMILA Themenkonferenz 9
Example of CUbRIK Human-enhanced computation: Trademark Logo Detection
Problem statement: identifying occurrences of trademark logos in a video collection through keyword-based queries
Special case of the classic problem of object recognition
Use case: a professional user wants to retrieve all the occurrences of logos in a large collection of video clips
Applications: rating effectiveness of advertising, subliminal advertising detection, automatic annotation, trademark violation detection
1010
Human-powered trademark logo detection demo
Goal: integrate human and automatic computation to increase precision and recall w.r.t. fully automatic solutions
04/13/2023 SMILA Themenkonferenz 11
Problems in automatic logo detection: Object recognition is affected by the quality of the
input set of images
Uncertain matches, i.e., the ones with low matching score, could not contain the searched logo
Trademark Logo Detection: problems in automatic logo detection
12
Contribution in human computation Filter the input logos, eliminating the irrelevant
ones Segment the input logos
Validate the matching results
13
Trademark Logo Detection: contribution of human computation
14
Trademark Logo Detection: pipeline
14
The CrowdSearcher framework for HC task management
15
15
CrowdSearch framework in the Logo detection application
16
Types of tasks• Automatic tasks• Crowd tasks: tasks that are executed
by an open-ended community of performers
16
Community of Performers
17
The application is deployed as a Facebook application
Seed community Information Technology department of Politecnico di Milano
Task propagationEach user in the seed community can propagate tasks through the social networks
17
Design of “Validate Logo Images”
The “LIKE” task variant requires to choose relevant logos among a set of not filtered images
The “ADD”task variant requires to add new relevant image URLs
Please add new relevant logos
URL…
Send
18
People to task matching & Task Assignment
19
Execution criteriaConstraints of task execution
Time budget for the experiment
Content Affinity criteriaQuery on a representation of the users’ capacities• Current state: manual selection of users• Future work: Geocultural affinityQuestions are dispatched to the crowd according to the user experience in answering questions• Expert user: an user that has already
answered to three questions
New users answer to “LIKE” questions
Expert users answer to “LIKE”+“ADD” questions
19
Task propagation
Propagation over the Facebook graph: Platform: CrowdSearcher
Automatic task generation starting from a set of design criteria (e.g., question type, public/private…)
Seed community: Information Technology department of Politecnico di Milano
Each user in the seed community can propagate tasks through the social networks
Work in progress: Twitter/LinkedIn tasks Task assignment according to expertise, geocultural
information, past work history
04/13/2023 CUbRIK Pipelines 1 20
Task execution
21
“LIKE” task variant “ADD” task variant
21
Output aggregation
22
“LIKE” task variantsTop-5 rated logos are selected as relevant logos
“ADD” task variantsNew images are fed back to the LIKE tasks
22
Experimental evaluation
Three experimental settings: No human intervention Logo validation performed by two domain
experts Inclusion of the actual crowd knowledge
Crowd involvement 40 people involved 50 task instances generated 70 collected answers
23
Experimental evaluation
24
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
No Crowd
ExpertsCrowd
No Crowd
Experts
CrowdNo Crowd
Experts
CrowdAleveChunkyShout
Precision
Reca
ll
Experimental evaluation
25
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
No Crowd
ExpertsCrowd
No Crowd
Experts
CrowdNo Crowd
Experts
CrowdAleveChunkyShout
Precision
Reca
ll
Precision decreases
Reasons for the wrong inclusion• Geographical location of the
users• Expertise of the involved users
Experimental evaluation
26
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
No Crowd
ExpertsCrowd
No Crowd
Experts
CrowdNo Crowd
Experts
CrowdAleveChunkyShout
Precision
Reca
ll
Precision decreases• Similarity between two
logos in the data set
Crowdsourced filtering of logos – Problem concept
04/13/2023 CUbRIK Pipelines 1 27
Google Images
Filter Tasks
Filtered logos
Added logosAdd
Tasks
Integration in SMILA
The demo has been integrated into the SMILA architecture
Two main parts: Indexing part: made of asynchronous
components (in a SMILA sense) Indexing of videos Matching phase Interaction with the crowd
Search part: end users query the system by keyword-based queries
04/13/2023 CUbRIK Pipelines 1 28
Integration in SMILA
04/13/2023 CUbRIK Pipelines 1 29
Integration in SMILA: Indexing part overview
04/13/2023 CUbRIK Pipelines 1 30
Reusable components
04/13/2023 CUbRIK Pipelines 1 31
Crawling Google Images + Flickr crawler
Multimedia processing SIFT-based low level feature extraction Video segmentation component Key-frame extractor Robust low level feature matching component
Data storage “Data service” for referencing multimedia
resources
Integration in SMILA: Indexing part – Job1, Input images retrieval
04/13/2023 CUbRIK Pipelines 1 32
Integration in SMILA: Indexing part – Job2, Logo collection indexing
04/13/2023 CUbRIK Pipelines 1 33
Integration in SMILA: Indexing part – Job3, video collection indexing
04/13/2023 CUbRIK Pipelines 1 34
Integration in SMILA: Indexing part – Job4, matching phase
04/13/2023 CUbRIK Pipelines 1 35
Integration in SMILA: Indexing part – Job5, matches filtering
04/13/2023 CUbRIK Pipelines 1 36
Demo: Search interface
04/13/2023 CUbRIK Pipelines 1 37
Demo: Search interface
04/13/2023 CUbRIK Pipelines 1 38
Demo: Search interface
04/13/2023 CUbRIK Pipelines 1 39
Indexed logos thatmatch against the
video collection
Demo: Search interface
04/13/2023 CUbRIK Pipelines 1 40
Video preview
Demo: Search interface
04/13/2023 CUbRIK Pipelines 1 41
High confidence matches
Demo: Search interface
04/13/2023 CUbRIK Pipelines 1 42
Low confidence matches
6 March 2012 The CUbRIK Project is .... 43
CUbRIK Showcases
CUbRIK will showcase its technology with Demonstrators of examples of innovation in two domains: (Digital Libraries) History of Europe (Business Processes) CUbRIK search for SMEs,
Technical evaluation in real-world conditions including users will be based on these Demonstrators
Thanks for your attentionwww.cubrikproject.eu
04/13/2023 SMILA Themenkonferenz 44