encontra presentation
DESCRIPTION
TRANSCRIPT
![Page 1: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/1.jpg)
EnContRA Engine for Content-Based Retrieval Approaches
Ricardo José São Pedro Dias 07/02/2012
![Page 2: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/2.jpg)
Context
ColaDI Project
– Platform for Project Collaboration in Industrial Design
Period: March 2010 / November 2011 (EnContRA – until February 2011)
![Page 3: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/3.jpg)
Objectives
General Framework for (Content-Based) Retrieval Approaches and Applications
– Features:
• Indexing
• Features Extraction
• Searching / Retrieval Algorithms
• Extensible Query Processing
![Page 4: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/4.jpg)
Advantages for Development
1. Modularity
2. Easy to use – Low learning curve
3. Fast development of new approaches
– Examples:
• A new descriptor
• A new searching / retrieval algorithm
• A new indexing structure
• Etc.
![Page 5: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/5.jpg)
Multimedia Support
Support for different multimedia types
– Pictures
– Drawings
– 3D Objects
– Audio / Music
![Page 6: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/6.jpg)
Typical Application Architecture
![Page 7: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/7.jpg)
EnContRA Modules
![Page 8: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/8.jpg)
CREATING A SIMPLE APPROACH Indexing and Retrieving Pictures using QBE
![Page 9: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/9.jpg)
Objectives
Create a simple Query by Example Image Retrieval Application
![Page 10: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/10.jpg)
Data Model – Input Data
![Page 11: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/11.jpg)
Objective
Extracts: Scalable Color
Query
QBE
![Page 12: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/12.jpg)
Pieces to Assembly
1. Image Descriptor: Scalable Color
2. Indexing Structure: In Memory Simple Index
3. Searching Algorithm: Simple Searcher
![Page 13: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/13.jpg)
Pieces to Assembly
1. Image Descriptor: Scalable Color
2. Indexing Structure: In Memory Simple Index
3. Searching Algorithm: Simple Searcher
![Page 14: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/14.jpg)
Choosing a Feature to be extract
Scalable Color Descriptor (Extractor)
DescriptorExtractor extractor = new ScalableColorDescriptor<IndexedObject>();
Extractor Descriptor
![Page 15: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/15.jpg)
Pieces to Assembly
1. Image Descriptor: Scalable Color
2. Indexing Structure: In Memory Simple Index
3. Searching Algorithm: Simple Searcher
![Page 16: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/16.jpg)
Choosing an Indexing Structure
In Memory Simple Index
AbstractIndex index = new SimpleIndex();
Descriptor Descriptor
![Page 17: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/17.jpg)
Pieces to Assembly
1. Image Descriptor: Scalable Color
2. Indexing Structure: In Memory Simple Index
3. Searching Algorithm: Simple Searcher
![Page 18: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/18.jpg)
Searching
Linear Search (for now!)
Searcher searcher = new SimpleSearcher<IndexedObject>();
Descriptor Descriptor
![Page 19: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/19.jpg)
Assembling all the components
Searcher searcher = new SimpleSearcher<IndexedObject>();
searcher.setDescriptorExtractor(extractor);
searcher.setIndex(index);
searcher.setObjectStorage(new SimpleObjectStorage(IndexedObject.class));
searcher.setResultProvider(new DefaultResultProvider());
Setting Main Properties
Not required (recommended)
![Page 20: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/20.jpg)
Indexing the Dataset
File [] pictures = getFilePictures(dataset);
for (File pic : pictures) {
BufferedImage image = ImageIO.read(pic);
searcher.insert(new IndexedObject(image));
}
Extracts descriptors and indexes them!
![Page 21: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/21.jpg)
Performing a Search – Similar
//Load the image query
BufferedImage image = readQuery();
//Perform the search using similar
ResultSet<IndexedObject> results =
searcher.similar(new IndexedObject(image), 20);
//Print the Results
printResults(results);
![Page 22: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/22.jpg)
Performing a Search – Query API
CriteriaBuilderImpl cb = new CriteriaBuilderImpl();
Path<IndexedObject> modelPath = new Path<IndexedObject>(IndexedObject.class); Similar similar = cb.similar(modelPath, new IndexedObject(image)); CriteriaQuery query = cb.createQuery().where(similar).limit(20); ResultSet<StringObject> results = searcher.search(query);
Query Building
Searching
![Page 23: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/23.jpg)
A MORE COMPLEX APPROACH Indexing and Retrieving Pictures using QBE
![Page 24: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/24.jpg)
Objectives
Create a Drawing Retrieval Application, by employing Query By Example (or Sketch)
– Queries:
• 2D Drawings (e.g., SVG files)
• Pictures
![Page 25: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/25.jpg)
Input Model
Drawing Model
public class DrawingModel implements IEntity<Long> {
…
private Drawing drawing;
private BufferedImage image;
…
@Indexed
public BufferedImage getImage();
@Indexed
public Drawing getDrawing();
}
![Page 26: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/26.jpg)
Model to IndexedObject
Indexed Object Factory
Instance
Picture & Vectorial
Image Indexed Object
Drawing Indexed Object
CEDD IdxObj
Edge IdxObj
ColorL IdxObj
Drawing IdxObj
![Page 27: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/27.jpg)
Pieces to Assembly
1. Descriptor: Image Descriptors + TopoGeo
2. Indexing Structure: NBTree
3. Searching Algorithm: NBTree Searcher
![Page 28: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/28.jpg)
Pieces to Assembly
1. Descriptor: Image Descriptors + TopoGeo
2. Indexing Structure: NBTree
3. Searching Algorithm: NBTree Searcher
![Page 29: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/29.jpg)
Descriptors
Image Descriptors
DescriptorExtractor ceddExtractor = new CEDDDescriptor<IndexedObject>();
DescriptorExtractor edgeHistogram =
new EdgeHistogramDescriptor<IndexedObject>();
DescriptorExtractor colorLayout = new ColorLayoutDescriptor<IndexedObject>();
TopoGeo Descriptor
TopogeoDescriptorExtractor topogeoDescriptorExtractor =
new TopogeoDescriptorExtractor();
![Page 30: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/30.jpg)
Pieces to Assembly
1. Descriptor: Image Descriptors + TopoGeo
2. Indexing Structure: NBTree
3. Searching Algorithm: NBTree Searcher
![Page 31: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/31.jpg)
BTree for Indexing
Parameters:
– the name of the index
– the type of objects to be indexed (class)
BTreeIndex exampleIndex = new BTreeIndex(“btreeName", Object.class);
![Page 32: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/32.jpg)
Pieces to Assembly
1. Descriptor: Image Descriptors + TopoGeo
2. Indexing Structure: NBTree
3. Searching Algorithm: NBTree Searcher
![Page 33: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/33.jpg)
NBTree Searcher
Two flavors:
– Regular (original) AbstractSearcher searcher = new NBTreeSearcher();
– Parallel To speed the search AbstractSearcher searcher = new ParallelNBTreeSearcher();
![Page 34: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/34.jpg)
Picture – Composed Searching
AbstractSearcher imageSearcher = new ImageSearchEngine();
imageSearcher.setQueryProcessor(new QueryProcessorDefaultParallelImpl()); imageSearcher.setIndexedObjectFactory(new ImageIndexedObjectFactory()); //Creating a combined searcher, with the selected descriptor for (Map.Entry<String, DescriptorExtractor> entry : availableDescriptors) { AbstractSearcher entrySearcher = new ParallelNBTreeSearcher(); entrySearcher.setQueryProcessor(new QueryProcessorDefaultParallelImpl()); entrySearcher.setIndex(new BTreeIndex("image." + entry.descriptorName, objClass); entrySearcher.setDescriptorExtractor(entry.extractor); imageSearcher.setSearcher("image." + entry.descriptorName, entrySearcher); } searcher.setSearcher("image", imageSearcher);
![Page 35: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/35.jpg)
Drawing Searcher
AbstractSearcher vectSearcher = new ParallelNBTreeSearcher();
vectSearcher.setQueryProcessor( new QueryProcessorDefaultParallelImpl());
vectorialSearcher.setIndex(new BTreeIndex(“vectIndex",
TopogeoDescriptor.class));
TopogeoDescriptorExtractor topogeoExtractor = new TopogeoDescriptorExtractor();
vectorialSearcher.setDescriptorExtractor(topogeoExtractor);
searcher.setSearcher("drawing", vectSearcher);
![Page 36: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/36.jpg)
Performing a Search – Individual
CriteriaQuery<DrawingModel> query = cb.createQuery(DrawingModel.class);
Path<DrawingModel> modelPath = query.from(DrawingModel.class); Path drawingPath = modelPath.get(“drawing”); Similar similar = cb.similar(drawingPath, new IndexedObject(drawing)); CriteriaQuery query = cb.createQuery().where(similar).limit(20); ResultSet<StringObject> results = searcher.search(query);
Query Building
Searching
![Page 37: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/37.jpg)
Performing a Search – Combined
Similar simD = cb.similar(drawingPath, new IndexedObject(drawing)); Similar simP = cb.similar(picturePath, new IndexedObject(image)); And andPredicate = cb.and(simD, simP); CriteriaQuery query = cb.createQuery() .where(andPredicate ).limit(20); ResultSet<StringObject> results = searcher.search(query);
Query Building
Searching
![Page 38: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/38.jpg)
Custom IndexedObject Factory
public class ImageIndexedObjectFactory extends AnnotatedIndexedObjectFactory{
…
protected List<IndexedObject> createObjects(List<IndexedField>
indexedFields) {
//create indexedObjects for the DrawingModel instances
…
}
…
}
![Page 39: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/39.jpg)
Custom Image Searcher
public class ImageSearchEngine implements AbstractSearcher<Long> {
…
protected List<IndexedObject> getIndexedObjects(Object o) throws IndexingException {
//create different indexedObjects for the same image, and use //them in different individual searchers
}
public ResultSet search(Query query) {
//create subqueries to perform search in the individual image //searchers
}
…
}
![Page 40: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/40.jpg)
QUERY API Some features of the Query API
![Page 41: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/41.jpg)
Operators
• AND/ OR
• EQUAL / NOT EQUAL
• SIMILAR
• NOT
![Page 42: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/42.jpg)
Query Processors
• Cascade Processor
– Each sub-expression at a time
• Parallel Processor
– Optimization for sub-expressions like AND and OR
![Page 43: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/43.jpg)
DEMOS Some demos developed during the project
![Page 44: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/44.jpg)
Demos
Available at http://www.youtube.com/inevopt
Android Visual Search Image & Vectorial Search
![Page 45: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/45.jpg)
GETTING ENCONTRA How to get EnContRA, and more documentation and support?
![Page 46: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/46.jpg)
Checkout/Push Source Code
Checkout
git clone [email protected]:inevo/encontra.git
Commit and Push
git commit –m “+ Add: Texture Layout Descriptor.”
git push
http://schacon.github.com/git/gittutorial.html
![Page 47: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/47.jpg)
Contributing and Compiling Modules
mvn install full deploy (compile, package, run tests)
mvn package full deploy (compile, package)
mvn –DskipTests=true install full deploy (skip tests to
speed up)
http://maven.apache.org/
![Page 48: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/48.jpg)
Documentation & Support
• EnContRA 101 – Dev Tutorial (almost finished)
• Javadocs
• Source code
• People:
– Me [email protected]
– Tiago Cardoso [email protected]
– Nelson Silva [email protected]
![Page 49: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/49.jpg)
![Page 50: Encontra presentation](https://reader035.vdocuments.net/reader035/viewer/2022062615/5487c538b4af9faf0d8b5523/html5/thumbnails/50.jpg)
EnContRA Engine for Content-Based Retrieval Approaches
Ricardo José São Pedro Dias 07/02/2012
The End