etiseo annotation rules
DESCRIPTION
ETISEO Annotation rules. Data structure Annotation tool and format Ground truth creation rules Reference data generation. Data structure - context. Context. Id Type (ex: zone) Name (ex: exit_zone) Polygon. Data structure - object. Physical object. Id Start time - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/1.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
ETISEO Annotation ETISEO Annotation rulesrules
•Data structure•Annotation tool and format•Ground truth creation rules•Reference data generation
![Page 2: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/2.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Data structure - Data structure - contextcontext
• Id• Type (ex: zone)• Name (ex: exit_zone)• Polygon
Context
![Page 3: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/3.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Data structure - objectData structure - object
• Id• Start time• Type (ex: person,
vehicle…)• Subtype (ex: car,
van…)• Info2D• Info3D
• Links• Speed• GT Nature• GT Status• GT Occluded (partially or
completely)• GT Connected with
Physical object
GT : specific to annotation purpose
![Page 4: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/4.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Data structure - eventData structure - event
• Id• Name• Start and end time• Status• Involved objects (physical object of interest
and contextual object)
Event
![Page 5: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/5.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
ETISEO Annotation ETISEO Annotation rulesrules
•Data structure•Annotation tool and format
•Ground truth creation rules•Reference data generation
![Page 6: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/6.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Annotation tool and Annotation tool and formatformat
• Database - Excel file• Context annotation - Viper-GT tool and format
Annotation delivered with video:
• Ground truth - Viper-GT tool and format• Reference data (computed from GT) – Specific
tool and Viper-GT format
Annotation delivered at the end of Etiseo:
![Page 7: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/7.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Annotation tool and Annotation tool and formatformat
![Page 8: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/8.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Annotation tool and Annotation tool and formatformat
Viper-GT format example<?xml version="1.0" encoding="UTF-8"?><viper xmlns="http://lamp.cfar.umd.edu/viper#" xmlns:data="http://lamp.cfar.umd.edu/viperdata#">
<config> <…></config><data>
<sourcefile filename="file:ETI-VS1-CR-4-C1.avi"><object framespan="1:100" id="4" name="physical_object">
<attribute name="nature"><data:svalue value="physical_object_of_interest"/>
</attribute><attribute name="type">
<data:svalue value="person"/></attribute><attribute name="info2D">
<data:bbox framespan="1:1" height="182" width="86" x="138" y="59"/><data:bbox framespan="2:2" height="183" width="86" x="139" y="59"/>
<data:bbox framespan="3:3" height="185" width="86" x="141" y="60"/> <data:bbox framespan="4:17" height="187" width="87" x="143" y="60"/> <…>
<data:bbox framespan="40:100" height="212" width="10" x="3" y="10"/></attribute><…>
</sourcefile></data>
</viper>
![Page 9: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/9.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
ETISEO Annotation ETISEO Annotation rulesrules
•Data structure•Annotation tool and format•Ground truth creation rules
•Reference data generation
![Page 10: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/10.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Ground truth creation Ground truth creation rulesrules
• Rules for objects: Contextual object Static physical object of interest Occlusion Artefacts Sudden illumination changes Held object Group - Crowd Crowded video clip Multi-view annotation
• Rules for events
![Page 11: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/11.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Ground truth creation – Ground truth creation – Rules for objectsRules for objects
Contextual object
• Annotated during its period of movement
![Page 12: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/12.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Ground truth creation Ground truth creation rulesrules
• Rules for objects: Contextual object Static physical object of interest Occlusion Artefacts Sudden illumination changes Held object Group - Crowd Crowded video clip Multi-view annotation
• Rules for events
![Page 13: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/13.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Ground truth creation – Ground truth creation – Rules for objectsRules for objects
Static physical object of interest
• Any physical object of interest is annotated on the whole video clip (ex: a parked vehicle)
First frame and last frame of a video clip
![Page 14: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/14.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Ground truth creation Ground truth creation rulesrules
• Rules for objects: Contextual object Static physical object of interest Occlusion Artefacts Sudden illumination changes Held object Group - Crowd Crowded video clip Multi-view annotation
• Rules for events
![Page 15: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/15.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Ground truth creation – Ground truth creation – Rules for objectsRules for objects
Occlusions – general rules
• Apparition of an object: the BBox increase with object apparition including only its visible part,
• All BBox are limited by the image boundary,• Partial and complete occlusion will be
differentiate in the evaluation (attribute “occluded”).
![Page 16: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/16.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Ground truth creation – Ground truth creation – Rules for objectsRules for objects
Static occlusions (by a contextual object)
• Partial occlusion: full BBox is drawn including hidden part,
![Page 17: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/17.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Ground truth creation – Ground truth creation – Rules for objectsRules for objects
Static occlusions (by a contextual object)
• Complete occlusion - two cases: all “escapes” of the contextual object are
visible: full BBox is drawn all the time,
![Page 18: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/18.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Ground truth creation – Ground truth creation – Rules for objectsRules for objects
all “escapes” of the contextual object are not visible: the object is no more annotated when completely occluded
![Page 19: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/19.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Ground truth creation – Ground truth creation – Rules for objectsRules for objects
• Partial or complete occlusion: in both cases, the full BBox is drawn including hidden part.
Dynamic occlusion (by another object)
![Page 20: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/20.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Ground truth creation – Ground truth creation – Rules for objectsRules for objects
Split object due to occlusion
• A unique BBox including all sub part is drawn.
![Page 21: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/21.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Ground truth creation Ground truth creation rulesrules
• Rules for objects: Contextual object Static physical object of interest Occlusion Artefacts Sudden illumination changes Held object Group - Crowd Crowded video clip Multi-view annotation
• Rules for events
![Page 22: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/22.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Ground truth creation – Ground truth creation – Rules for objectsRules for objects
Artefacts• Only the part constituting the objects forms the
BBox, artefacts (shadow, reflection…) are ignored for instant.
• Annotated in video data set 2
![Page 23: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/23.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Ground truth creation Ground truth creation rulesrules
• Rules for objects: Contextual object Static physical object of interest Occlusion Artefacts Sudden illumination changes Held object Group - Crowd Crowded video clip Multi-view annotation
• Rules for events
![Page 24: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/24.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Ground truth creation – Ground truth creation – Rules for objectsRules for objects
Sudden illumination changes (tagged frames)• Static object
• Object in movement: position interpolation
![Page 25: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/25.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Ground truth creation Ground truth creation rulesrules
• Rules for objects: Contextual object Static physical object of interest Occlusion Artefacts Sudden illumination changes Held object Group - Crowd Crowded video clip Multi-view annotation
• Rules for events
![Page 26: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/26.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Ground truth creation – Ground truth creation – Rules for objectsRules for objects
Held object
Example: bag-person, person-bike, door-vehicle
• The holder and the held object are both annotated (if visible),
• The carried object is connected to the holder.
![Page 27: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/27.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Ground truth creation – Ground truth creation – Rules for objectsRules for objects
Specific case
• Person getting in/out a vehicle: annotated only when visible
![Page 28: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/28.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Ground truth creation Ground truth creation rulesrules
• Rules for objects: Contextual object Static physical object of interest Occlusion Artefacts Sudden illumination changes Held object Group - Crowd Crowded video clip Multi-view annotation
• Rules for events
![Page 29: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/29.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Ground truth creation – Ground truth creation – Rules for objectsRules for objects
Group - Crowd• Group: multiple bounding boxes (persons +
group)
• Crowd : unique Bbox when individual annotation is impossible.
![Page 30: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/30.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Ground truth creation Ground truth creation rulesrules
• Rules for objects: Contextual object Static physical object of interest Occlusion Artefacts Sudden illumination changes Held object Group - Crowd Crowded video clip Multi-view annotation
• Rules for events
![Page 31: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/31.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Ground truth creation – Ground truth creation – Rules for objectsRules for objects
Crowded video clip• Only object involved in event of interest are
annotated. No false detections evaluation
![Page 32: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/32.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Ground truth creation Ground truth creation rulesrules
• Rules for objects: Contextual object Static physical object of interest Occlusion Artefacts Sudden illumination changes Held object Group - Crowd Crowded video clip Multi-view annotation
• Rules for events
![Page 33: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/33.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Ground truth creation – Ground truth creation – Rules for objectsRules for objects
Multi-view annotation
![Page 34: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/34.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Ground truth creation Ground truth creation rulesrules
• Rules for objects: Contextual object Static physical object of interest Occlusion Artefacts Sudden illumination changes Held object Group - Crowd Crowded video clip Multi-view annotation
• Rules for events
![Page 35: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/35.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Ground truth creation – Ground truth creation – Rules for eventsRules for events
• Id• Name• Start and end time• Status• Involved objects:
Id of the physical object of interest in GT files
Id of the contextual object in context information files
Event – data structure
![Page 36: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/36.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Ground truth creation – Ground truth creation – Rules for eventsRules for events
• Choice of event for each video clip in the event ontology => defined in the database xls file (provided with video)
Sequence Name ETI-VS1-RD-15
… … …
Content
Physical objects type Person, vehicle
Events of interest gets_in, gets_out, parked
![Page 37: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/37.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Ground truth creation – Ground truth creation – Rules for eventsRules for events
• Automatic annotation of events with quantitative constraints, example:
Related to kinematics properties: stopped, running…
Related to spatial properties: close_to, inside_zone…
• Manual annotation of other events: holding, pushing, waiting, gets_in/out,…
![Page 38: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/38.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Ground truth creation – Ground truth creation – Rules for eventsRules for events
Example
![Page 39: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/39.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
ETISEO Annotation ETISEO Annotation rulesrules
•Data structure•Annotation tool and format•Ground truth creation rules•Reference data generation
![Page 40: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/40.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
Reference data Reference data generationgeneration
• Filtering on object size• Object speed computation• Group generation• Multi-view and 3D reconstruction• Events with quantitative constraints
generation
![Page 41: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/41.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
ETISEO Annotation ETISEO Annotation rulesrules
Documentation:• ETI-Data structure and output format• ETI-Video Annotation Rules
![Page 42: ETISEO Annotation rules](https://reader036.vdocuments.net/reader036/viewer/2022062309/568151b5550346895dbfe308/html5/thumbnails/42.jpg)
INRETS, Villeneuve d’Ascq, December 15th-16th 2005
ETISEO Annotation ETISEO Annotation rulesrules
Questions …