Introducing

of handwritten icons

Ralph Niels, Don Willems and Louis Vuurpijl

August 20th, 2008ICFHR, Montreal

Introducing the NicIcon databaseof handwritten icons

Ralph Niels

Don Willems

Louis Vuurpijl

Icon

NicI

• Introduction• Domain: crisis management

• Data collection• Icon design• Method• Data

• Classification experiment• Method• Results

• Conclusion and URL

Overview

=

Introducing the NicIcon databaseof handwritten icons

Ralph Niels

Don Willems

Louis Vuurpijl

NEW!

Crisis management (CM)

• Scenario: tunnel disaster• Distributed computer systems to support CM• Multiple modalities: speech, gestures, and pen

Introducing the NicIcon databaseof handwritten icons

Ralph Niels

Don Willems

Louis Vuurpijl

The car is here

Iconic pen gestures

• Faster than handwriting• Easy to learn & remember• Visual meaningful shape

Introducing the NicIcon databaseof handwritten icons

Ralph Niels

Don Willems

Louis Vuurpijl

Database

• Public databases available for several handwriting applications

• Not for iconic pen gestures

• Based on symbology reference by USA Homeland Security Workgroup– Used in e.g., USA, Australia, New-Zealand

Introducing the NicIcon databaseof handwritten icons

Ralph Niels

Don Willems

Louis Vuurpijl

Icons

Introducing the NicIcon databaseof handwritten icons

Ralph Niels

Don Willems

Louis Vuurpijl

Flood Accident Car Bomb Roadblock

Electricity Casualty Fire Fire brigade Police

Injury Gas Paramedics Person

Data collection

• 35 participants• Online and offline• Variation in size• Per person:

– 22 pages– 55 instances / icon

Introducing the NicIcon databaseof handwritten icons

Ralph Niels

Don Willems

Louis Vuurpijl

Database content

• Online: 26,163 iconic gestures(24,144 reported in paper)

Introducing the NicIcon databaseof handwritten icons

Ralph Niels

Don Willems

Louis Vuurpijl

Tablet: Wacom Intuos2 A4 oversize

Database content

• Offline: 770 scanned pages

Introducing the NicIcon databaseof handwritten icons

Ralph Niels

Don Willems

Louis Vuurpijl

Scanner: HP Scanjet 7400C flatbed(@ 300dpi, 24 bit colour)

Classification experiment

Data sub sets• Stratified

• Writer dependent (WD), writer independent (WI)

Complete data set(26,163 icons)

Evaluation set(40%)

Train set(36%)

Test set (24%)

WDEvaluation set

(40%)

WDTrain set

(36%)

WDTest set (24%)

WIEvaluation set

(40%)

WITrain set

(36%)

WITest set (24%)

Introducing the NicIcon databaseof handwritten icons

Ralph Niels

Don Willems

Louis Vuurpijl

Classification experiment

Multi classifier system• Feature sets

• 28 geometric features (Willems & Vuurpijl)

• 30/60 coordinates running features (Schomaker & Vuurpijl)

• 1185 features (Willems, Niels, Van Gerven, Vuurpijl)

• Feature classifiers• Support Vector Machine• Multi-Layered Perceptron

• Template matching• Dynamic Time Warping (Niels & Vuurpijl)

Introducing the NicIcon databaseof handwritten icons

Ralph Niels

Don Willems

Louis Vuurpijl

NEW!

on online data

New feature set (‘m-fs’)

• Features from literature– Geometrical, temporal, pressure

Introducing the NicIcon databaseof handwritten icons

Ralph Niels

Don Willems

Louis Vuurpijl

D. Rubine, Specifying gestures by example, Computer Graphics. 25 (4) (1991) 329–337.

J. LaViola Jr. & R. Zeleznik, A practical approach for writer-dependent symbol recognition using a writer-independent symbol recognizer, IEEE Transactions on pattern analysis and machine intelligence. 29 (11) (2007) 1917–1926.

L. Zhang & Z. Sun, An experimental comparison of machine learning for adaptive sketch recognition, Applied Mathematics and Computation. 185 (2) (2007) 1138–1148.

and many others…

New feature set (‘m-fs’)

• Over complete icon, but also…– Mean over strokes– Std. dev. over strokes

Introducing the NicIcon databaseof handwritten icons

Ralph Niels

Don Willems

Louis Vuurpijl

Feature definitions in technical report at website

New feature set (‘m-fs’)

• Feature selection– 1185 features, which are the best?– Sort them on best individual performance– Add 1 by 1 to classifier

• Performance maximizes at:– 545 features for WI– 660 features for WD

• Selected features:– ± 1/3 full icon– ± 1/3 mean– ± 1/3 standard deviation

Introducing the NicIcon databaseof handwritten icons

Ralph Niels

Don Willems

Louis Vuurpijl

The best features

Breaking news: the best features

Area Sine first /last sample

Length ofdiagonal

Verticaloffsets

Averagecentroidal

radius

Introducing the NicIcon databaseof handwritten icons

Ralph Niels

Don Willems

Louis Vuurpijl

Classification results

Classifier Features WD WI

SVM g-28 84.8 73.0

m-fs 99.3 96.4

af-30 98.1 90.3

af-60 98.0 98.6

MLP g-28 84.0 78.5

m-fs 98.7 96.3

af-30 94.5 85.2

af-60 94.4 85.1

DTW - 98.4 93.6

MCS - 99.51 97.83

g-28: 28 geometric features(Willems & Vuurpijl)

m-fs: 1185 features(Willems, Niels, Van Gerven, Vuurpijl)

af-30/af-60: 30/60 running features (Schomaker & Vuurpijl)

DTW: Dynamic Time Warping(Niels & Vuurpijl)

MCS: Multiple classifier system (majority voting)

Introducing the NicIcon databaseof handwritten icons

Ralph Niels

Don Willems

Louis Vuurpijl

NEW!

Misclassifications

• Wrong box

• Sloppy drawing

• Retracing

Introducing the NicIcon databaseof handwritten icons

Ralph Niels

Don Willems

Louis Vuurpijl

Discussion

• Results already quite good (WD: 99.5%, WI: 97.8%),but gain is still possible

• Which features are important?– For different domains

• Performance in interactive experiment• Offline data still open• Mapping online -> offline

Introducing the NicIcon databaseof handwritten icons

Ralph Niels

Don Willems

Louis Vuurpijl

http://unipen.nici.ru.nl/NicIcon

Introducing the NicIcon databaseof handwritten icons

Ralph Niels

Don Willems

Louis Vuurpijl

Freely available:• Online (Unipen)• Offline (PNG)• Technical report about

new feature set