exploiting the gap between human and machine abilities in ...govind/amalia_dissertation.pdf ·...
TRANSCRIPT
Exploiting the Gap Between Human and Machine Abilities
in Handwriting Recognition for Web Security Applications
Amalia Rusu
Department of Computer Science and Engineering
University at Buffalo
2
Outline
� Objectives
� Motivation
� Background
� Previous Work
� Research Challenges
� Solutions
� Experiments and Results
� Research Ideas for Future Work
� Contributions
3
Objectives� Quantify human vs. machine ability in reading handwriting under noisy
conditions � Develop HCAPTCHAs-based HIP system on the ability gap between
humans and machines in handwriting recognition
The gap in the ability in recognizing handwritten text between humans and computers.
0
10
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6 7 8 9 10 11 12Transformation
Acc
ura
cy
WMR-4000
WMR-40000
Accuscript-4000
Accuscript-40000
CMR-4000
CMR-40000
Humans
4
Motivation
Why handwritten CAPTCHAs?
� Machine recognition of handwriting is more difficult than printed text
� Handwriting recognition is a task that humans perform easily andreliably
� Several machine printed text based CAPTCHAs have been already broken
� Speech/visual features based CAPTCHAs are impractical
� Handwritten CAPTCHAs thus far unexplored by the research community
5
BackgroundHuman Interactive Proofs (HIPs)
� New family of security protocols which allow a person to authenticate as belonging to a select group, for example human as opposed to machine, adult as opposed to a child, himself opposed to anyone else, etc.
� Operate over a network without the burden of passwords, biometrics, special mechanical aids, or special training
Example of HIP system interface to confirm user registration
Please enter the handwritten word as it is shown below:
If you cannot read this image click here
6
� CAPTCHAs are tests used by HIP systems � Completely Automatic Public Turing test to tell Computers and Humans
Apart – CAPTCHA
� CAPTCHA should be automatically generated and graded
� Tests should be taken quickly and easily by human users
� Tests should accept virtually all human users and reject software agents
� Tests should resist automatic attack for many yearsdespite the technology advances and prior knowledgeof algorithms
� Exploits the difference in abilities between humans and machines
(e.g., text, speech, facial features recognition, solving puzzles etc)
� A new formulation of the Alan Turing’s test - “Can machines think?”
BackgroundCAPTCHAs
7
� Suppressing SPAM and wormsOnly accept an email if I know there is a human behind the other computer. Prove you are
human before you can get a free email account.
� Search engine bootsThere is an html tag to prevent search engine bots from reading web pages; it only serves to say
"no bots, please“, but does not guarantee that bots won't enter a web site.
� Thwarting password guessing
Prevent a computer from being able to iterate through the entire space of passwords.
� Blocking denial-of-service attacksPrevent congestion based DoS attacks from denying any user’s access to web servers targeted
by those attacks.
� Preventing ballot stuffingCan the result of any online poll be trusted? Not unless the poll requires that only humans can vote.
� Protecting databasesI.e. eBay: protecting the data from auction portals that search across auction sites to provide listings and price information for their users, but prohibiting copying that data
BackgroundHIP and CAPTCHA Applications for Cyber Security
8Backgroundhttp://www.nytimes.com/2007/06/11/technology/11code.html?pagewanted=1&ei=5070&en=027dd9d726115b61&ex=1186804800&adxnnl=0&adxnnlx=1186650546-8DASjhLwzui2PWUBZqxLLg
“You can make a captcha absolutely undefeatable by computers, but at some point, you are turning this from a human reading test into an intelligence test and an acuity test,” said Michael Barrett, the chief information security officer at PayPal, a division of eBay.”
“Aleksey Kolupaev, works for an Internet company in Kiev, Ukraine, and in his spare time, with his friend Juriy Ogijenko, he develops and sells software that can thwart captchas by analyzing the images and separating the letters and numbers from the background noise. They charge $100 to $5,000 a project, depending on the complexity of the puzzle. Mr. Kolupaev said he had worked both for legitimate companies that want to test their own security and for spammers who seek to infiltrate Web sites.On his Web site, ocr-research.org.ua, Mr. Kolupaev boasts of cracking the captchas of companies like MySpace and PayPal; the site also ranks the effectiveness of each captcha. He says he believes that his work makes the Internet more secure because companies tend to improve the captchas that he critiques.Internet companies have responded to these challenges by making their captchas more complex. On YouTube, for example, the letters and numbers in the captcha float on an uneven grid of colors. On the technology news site Slashdot, random squiggly lines slice through the letters and numbers, as if a child had scrawled with a pen on each puzzle.All these tricks are attempts to disguise the boundaries of the characters, so that software cannot identify the numbers and letters. But often these measures prove too tough for humans to decipher as well. “
9BackgroundUsing CAPTCHAs to digitize books
Protect your site from abuse and help digitize books. Use reCAPTCHA on your site for free.http://recaptcha.net
“reCAPTCHA improves the process of digitizing books by sending words that cannot be read by computers to the Web in the form of CAPTCHAsfor humans to decipher. More specifically, each word that cannot be read correctly by OCR is placed on an image and used as a CAPTCHA.”
10
Outline
� Objectives
� Motivation
� Background
� Previous Work
� Research Challenges
� Solutions
� Experiments and Results
� Research Ideas for Future Work
� Contributions
11
Previous Work
AltaVista URL filter uses isolated random characters and digits on a cluttered background.
PessimalPrint uses a degradation model simulating physical defectscaused by copying and scanning of printed text.
BaffleText uses pronounceable character strings that are not in the English dictionary and render the character string using a font into an image (without physics-based degradations); then generate a mask image as shown above.
[Coates, Baird, Fateman 2001][Chew, Baird 2003]
[altavista.com/sites/addurl/newurl]
12
Previous Work
GimpyType 3 different English words appearing in the picture above.
EZ-Gimpy uses real English words.
Gimpy-R uses nonsense words.
Character morphing algorithm that transforms a string into its graphical form.
[Xu, Lipton, Essa, Sung, Zhu 2003]
[Carnegie Mellon University, www.captcha.net 2000]
13
Previous Work
PIXUses a large database of labeled images. All of these images are pictures of concrete objects (a horse, a table, a house, a flower, etc). The program picks an object at random, finds 4 random images of that object from its database, distorts them at random, presents them to the user and then asks the question "what are these pictures of?"
ECOSounds can be thought of as a sound version of Gimpy. The program picks a word or a sequence of numbers at random, renders the word or the numbers into a sound clip and distorts the clip. It then presents the distorted sound clip to its user and asks the user to type in the contents of the sound clip.
ARTiFACIALPer each user request, it automatically synthesizes an image with a distorted face embedded in a cluttered background. The user is asked to first find the face and then click on 6 points (4 eye corners and 2 mouth corners) on the face.
[Carnegie Mellon University, www.captcha.net 2000] [Rui, Liu 2003]
14Research ChallengesMain components that build up the structure of this dissertation
15
Research Challenges
I. Generation of random and infinitely many distinct handwritten CAPTCHAs
II. Identifying and exploiting the weaknesses of state-of-the-art handwriting recognizers
III. Controlling distortion - so that HCAPTCHAs are human readable (conform to Gestalt laws/Geon theory) but not machine readable
16
Outline
� Objectives
� Motivation
� Background
� Previous Work
� Research Challenges
� Solutions
� Experiments and Results
� Research Ideas for Future Work
� Contributions
17
Generation of random and infinite many distinct handwritten text images� Use handwritten word images that current recognizers cannot read
� Handwritten US city name images available from postal applications
� Collect new handwritten word samples
SolutionsChallenge I
� Create real (or nonsense) handwritten words and sentences by gluing isolated upper and lower case handwritten characters or word images
18
Generation of random and infinite many distinct handwritten text images� Use handwriting distorter for generating synthetic “human-like” samples
� Template-based synthetic handwritten characters generation� Synthetic text is generated by concatenating perturbed versions of the character
templates
Solutions Challenge I
Synthetic human-like word(s) samples.
[Rusu, Govindaraju 2007]
19
Solutions Challenge I
Tracing of Character Templates
The GUI of the tracing program.
A traced template for character x.
20
Solutions Challenge IGeneration of Synthetic Samples
Illustration of various nonlinear transformations performed individually: a) only ascent line variation, b) only x-line variation, c) only descent line variation, d) only text width variation, e) only shearing variation, and f) only baseline variation.
The finalized synthetic handwriting sample with varying width and thickness.
Perturbations of curve-defining points with various degrees of perturbation.
21Solutions Challenge IGeneration of Synthetic Samples
The GUI of the generator program.
22
[Xue, Govindaraju 2002]
[Kim, Govindaraju 1997]
[Favata 1996]
� Word Model Recognizer (WMR)� Character Model Recognizer (CMR)� Accuscript
1 2 3 4 5 6 7 8 9
w[7.6]
w[7.2]r[3.8]
w[5.0]
w[8.6]
o[7.6]r[6.3]
d[4.9]
w[5.0]
o[6.6]
o[6.0]
o[7.2]o[10.6] d[6.5]
d[4.4]
r[7.5]r[6.4]
o[7.8]r[8.6]
o[8.7]r[7.4]
r[7.6]
o[8.3]
o[7.7]r[5.8]
1 2 3 4 5 6 7 8 9
o[6.1]
Find the best way of accounting for characters ‘w’, ‘o’, ‘r’, ‘d’ buy consuming all segments 1 to 8 in the process
Distance between lexicon entry ‘word’ first character ‘w’ and the image between:- segments 1 and 4 is 5.0- segments 1 and 3 is 7.2- segments 1 and 2 is 7.6
Lexicon Driven Model
� segmentation-then-recognition scheme
� segments, isolates and recognizes the characters which make up the word
� uses modules for preprocessing, segmentation, character recognition and lexicon ranking
� grapheme-based recognizer
� extracts high-level structural features from characters such as loops, turns, junctions, arcs, without previous segmentation
� uses a stochastic finite state automata model based on the extracted features
� uses static lexicons in the recognition process
� lexicon driven approach
� chain code based image processing � pre-processing
� segmentation
� feature extraction
� dynamic matching
JunctionLoops
LoopTurns
End
End
Grapheme Based Model
SolutionsChallenge IIExploit the Source of Errors for State-of-the-art Handwriting Recognizers
4
56
7 82 3
1
1 32 4 5 6 7 8i[.8], l[.8] u[.5], v[.2]
w[.6], m[.3]
w[.7]
i[.7]u[.3]
m[.2]m[.1]
r[.4]
d[.8]o[.5]
-Image from 1 to 3 is a ‘u’ with 0.5 confidence-Image from segment 1 to 4 is a ‘w’ with 0.7 confidence-Image from segment 1 to 5 is a ‘w’ with 0.6 confidence and an ‘m’ with 0.3 confidence
Find the best path in graph from segment 1 to 8 : w o r d
23SolutionsChallenge II
Exploit the Source of Errors for State-of-the-art Handwriting Recognizers
Speed and accuracy of Handwriting Recognizers. Feature extraction time is excluded. Testing platform is an Ultra-SPARC [figure taken from Xue and Govindaraju, 2002].
66.4958.140.99466.5658.141.82720000
86.2975.380.08986.2975.380.1441000
94.0689.120.03194.1389.220.044100
98.7796.560.02198.7396.530.02710
Top 2Top 1Top 2Top 1
accuracytime (secs)
accuracytime (secs)
Grapheme ModelLexicon DrivenLexicon size
24
Source of Errors for State-of-the-art Handwriting Recognizers� Image quality
Background noise, printing surface, writing styles
SolutionsChallenge II
Add lines, grids, arcs, background noise, convolution masks and special filters
25
Source of Errors for State-of-the-art Handwriting Recognizers� Image features
Stroke width, slope, rotations
SolutionsChallenge II
Generate variable stroke width, slope, rotate, stretch, compress word
26
Source of Errors for State-of-the-art Handwriting Recognizers� Segmentation errors
Over-segmentation, merging, fragmentation, ligatures, scrawls
SolutionsChallenge II
Delete ligatures, use touching letters/digits, merge characters for over segmentation or to be unable to segment
27
Source of Errors for State-of-the-art Handwriting Recognizers
� Recognition errors
Confusion with
similar lexicon entries,
large lexicons
SolutionsChallenge II
Increase lexicon size and density; lexicon availability
T r u t h
W M R r e s u l t s
( T o p c h o i c e f i r s t )
A c c u s c r i p t r e s u l t s
( T o p c h o i c e f i r s t )
I m a g e
O r l a n d o o v l a n d o o v l a v d o o n l a n d o o r l a n o l o o r l a u d o o v i a n d o o r l a h d o a r l a n d o o r l a n d o
o v l a n a o
o l l a n d o o v l a n d o o r l a n o l o o r l a n d o
o v l a n a o o v l a v d o o n l a n d o o v i a n d o o r l a n d a a r l a n d o
L a c k a w a n n a l a c k a e v a n a l a c k a w a w a l a c k a w a u a l a c k o w a n a l a c k a w a n a l a c k a w a n n a
l a c k a w a r n a l a c k a w a n r a l a c k a m a m a l a c ta w a n a
l a c k a w a r n a l a c ta w a n a l a c k a w a r r a l a c k a w a w a l a c k a w a n a l a c k a w a u a l a c k a w a n n a
l a c k o w a n a l o c r a w a r a l a c k a w a n r a
C l a r e n c e c l a r l n c l c l a r l n c e c l a r e n c l c e a r e n c e c l a r e n c e
c b a r e n c e c l o r e n c e c l a h e n c e a a r e n c e c l a w c e
c l a ie n c e c l a r e n c e
c l a te n c e c l a r l n c e c e a r e n c e c l a v e n c e c l a r e n x e c l a s e n c e c l o r e n c e c l a ie x c e
B u f f a l o b u f f a i o b u f f a l o
b u t f a l o b u i f a l o b u f f r i o r u f f a l o b u l f a l o b u f i a l o b u e f a i o b u l l a l o
r u f f a l o b u f f a l o
b u f f r l o b u f f a i o b u f f r i o b u l f a l o b u i f a l o b u t f a l o b u e f a l o b u f i a l o
28
Postal applications / Address InterpretationCensus forms, medical forms reading Bank check reading
CAPTCHAHuman Interactive ProofsChallenge/Response protocolsCyber security applications
Processing of historical documents (original paper aged and deteriorated, palm leaf manuscripts)Information leakage through document redaction
SolutionsChallenge III Ensuring Human Recognition
29
� Gestalt psychology is based on the observation that we often experience things that are not a part of our simple sensations
� What we are seeing is an effect of the whole event, and is more than the sum of the parts (holistic approach)
� Organizing principles - Gestalt laws:� law of closure� law of similarity� law of proximity� law of symmetry� law of continuity� law of familiarity� figure and ground
� By no means restricted to perception only – that is just where they were first noticed� memory
OXXXXXX XOXXXXX XXOXXXX XXXOXXX XXXXOXX XXXXXOXXXXXXXO
**********
**********
**********
[ ] [ ] [ ]
SolutionsChallenge III
30
� Geon Theory of Pattern Recognition – recognition by components
� Geons are building blocks of perception
� Object recognition = perceiving Geons (geometrical ions)
Solutions Challenge III
a) Basic geons. b) Objects constructed from geons.
Object recognition is size invariant.
Object recognition is rotational invariant.
31
� Two important aspects of Geons� Edges
� Intersections
SolutionsChallenge III
Evidence of Geon Theory when objects are lacking some of their components: Recoverable vs. Non-recoverable objects.
32
Gestalt laws: law of proximity, symmetry, familiarity, continuity, background
Create horizontal or vertical overlaps
SolutionsChallenge III
Control Overlaps
For same word, smaller distance overlapsFor different words, bigger distance overlaps
33
Gestalt laws: law of closure, proximity, continuity, familiarity
Add occlusions by circles, rectangles, lines with random anglesEnsure small enough occlusions such that they do not hide letters completely
SolutionsChallenge III
Control Occlusions
34
Gestalt laws: law of closure, proximity, continuity
Add occlusions by waves from left to right on entire image, with various amplitudes / wavelength or rotate them by an angleChoose areas with more foreground pixels, on bottom part of the text image (not too low not to high)
Control Occlusions
SolutionsChallenge III
35
Gestalt laws: law of closure, proximity, continuity, background
Use empty letters, broken letters, edgy contour, fragmentationBreak characters such that general image processing techniques cannot reconstruct the original image
Control Letter Fragmentation
SolutionsChallenge III
36
Gestalt laws: law of background, familiarity
Add occlusion using the same pixels as the foreground pixels (black pixels), arcs, or lines, with various thickness
�Curved strokes could be confused with part of a character�Use asymmetric strokes such that the pattern cannot be learned
Control Extra Strokes - Waves
SolutionsChallenge III
37
Gestalt laws: law of background, familiarity
Add occlusion using the same pixels as the foreground pixels – black jawsExtra strokes are confused with character components when they are about the same size, thickness, and curvature as the handwritten characters
Control Extra Strokes - Jaws
SolutionsChallenge III
38
Gestalt laws: law of closure, proximity, continuity, background
Split the image in two parts on horizontal and displace the parts in opposite directionsLearn reasonable position for horizontal displacement, adjust and decide the range based on human/machine results
Control Word Displacement
SolutionsChallenge III
39
Gestalt laws: law of closure, proximity, continuity, background
Split the images in four parts, either by a vertical/horizontal line or by diagonals, and spread the parts apart – mosaic effectSymmetry in displacement helps image reconstruction for humans - find best range and apply accordingly
Control Word Displacement
SolutionsChallenge III
40
flip-flop
vertical mirror
horizontal mirror
Gestalt laws: memory, internal metrics, familiarity of letters
Change word orientation entirely, or the orientation for few letters onlyUse variable rotation, stretching, compressing
Control Letter/Word Orientation
SolutionsChallenge III
41
Human abilities rely on context – comprehension helps disambiguate uncertainty
Use the words in sentences
Other CAPTCHAs - Reading Comprehension
I see a on the wall.
The is a great source of information.
SolutionsChallenge III
42
Devanagari CAPTCHA
Other CAPTCHAs – CAPTCHA for other scripts
SolutionsChallenge III
Various transformations applied on Devanagari symbols: a) Displaced images, b) Mosaic images, c) Noisy images, d) Overlapped images, e) Varying horizontal stroke width, f) Varying vertical stroke width.
a. b. c.
d. e. f.
43
Other CAPTCHAs – graph-based CAPTCHA
SolutionsChallenge III
44
Methods to attack HCAPTCHAs
SolutionsChallenge III
Background noise that cannot be reverted. The truth words are: Los Angeles, Silver Creek, Young America.
Background noise that can be reverted. The truth words is: Wlsv.
Background noise
45
Methods to attack HCAPTCHAs
SolutionsChallenge III
a) Before pre-processing, b) After pre-processing. The truth words are: WSeneca, Young America.
Wave transformation
46
Methods to attack HCAPTCHAs
SolutionsChallenge III
Overlapping transformation with bad reverse: a) before pre-processing, b) after pre-processing. The truth words are: Matthews, Paso.
Overlapping
47
Input. � Original (randomly selected) handwritten image (existing US city name
image or synthetic word image with various length or sentence)� Lexicon containing the image’s truth wordOutput. � Handwritten CAPTCHA imageMethod. � Randomly choose a number of transformations � Randomly establish the transformations corresponding to the given number� If more than one transformation is chosen then
� A priori order is assigned to each transformation based on experimental results� Sort the list of chosen transformations based on their priori order and apply them
in sequence, so that the effect is cumulative
[Rusu, Govindaraju 2004]
SolutionsHandwritten CAPTCHA Generation Algorithm
48
Outline
� Objectives
� Motivation
� Background
� Previous Work
� Research Challenges
� Solutions
� Experiments and Results
� Research Ideas for Future Work
� Contributions
49
Experiments and ResultsHCAPTCHA-based HIP as a Challenge Response Protocol for Security Systems
� Initialization
� Handwritten CAPTCHA Challenge
� User Response
� Verification
Automatic Authentication Session for Web Services.
Internet
User
Authentication Server
Challenge
Response
User authentication
The user initiate the dialog and has to be authenticated by server
Internet
User
Authentication Server
Challenge
Response
User authentication
The user initiates the dialog and has to be authenticated by server
50
Experiments and Results
HCAPTCHA test online at URL: http://www.cedar.buffalo.edu/˜ air2/captcha/captcha.php
Handwriting-based HIP system: challenges and verification online.
51
� No risk of image repetition� Image generation completely automated: words, images and distortions
chosen at random
� The transformed images cannot be easily normalized or rendered noise free by present computer programs, although original images must be public knowledge
� Deformed images do not pose problems to humans� Human subjects succeeded on our test images
� Test against state-of-the-art: Word Model Recognizer, Character Model Recognizer, and Accuscript� HCAPTCHAs unbroken by state-of-the-art recognizers
Experiments and ResultsTesting and Evaluation
52
Handwritten US city name images that defeat both WMR and Accuscript recognizers.
Handwritten “nonsense” word images that defeat both WMR and Accuscript recognizers.
Experiments and Results
53
Low accuracy of handwriting recognizers vs. humans on a subset of test images .
3.19%
4.41%
Accuscript
12.04%Random Non-sense Words
9.28%City Names
WMRTest Images
82%
Humans
Accuracy
12
Number of
Students
0%
Accuscript
Accuracy
0%
WMR
Accuracy
15
Number of
Test Images
Low accuracy of handwriting recognizers for general image transformations. The lexicons are created so as to contain all the truths of test images. Total number of tested images is
4,127 for city names and 3,000 for non-sense words. (and so is the lexicon size)
[Rusu, Govindaraju 2004]
Experiments and Results
54
The accuracy of handwriting recognizers. A set of 4,100 images was tested for each kind of transformation using lexicons with size 4,000 and 40,000. [Rusu, Govindaraju 2005-6]
Experiments and Results
N/A
N/A
3.4%
4.9%
6.8%
5.7%
5.8%
25.6%
1.7%
3.1%
3.5%
0.3%
0.0%
1.2%
5.2%
4,000
CMR
N/A
N/A
2.1%
3.2%
5.1%
4.3%
4.2%
19.9%
1.3%
2.2%
2.4%
0.2%
0.0%
0.9%
3.8%
40,000
0.2%0.7%0.1%0.5%Flip-Flop
0.9%4.4%0.5%3.8%Overlap Different Words
0.4%2.4%3.6%12.9%Horizontal Overlap (Large)
0.6%2.9%10.7%24.4%Horizontal Overlap (Small)
3.9%12.6%14.4%27.9%Vertical Overlap
0.4%1.6%5.3%16.4%Black Waves
4.3%10.6%7.0%15.4%Occlusion by waves
17.4%32.3%20.3%35.9%Occlusion by circles
0.8%3.6%1.3%5.1%Jaws/Arcs
3.0%9.0%6.4%14.3%Mosaic
3.4%8.8%10.3%19.8%Displacement
0.0%0.0%0.2%0.5%High Fragmentation
0.2%0.2%0.0%0.0%Small Fragmentation
0.0%0.1%0.4%0.9%Empty Letters
2.5%6.4%5.7%12.7%All Transformations
40,0004,00040,0004,000Lexicon Size
AccuscriptWMRHW Recognizer
55
The accuracy of human readers. A word image is recognized correctly when all characters are recognized.
[Rusu, Govindaraju 2005]
Experiments and Results
65.2%89Horizontal Overlap (Large)
76.7%90Horizontal Overlap (Small)
87.5%88Vertical Overlap
80.0%90Black Waves
80.5%87Occlusion by waves
67.8%90Occlusion by circles
71.9%89Jaws/Arcs
74.4%90Mosaic
78.6%89Displacement
74.4%90High Fragmentation
73.9%88Small Fragmentation
82.0%89Empty Letters
76.1%1069All Transformations
AccuracyTested ImagesTransformations
Max. value
Min. value
56
Experiments and Results
AP 2AP 2 =18 =32
AP2
Parameterization of the difficulty levels of HCAPTCHAs� Determine image complexity
• Human visual perception is sensitive to contrast and border perception
• Perimetric complexity vs. effective image density� Perimetric complexity =
� Image density = the number of black pixels / the number of total pixels
� Test humans’ difficulty and recognizers’ accuracy vs. image complexity
57
Experiments and Results
0
2000
4000
6000
8000
10000
12000
14000
16000
18000
20000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Sample images
Per
imet
ric
Co
mp
lexi
ty
ArcJawsBlackWaveEmpty LettersSmall FragmentationHigh FragmentationHoriz Overlap SmallHoriz Overlap LargeMosaicVertical OverlapDisplacementOcclusion by CirclesOcclusion by Wave
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Sample images
Imag
e d
ensi
ty (b
lack
pix
els
/ all
pix
els)
ArcJawsBlackWaveEmpty LettersSmall FragmentationHigh FragmentationHoriz Overlap SmallHoriz Overlap LargeMosaicVertical OverlapDisplacementOcclusion by CirclesOcclusion by Wave
0
20
40
60
80
100
0 5000 10000 15000 20000
Perimetric Complexity
Hu
man
s R
eco
gniti
on A
ccu
racy
A snapshot of handwritten images and corresponding perimetric complexity (perimeter squared divided by area of black pixels). A snapshot of handwritten images and
corresponding image density (black pixels vs. total pixels).
Humans recognition accuracy vs. perimetriccomplexity as a percent of correct answers per bin (with a total range for perimetriccomplexity of 100 equal bins, complexity range 0-20000).
The gap between humans and machines in reading handwriting by category of distortions – Humans’ accuracy vs. machine accuracy for each transform
– Determine most effective transformations that increase the gap
[Rusu, Govindaraju 2006]
58
Outline
� Objectives
� Motivation
� Background
� Previous Work
� Research Challenges
� Solutions
� Experiments and Results
� Research Ideas for Future Work
� Contributions
59
Research Ideas for Future Work
� Develop a new authentication mechanism of e-mail addresses� In order to contact a person by e-mail, first receive back an alias e-mail address to use
instead of the original one, which has been rendered by the deformation algorithm
� Explore other instances of open pattern analysis problems to build CAPTCHAs - to benefit both online security and pattern recognition research� Recognizing engineering drawing, describe the weather in a scene, recognize a person in a
photo, count the same objects in an image, etc.
� Develop methods for computing a password/PIN assurance and similarly quantify and evaluate different kinds of CAPTCHAs
� Similarly investigate handwritten HIPs for enabling the detection of phishingattacks� Unlike traditional HIPs where the computer issues a challenge to the user over a network, in
this case the user issues a challenge to the server- The user authenticates the server and can detect which websites are spoofed in order to trick users
into revealing private information
60
Research Ideas for Future Work
� Explore other authentication schemes for individuals and groups – group membership that distinguish classes of people
� Use methodologies used to create HIPs for generating new graphical passwords schemes that incorporate the cognitive aspects of human reading and understanding
� Explore general aspects of security and biometric systems that identify people based on handwriting and CAPTCHAs� Handwriting recognition and biometrics share many similar techniques for image analysis, feature
extraction, segmentation, pattern classification and matching
� Create research bridges between fields who have forensics and information security in common: cryptography, biometrics, image analysis, secure processing, computer networks, legal and ethical issues, privacy, and more
61
Contributions
� Machine Learning and Pattern Recognition - Exploring human versus machine abilities in handwriting recognition will help researchers improve machine capabilities by incorporating the cognitive aspects of human reading
� Cognitive Science - Advancing the understanding of “how” humans read handwriting assisted by cognition
� Developing a novel HIP security protocol using handwriting recognition
� Empirically proving that handwritten HIPs are superior to currently used HIPs for cyber security applications
62� Securing Cyberspace using Handwriting Recognition and Gestalt Laws of Perception, with Venu Govindaraju, in preparation, to be submitted to International Journal on Document Analysis and Recognition (IJDAR).
� Synthetic Handwriting Generator for Cyber Security, with Uros Midic, and Venu Govindaraju, accepted, to appear in Proc. of 13th Conference of the International Graphonomics Society (IGS 2007).
� The Influence of Image Complexity on Handwriting Recognition, with Venu Govindaraju, in Proc. of 10th IAPR International Workshop on Frontiers of Handwriting Recognition (IWFHR 2006), IEEE Computer Society.
� Graph-based CAPTCHA: An application for Cyber Security, with Adrian Rusu, Christopher Clement and John Wanies. Poster at 17th Annual Student Research Poster Symposium & 60th Annual Eastern Colleges Science Conference, Saint Joseph's University Chapter of Sigma Xi, Philadelphia, PA, April 22, 2006 – received Outstanding Poster Presentation in Mathematics, Computer Science, Engineering, Geology, and Physics Posters.
� Tree-Based CAPTCHA, with John Wanies, Christopher Clement and Adrian Rusu. Poster at 9th Annual Rowan Science, Technology, Engineering, & Math (STEM) Student Research Symposium, Rowan University, Glassboro, NJ, April 21, 2006.
� A Human Interactive Proof Algorithm Using Handwriting Recognition, with Venu Govindaraju, Proc. of 8th International Conference on Document Analysis and Recognition (ICDAR 2005), IEEE Computer Society, ISBN 0-7695-2420-6, Vol. 2, pp. 967-971, 2005.
� Visual CAPTCHA with Handwritten Image Analysis, with Venu Govindaraju, Proc. of 2nd International Workshop on Human Interactive Proofs (HIP 2005), LNCS Vol. 3517, pp. 42-52, 2005.
� Challenges that handwritten text images pose to computers and new practical applications, with VenuGovindaraju. Proc. of IS&T/SPIE 17th Annual Symposium Electronic Imaging Science and Technology, Conference on Document Recognition and Retrieval XII, SPIE Vol. 5676, pp. 84-91, 2005.
� Handwriting Word Recognition: A New CAPTCHA Challenge, with Venu Govindaraju. Proc. of 5th International Conference on Knowledge Based Computer Systems, Allied Publishers Pvt. Ltd, pp. 347-357, 2004.
� Handwritten CAPTCHA: Using the Difference in the Abilities of Humans and Machines in Reading Handwritten Words, with Venu Govindaraju. Proc. of 9th IAPR International Workshop on Frontiers of Handwriting Recognition (IWFHR 2004), IEEE Computer Society, ISBN 0-7695-2187- 8, pp. 226-231, 2004.
63
Thank You