Hsuan He Fang

3D Multi-Touch System by Using

Coded Optical Barrier on Embedded Photo-Sensors

Hsuan-He Fang* Institute of Electro-Optical Engineering, National Chiao Tung University, Hsinchu 30010, Taiwan

Guo-Zhen Wang Department of Electronics Engineering & Institute of Electronics,

National Chiao Tung University, Hsinchu 30010, Taiwan

Chia-Wei Chang Institute of Electro-Optical Engineering, National Chiao Tung University, Hsinchu 30010, Taiwan

Yi-Pai Huang Display Institute, National Chiao Tung University, Hsinchu 30010, Taiwan

Responding Author*－Tel : +886-3-5712121 ext. 59210, E-mail: mikker.eo01g@nctu.edu.tw

Abstract

Limited by construction complexity, bare finger touch systems are not ready to be implemented on

current mobile devices. Hence, we proposed a system using coded optical barrier with less hardware

and software complexity; based on the construction, touch algorithm is programed to obtained 3D

location (x,y,z) of input(s). Finally, our concept was implemented on a 4-inch panel. The system was

able to sense up to 3 touch inputs simultaneously within 35 mm working range with multi-touch

applicable.

Keywords－3D touch, near-distance touch, in-cell photo sensor, depth sensing, bare finger

(a) Presentation Style: Oral preference

(b) Topical Section: Touch and Interactive Displays / Novel Touch Configurations and Applications

(c) The first author (Hsuan-He Fang) who will be the presenter is currently a master student.

Hsuan-He Fang

3D Multi-Touch System by Using Coded Optical Barrier on Embedded Photo-Sensors

Hsuan-He Fang1, Guo-Zhen Wang2, Chia-Wei Chang1 and Yi-Pai Huang3

1Institute of Electro-Optical Engineering, National Chiao Tung University, Hsinchu 30010, Taiwan

2 Department of Electronics Engineering & Institute of Electronics,

National Chiao Tung University, Hsinchu 30010, Taiwan 3Display Institute, National Chiao Tung University, Hsinchu 30010, Taiwan

Abstract Limited by construction complexity, bare finger touch systems

are not ready to be implemented on current mobile devices.

Hence, we proposed a system using coded optical barrier with

less hardware and software complexity; based on the

construction, touch algorithm is programed to obtained 3D

location (x,y,z) of input(s). Finally, our concept was

implemented on a 4-inch panel. The system was able to sense up

to 3 touch inputs simultaneously within 35 mm working range

with multi-touch applicable.

1. Background 3D touch is a platform that user performs natural gestures to

manipulate virtual 3D images. In general, 3D touch systems can

be separated into two classes by their working range. The first

class, which is based on machine [1] and camera constructions

[2], works in far-distance environment; it is already well-applied

on TV game consoles. However, their construction limitations,

such as additional device and blind range issues, restrict the

feasibility to be implemented on mobile devices. Therefore, the

second class, 3D touch in near-distance should be established.

The most promising candidate is in-cell photo sensor

touchscreen [3]. However, it is not ready for 3D touch yet owing

to low sensitivity. To conquer the issue, extra optical designs

should be constructed. They can be diversified into two types.

First, lighting mode [4] system, where user holds patterned light

pen to interact. However, because of the extra light source,

systems of lighting mode are considered inconvenient. Second,

reflecting mode system, where ideally light source can be

integrated with display so that user can perform nature gestures;

it’s hence more user-friendly. Nevertheless, no one is perfect by

far; new construction with new algorithm should be established.

2. Prior Approaches and Objectives In a reflecting mode system, ideally, user can naturally

manipulate computer with bare hand; so it’s also named bare

finger touch. Listed below are three prior systems but still

needed to be improved. The limitation and comparison is

described in Figure 1. First, 3D finger touch with sequential

illuminator [5] was proposed by M.C. Ma et al. It used

sequential light at different tilt angles to spot the touch point.

However, in the construction, slow frame rate was induced due

to capturing sequence. In addition, complex lateral light source

made it difficult to be realized on products. Second, a system

LCD with integrated 3-dimensional input device [6] was

proposed by C. Brown et al. Their key contribution is to propose

a construction of directional sensor. It analyzed disparity in the

sensor to obtain depth value. However, owing to the sensor

design, working range was too restricted for practical use. In

addition, constrained by the construction, the aforementioned

two systems were unable to support multi-touch. Third, an LCD

display system with depth-sensing capability based on coded

aperture imaging [7] was proposed by S. Suh et al. The system

uses dynamic lensless imaging system to enlarge working range.

Nevertheless, in the construction, switches between display

mode and touch mode would reduce frame rate and cause image

flickering. Moreover, processing numbers of images accounts

for huge computation.

Briefly sum up, bare finger touch system using in-cell photo

sensor is still needed to be optimized, which should support

sufficient working range, reduced construction complexity and

less computation loading. Therefore, a new construction which

uses coded optical barrier on photo-sensors and new depth

sensing algorithm are present.

Figure 1. Comparison chart among bare finger touch systems (*proposed system)

Hsuan He Fang

3. System Architecture The present paper designed depth sensors to achieve near-

distance touch. Based on traditional LC display, there are three

main components. First, IR light is integrated with display

backlight system, acting as touch light source. With IR light

source, display image would not be affected. Second, IR photo

sensor is embedded on the TFT substrate as common in-cell sensor. Third and the most distinct portion of the whole system

is optical barrier. Optical barrier is located above and patterned

for each pixel of photo sensor. With optical barrier on top,

optical sensor becomes depth-sensitive and it will be detailed in

next paragraph. Besides, depth sensors are at black matrix

region, so aperture ratio would not be affected much and sensor

would not be influenced by backlight system.

In depth sensor, we designed an aperture above each photo

sensor, as depicted in Figure 2. Aside from fixed pitch of LC

cell ( ) in the panel, all we design is displacement between the

sensor and the aperture ( ). More to our concept, we classified

depth sensor into two types: xy-sensor and multiple z-sensors. In

xy-sensor, the aperture is directly located above the photo sensor

(i.e. ). That is, if an input is above the panel, image

captured by xy-sensor is for the system to locate 2D coordinate

( ). On the other hand in z-sensor, aperture is slightly shifted

with displacement ( ) so that the sensor is able to capture

directional light input. Moreover, we further devise the

displacement ( ) and design spacing between xy and z-sensor

( ) to be proportional (i.e. ⁄ ⁄ ). Hence the z-sensor is

able to sense the input at specific depth ( ) directly above the

corresponding xy-sensor. Nonetheless, multiple z-sensors are

around the xy-sensor, and the displacement ( ) of each z-sensor

is different in order to sense the input at different depth ( ).

Moreover, as an example depicted in Figure 2, -sensor can

capture strong signal while -sensor capture weak signal.

Generally, captured intensity of each z-sensor is highly related

to the depth (z) of input; hence a data base, which records

captured intensity and known depth, is constructed for depth-

sensing. Finally according to the database, a continuous working

range is constructed.

Figure 2. Depth-sensing principle and system architecture

4. Algorithm Touch algorithm is designed to analyze captured images to yield

3D position (x,y,z) of each touch points; it is mainly composed

of image capturing, 2D coordinate location and depth sensing, as

illustrated in Figure 3. (A) In capturing procedure. IR touch

backlight is emitted, and sensors capture distributed light

reflected by inputs; hence, one xy-image and multiple z-images

are obtained. Following, noise suppression is conducted on all

the images to reduce background noises, Gaussian noises and

panel defects, etc. (B) Followed by the 2D coordinate location

procedure. The xy-image is input and transformed into black and

white image. Location (x and y) of white pixel are regarded as

features. ISODATA [8] clustering cluster these features by

minimum distance principle. That is physically, white pixels in

adjacent positions can be grouped together as touch point(s); it

eventually separates and renders the geometrical center(s) (x,y)

of touch point(s). (C) After, 2D coordinates (x,y) are fed into the

depth-sensing engine. For each touch point, z-sensors at

corresponding location are referenced and intensity values are

extracted. Following, the intensity values ( ) of every -

sensors are input to a maximum likelihood trained model to

render depth value (z).

Figure 3. Touch algorithm

In addition, the trained model is constructed from the intensity-

depth data base which records intensity ( ) captured by -

sensors at known step of depth discretely. Following in touch

mode, we calculate the difference between and , then

retrieve probability ( ) of each -sensor at step in

Gaussian model, equation (1). Following, the probability values

are multiplied among -sensors to render final probability

values ( ) of the input at each step of depth, equation (2);

then they are normalized to be the features ( ) for training,

equation (3). Finally, a trained model with maximum likelihood

[9] weighting ( ) is then executed to render depth value ( ),

equation (4).

: number of z-sensors : number of depth steps

: standard deviation of intensity captured by -sensor in

trained model

( )

(

( )

) (1)

(2)

(3)

∑ (4)

Hsuan-He Fang

5. Experiments To test the feasibility of depth sensing, a prototype was bulit on

a 4-inch panel with in-cell photo sensor. Unfortunitly, there was

no optical barrier in the panel, so we had to attach home-made

mask on the cover glass, as shown in Figure 4. Consequently,

the spacing between sensor layer and barrier (d) was increased,

and resolution of the depth sensor was decreased. Moreover, in

the prototype, we designed one xy-sensor surrounded by four z-

sensor to construct a 50 mm working range, which was

sufficient for mobile devices. Futhermore about the prototye, the

backlight system was not implemented, we used flash light with

diffuser in place of bare finger. In addtion, limited by driver IC,

we could only obtained 1 captured image with xy- and z-images

included rather than separate images.

Figure 4. Experimental setup

5.1. Z-sensor Response First of all, we built the intensity-depth database. The input

object was moved from 0 to 50 mm above the panel with step of

5 mm, and we recorded intensity values of every z-sensors at

each step. The database can be represented as an I-D curve, as

shown in Figure 5. In the figure, ( ) stood for the captured

intensity of -sensor where the object was at step n of depth.

Figure 5. Z-sensor response (intensity-depth curve)

In the curve, we observed that the z-sensor response followed

our concept within 20 mm; four z-sensors in turn started to

capture signal while flashlight was at 5, 10, 15, 20 mm

separately. However, only -sensor obtained weakened signal

when input over 35 mm, the other z-sensors still stayed

saturated. We believed the reason was the home-made mask

where we could only attached on instead of embedded in the

panel. Such situation ( ) caused design difficulties in

alignment ( ) and issues of incompatible size between sensor

and aperture.

5.2. Depth Accuracy The ability to estimate depth value of input was also testified.

An example is shown in Figure 6 for clear understanding. In the

processed image inset, 2D coordinate (x,y) squared in red was

rendered by ISODATA clustering; meanwhile corresponding z-

sensors were squared in different color. Following, intensity

values of z-sensors ( ) were extracted; hence by using

equation (1), the intensity values were transformed into

probability density functions ( ), also shown in the inset.

Followed by the normalization, features ( ), representing

where the object would likely be, were obtained through using

equation (2)-(3). Finally, maximum likelihood model calculated

the features and retrieved depth values (z). In this case, the error

was 3 mm.

Figure 6. An example with processed data and pdf

More to the depth accuracy, we further estimated the errors at

different steps within the working range. At each step, we

measured the inputs 15 times; the depth response is illustrated in

Figure 7. The maximum error was 5 mm within 35 mm working

range, which was acceptable for near-distance touch; but it went

too large to interact beyond 40 mm. Moreover, for every step,

calculated depth values floated in an error range; the issue was

believed to be resulted from z-sensor response already discussed

in 5.1. However, if we had a panel with embedded depth-sensor

( ), we could ensure error would be less than 1 mm.

Figure 7. Depth error measurement

Hsuan He Fang

5.3. Multi-touch We also clarified the feasibility of multi-touch function. An

example of 2-touch points is shown in Figure 8, two touch

points were put at 0, 10 mm separately. For clear understanding,

the procedure flow is narrated as followed. In the figure, the

arrow 1 indicated the procedure of 2D coordinate location; xy-

image was extracted from the captured image, then ISODATA

clustering rendered the 2D position (x’,y’) of each touch points.

Following, arrow 2, the locations were mapped into the original

coordinate system (x,y) in captured image. In the final

procedures 3 in the figure, the intensity values of z-sensors were

referenced. Moreover, remind of the system architecture in

Figure 2, the spacing between xy- and z-sensors is fixed ( ). We

only regarded and extracted ( ) at correct position, while all

other valued pixels were neglected. Furthermore to the example,

the detail processed data were listed in Table 1, which included

information of z-sensor reference in red dot-line square.

Figure 8. A example of 2 touch points and separation

Table 1. Processed data of the example in Figure 8.

Moreover, we also did experiments for more inputs. However,

whether the case was success depended on extent to how much

overlapping was. In the experiments, we could resolve utmost

three inputs. Last but not least, overlapping is an issue existing

in all interactive systems, yet by how much a system outcomes.

6. Impact We proposed a novel 3D touch system that is applicable on

mobile devices. The system’s working range in near-field and it

is possible to operate with bare fingers. The key contributions of

our works were to propose a simple depth sensing structure

which consists of coded optical barrier above embedded photo

sensors. By controlling the gap and displacement between

barrier aperture and photo sensors, the depth(z) position of

finger tip can be captured easily. Besides, along with touch

algorithm, the system was able to separate multiple touch points

and render 2D coordinate (x,y) by using ISODATA clustering;

plus, depth values (z) were retrieved through a maximum

likelihood model. Finally, we built a prototype on a 4-inch panel

to test the feasibility. The system was composed of 1 xy-sensor

and 4 z-sensors where the working range in depth was 0 to 35

mm with maximum error less than 5 mm. In addition, the system

supported upmost 3 inputs simultaneously.

7. Acknowledgement We’d like to express our appreciation to National Science

Council in Taiwan for financial support under contrast

Academic Projects No. NSC101-2221-E-009-120-MY3.

Meanwhile, we appreciate AU Optronics for useful advices and

panel support.

8. References [1] S. Feiner, B. MacIntyre, T. Hollerer and A. Webster,“ A

touring machine: Prototyping 3D mobile augmented reality

systems for exploring the urban environment”, in First

International Symposium on Wearable Computers Digest of

Papers, pp.74-81, 1997.

[2] A. Olwal, S. DiVerdi, N. Candussi, I. Rakkolainen and T.

Hollerer,“ in Proceedings of the IEEE conference on

Virtual Reality, pp. 279-280, 2006.

[3] A. Abileah, W.d. Boer, T. Larsson, T. Baker, S. Robinson,

R. Siegel and N. Fickenscher,“ Integrated Optical Touch

Panel in a 14.1” AMLCD,” Planar Systems Inc., 2004.

[4] G.Z. Wang, M.C. Ma, H.Y. Tung, Y.P. Huang, H.W.

Tseng, J.C. Lo and C.H. Kuo,“ 50.4: A Virtual Touched 3D

Interactive Display with Embedded Optical Sensor Array

for 5-axis Detection (x, y, z, θ, φ),” SID Symposium Digest

of Technical Papers, vol. 42, pp.737-740, 2011.

[5] M.C. Ma, G.Z. Wang, and Y.P. Huang, “P-199: 3D Finger

Touch with Sequential Illuminator,” SID Symposium Digest

of Technical Papers, vol. 42, no. 1, pp. 1848-1851, 2011

[6] C. Brown, D. Montgomery, J. L. Castagner, H. Kato, and

Y. Kanbayashi, “ 31.3: A system LCD with integrated 3-

dimensional input device,” SID Symposium Digest of

Technical Papers, vol. 41, pp. 453-456, 2010.

[7] S. Suh, C. Choi, K. Yi, D. Park and C. Kim,“An LCD

Display System with Depth-Sensing Capability Based on

Coded Aperture Imaging,” SID Symposium Digest of

Technical Papers, vol. 43, pp. 1574-1577, 2012.

[8] Ball, H. Geoffrey, Hall and J. David,” Isodata: a method of

data analysis and pattern classification,” Stanford Research

Institute, Menlo Park,United States. Office of Naval

Research. Information Sciences Branch, 1965.

[9] C.M. Bishop, Pattern Recognition and Machine Learning,

2nd edition. Springer 2007. Chapter. 2, 3.