A thesis submitted in partial satisfaction of

the requirement for the degree of

Master of Computer Science and Engineering

in the Graduate School of

the University of Aizu

Optimization of Information System

Resources Based on Information of System

User Behaviors

by

Yoshinobu Tanno

March 2012

iii

Contents

Chapter 1. Introduction...........................................................................................1

1.1. Related Works .....................................................................................................3

1.1.1 Green Computing…………………………………………………………………….3

1.1.2 Smart Grid…………………………………………………………………………….3

1.2. Goal of this research ...........................................................................................4

Chapter 2. Proposed Scheme …………………………………………….....................5

Chapter 3. Sample Implementation of the Proposal Scheme…………….………..6

3.1 Tracking Human Behavior …………………………………………………………...6

3.1.1 Sensors for Detecting Human ……………………………………………………...6

3.1.2 Implementation of the proposal Tracking Human Behavior...........................8

3.2. Electronic Power Controller............................................................................10

3.2.1. Management System for Display ………………………………………………..11

3.2.2. The Proposal Management System for Machine……………………………...12

Chapter 4. Performance Evaluation of the Proposal Scheme ..............................14

4.1. Research of Consumed Power……………..........................................................14

4.2. Preciseness of the Proposed Tracking Human Behavior…………………….....16

Conclusion ................................................................................................................17

Reference ..................................................................................................................18

iv

List of Figures

Figure 1 Passage of Electric Power Generation in Key Country……………2

Figure 2 Penetration of Personal Computer in Japan………………………..2

Figure 3 Goal of this research…………………………………………………….4

Figure 4 Sample in Human Tracking from Motion……………………………7

Figure 5 Images of Sensor Data in Kinect………………………………..…….8

Figure 6 Front Case Sample Data……………………………………………….10

Figure 7 Experiment of Environment for This System……………………….11

Figure 8 Sample Implementation of Machine Power Switching…………….12

Figure 9 The Rate of Working Time in Life Time of Japanese …….……...15

Figure 10 Comparing Power Consumption of Display………………………..15

List of Tables

Table 1 Accuracy of Human Body direction…………………………………….16

v

Acknowledgement

I would like to express my appreciation to Professor Takafumi Hayashi for his appropriate advice and diligent efforts in my research. And I also thank the members of our laboratory.

Abstract

Nowadays, consumed power increased with increasing computer usage.

Increasing dissipation power affects country that cannot supply stable

electric-power. Lack of stable power supply in a country damage on public

institution which needs to manage continuously in hospital and bank. There

are many methods to manage power supply but the existing method cannot

ensure stable power supply.

This research proposed a scheme for reducing power consumption because

increasing electric power is difficult to build electric generation plant. And this

research utilizes human behavior because it enables to detail control and it is

possible to reduce power optimally because computer can be integrated under

a proper management.

There are two core approaches to realize reducing power in this research.

First, managing power of personal computer structured system in display and

machine electric current source. This system used Windows, client server

model, wake on local area network, active directory, kinect-sdk-beta1 and can

realized to control power. Second, detecting human body direction back or

front used kinect sensor. The Kinect sensor can detect position of human parts

but cannot detect detail of human motion and pose. Therefore, a system to

detect human body direction was constructed and verified. The result

detecting human body direction back or front calculated 73.46±5%. Result of

these two shows realizable reducing power. In the future, human body

direction needs to ensure more flexible energy saving.

1

Chapter 1. Introduction

Power consumption increases is caused by spreading consumer electronics.

Moreover, the electronics will require more electronic power, because they will

be embedded with high performance computers. As a result, there is potential

of a power shortage. If the power shortage suddenly happened, people can be

injured from situations that machines such as medical instruments stop

working by no-supply electronic power. By the way, the total of supplying

electric power is depended on the amount of electricity generated from power

plants. Though their supply are fixed, the demand of electric power increase,

shown in Fig 1 [1]. And Fig 2 [2] shows the shipment of computers. Shipment

of this figure has increased in each of the years. The results, shows that

continue to increase it and increase power consumption. The problem is

practical from their conditions. So the technologies to manage power

consumption are required for avoiding the electric power crisis. There are two

means to avoid the crisis; increasing a supply of electric power or adopting

technologies to economize on electric power.

At first, avoiding the crisis is thought to be in increasing a supply of electric

power. Electric-generating capacity is decided by number of electric

generation plant. So, it’s growth increase in the supply of electric power. In

Japan produces the electric power through atomic power generation, thermal

power, hydropower generation, and so on. Atomic power generation overtops

other way to generate electricity, in contract damage of radiation leak

problems (Chernobyl, Three Mile Island, Fukushima). Thermal power is

superiority of electricity production along with atomic power, but fossil fuel

used. Therefore, the country has not fossil fuel needs to import it. So it is

difficult to supply electric power stationary for exhaustion and price

movements of fossil fuel. Hydraulic power is stable in the power production

unlike thermal power. But its electric production is less than atomic power

and thermal, it has the problem of destruction of nature. Each ways to

generate electricity have the problems and difficult to build the electric

generic plant.

2

Figure 1 Passage of Electric Power Generation in Key Country

Electric power generation of countries, China, Korea, India,

Brazil, Japan, America, France, Germany and Russia, from 1990 to 2009.

Vertical line shows percentage based on amount of power generation in 1990.

Figure 2 Penetration of Personal Computer in Japan

Figure 2 shows the saturation level of personal computer in Japan. (a) is the

saturation level in the number of households from 1987 to 2011. (b) is the level

which was measured from households excluded alone households.

(b)

(a)

3

1.1 Related Works

Adopting Technologies to Economize on Electric Power

There are two projects.

1.1.1 Green Computing

Green computing has been primarily minimizing power usage for datacenters

and technical equipment (such as desktops, projects) [3]. Green computing also

includes the goals of controlling and reducing the environmental footprint of computing

by minimizing the use and discharge of hazardous materials, conserving water and other

scarce resources, and reducing waste throughout the value chain [4]. To realize it, this

project virtualized machine resource and platform with VMware Server or Xen

and so on. Less number of machine, it is possible to reduce power. Other this

project refines parts of machine and machine for reducing power. For example,

there are reducing server named NX51 and cooling fan named Pulse Width

Modulation [5].

1.1.2 Smart Grid

Smart grid is a modern electric power-grid infrastructure for improved

efficiency, reliability, and safety, with smooth integration of renewable and

alternative energy sources, through automated control and modern

communication technologies [6]. Typical technology of smart grid is smart

metering system. Smart metering systems eliminate many labor-intensive

business processes, such as manual meter reading, field trips for service

connects and disconnects, on-demand reads, power outage and restoration

management, and other metering support functions. Smart metering systems

that continually communicate with smart meters are able to report loss of

voltage to an outage management system (OMS). A well-designed OMS

system can group and analyze customer calls using feeder models that provide

information about likely fault locations [7]. This information, in turn, is

passed along to crews that are working in the field to resolve fault conditions.

This used Google Power Meter and Microsoft Hohm [8].

4

1.2 Goal of This Research

It is not simply for Green computing to keep up the operational energy

consumption of computing equipment [9]. Because green computing has the

problem of economic, law, cycle of consumed power. And smart grid reduces

power what monitor and manage machine power. But there is a possibility of

high consumed power because smart grid need to improve monitor and

security [10] [11].

Both related approaches have disadvantage and could not keep pace human

action in real time. So this research proposed scheme which enables power to

manage automatically, more detail control than manually control regulation.

The goal of this research is illustrated in fig 3. Detecting detail of human

body direction in this research can develop novel method which individual

certification and motion recognition because detecting human body direction

used joint position X Y Z. A lot of join position data permit individual

certification and a lot of human body direction enables to recognize motion. If

this goal achieved, it is available various sectors in security, robot [15],

machine management, user customization, reducing power.

Figure 3 Goal of this research

Figure 3 shows goal of this research. Human body direction takes off

individual certification and cognition of action. These categories have an

impact in many sectors.

5

Capter2. Proposed Scheme

The proposed scheme can realize an effective power consumption

management based on human behavior. The proposed scheme requires only

small set of system which itself requires very small power consumption and

low management cost. The proposed scheme can manage the power

consumption of various systems by direct control with simple control. The

proposed scheme detects human pose using depth camera and manages

electronic consumption of devices connected to network. If the system

recognizes that the human does not use a particular device, the device is

changing suspension mode with orders from the system.

The proposed scheme uses an electronic connector embedded with systems

for turning on or off and depth camera for tracking human behavior. This

system only controls sensors in the connector for managing electronic power.

Therefore, target device of which a proposed scheme manages electronic power

is not need to embed with any other sensor.

Depth camera is a camera to measure distance of objects on its angle of view.

So it can recognize objects or human from a shape which can be formed from

group of objects on same distance.

A system which reduces amount of consuming electronic power is required to

reduce the number of mistaken movement decreases. Because one of most

consumption of electronic power is occurred when the device is starting,

amount of electricity consumption increases by turning on or off a device by

mistaken movement. In the case using time-based power manager, it cannot

detect which device is used or not. For example, that manager cannot

recognize behavior about watching display without no-input, but proposed

scheme can do using tracking human behavior with depth camera.

This research used tracing human behavior because it enables to detail

control.

Computer can supply and manage power in display controls inside of system.

And it is possible to reduce power optimally because computer can be

integrated management.

6

Capter3. Sample Implementation of

the Proposal Scheme

The proposed method consists of two function; tracking system of human

behavior and management system of electric power. This section explains

about constructed system. First, construction of tracking system is shown in

section 3.2. Then management system is shown in section 3.3.

3.1 Tracking Human Behavior

3.1.1 Sensors for Detecting Human

In this paper, Kinect sensor is used for tracking human behavior. This

sensor, which is made by Microsoft, consists of an infrared camera and an

infrared laser emitter for measuring distance. To measure the distance, it

measures depth as a triangulation process [12]. The laser source emits a single

beam which is split into multiple beams by a diffraction grating to create a

constant pattern of speckles projected onto the scene. This pattern is captured

by the infrared camera and is correlated against a reference pattern. The

reference pattern is obtained by capturing a plane at a known distance from

the sensor, and is stored in the memory of the sensor. The sensor using their

measuring method is classified in a depth camera. A RGB camera is also

equipped on a Kinect. So the sensor can provide information of depth and

camera image, simultaneously.

To tracking human behavior is required that the system can extract

human-posture from sensor data. For recognizing human posture, the

proposed method adopted a depth camera. One of popular method to tracking

walkers on station or street is a camera based method [13]. However, the

method using camera is difficult to recognize human-posture from images,

because it tracks walker behavior with gaps of before and after camera image.

On the other side, a depth camera is used to measure distances between the

cameras and objects and provides a depth map that is an aggregate of a result

about measuring distance. Therefore, the depth camera classifies recognized

object into human or others using the depth map. In other words, depth

camera can recognize human-body [14].

7

Fig 4 shows a result of detecting human silhouette in real time. Human

postures can be extracted from each image. Then the proposed method tracks

the behavior using a set of extracted postures. Kinect can detect RGB Image,

Depth Image, Skeletal Image, Position Data, shown in Fig 5. However, it is not

enough to detect human body direction, and the proposed method for tracking

required to detect human body direction.

Human State1 State2 State3 State4 State5

Motion

Figure 4 Sample in Human Tracking from Motion

Using sensor, human silhouette can be got continuously from motion. Frame

rate of this sensor is about 30fps in this research.

RGB Image (a) Depth Image (b)

8

Skeletal Image (c) Position Data (d)

Figure 5 Images of Sensor Data in Kinect

(a) Camera image. (b) Depth image. Each depth divided into color. (c)

Skeletal image measured from depth information (d) Axis information of body

parts. Kinect can estimate 20 parts of body from both camera image and depth

image. Then axis of each part is measured from the estimated result.

3.1.2 Implementation of the Proposal Tracking Human

Behavior

To detect human body direction back or front in a captured image, ten front

and back position data are used. These data and other sample data in 800

images and position data (Fig 6) are compared. Proposed scheme used

k-means algorithm because K-means is another popular clustering algorithm that has

been used in a variety of application domains, such as image segmentation and

information retrieval [16]. K-means can calculate distance of each object. And k of

value defined two, after got out except front and back body direction. To detect

back or front body direction, k-Nearest Neighbor method (k=3) because using

three points can be trilateration which raises probability of correct.

Detail of this method show following.

Definition

S: Sample case data

B: Back and Front case data

P: Human parts

D: Distance

First: sample data get out one data

9

Second: compare between got out x y z position data in human parts of 20

and back and front sample position data x y z (total 20 images data)

in human parts of 20 images in following

Third: sorted according to the 20 of the D in ascending order

Forth: get out top 3 data from sorted 20 of D

Fifth: image is judged as front or back by majority vote in top 3 of destination

data.

Repeat number of sample images from first to fifth.

This figure shows scrrenshot from running Kinect program in kinectSDK.

Back and Front sample based this data. Depth stream, Skeleton, Color

Video Stream is same in fig 5. (a) show frame rate.

Back Case Sample Data

(a)

10

Front Case Sample Data

Figure 6 Back Case Sample Data and Front Case Sample Data

This figure explain how used images data to detect human body direction.

These images are detected data of laboratory member and the author when

they open or close door in laboratory. Position data of these images is based

data to compare other images data.

3.2. Electronic Power Controller

More PC increase, more power increase. So this research implement to

reduce power system for PC related display and machine of power switching.

To implement, this system uses windows OS because Windows supported

Microsoft SDK and Kinect SDK. And it use Active Directory in machine

switching system because Active Directory can be managed other machine

efficiency. Therefore, existing machine also can be utilized in this system

because target device not need to embed with any other sensor for controlling

electronic power.

There are two systems for reducing electronic power. One is power of display.

Switching system. Two is power of machine switching system. This system

based environment of experiment for system shows Fig 7.

11

Figure 7 Experiment of Environment for This System

Figure 7 shows area to detect human data. If user goes to workspace, sensor

detects user and client run.

3.2.1. Management System for Display

While resting time is the user turns us back to display, display power switch

off. As a result, power can be reduced. When user in workspace Fig. 7 backs to

display, power of display is off.

Environment of usage

Machine: ProLiant ML 115 G5

Developing environment: Visual Studio

OS: Windows7 Professional 64 bit

Programing Language: C#

SDK: kinectSDK-v1.0-beta1-x64.msi

1 set a Kinect to machine

2 register back and front position data

3 execution of file

12

3.2.2. The Proposal Management System for Machine

It is possible for this system to control detail power which human cannot be

difficult to control. If user goes work space in as illustrated in Fig 7, the sensor

detects the user. When detecting user at first time, server sends start command to

client. While user works in work space, sensor continues to detect user. If user go

back home, sensor stops detecting user. Serve sends shutdown command to client

when Time of not detecting user is long time. This situation shows Fig 8.

Figure 8 Sample Implementation of Machine Power Switching

As a function of human body direction, Sensor in server sends shutdown or

run command to client machine flexibly.

Environment of usage

Machine: ProLiant ML 115 G5

OS: Windows7 Professional 64 bit (Client), Windows Server 2008 32bit (Server)

Developing environment: Visual Studio

Programing Language: C#

SDK: kinectSDK-v1.0-beta1-x64.msi

13

Client Server model

Server

1 install DNS server

2 install Active Directory Domain Server

3 add client machine name in Active directory

Client

1 set DNS address to server IP address

2 join server ’s domain

Setup power reduce system

1 server or other machine set Kinect

2 register back and front position data

3 execution of file

Machine power switching on System

If human is detected, server or other machine sends WoL (Wake on LAN)

command to client machine. WoL switches power in machine because WoL

send Magic Packet to Network card in machine. Magic packet is particular

packet, send mac address of client machine to broadcast sixteen [17]. Even if

machine shutdown, machine can be run because network card of client is wait

body direction. Everyone can use it because most of latest machine have this

function. Only WoL, Server not needs Active Directory.

Machine power switching off

To run remote machine, need to set sensor in server or other machine. To stop

client machine with this sensor, this server must have faculty of client’s

administrator. So this system needs active directory functions. Active

directory is a directory service made in Microsoft [18]. This server can be

structured client server model easily. Moreover this service is what server has

client’s administrator account. In fact server can send shutdown command to

client as administrator. This system run when Kinect is not detect any human

long time.

14

Capter4. Performance Evaluation of

the Proposed Scheme

4.1 Research of Consumed Power

To show proof of importance of reducing power, show electricity consumption

of a day in display because data of a day is vital unit. Used display is SHARP

LL-T1620H, to get apparent power used arduino shield named

WATTMETER2 and to get active power used smart outlet named FX-5204PS.

Apparent power is the product of effective current and effective voltage. Case

of direct current, apparent power and active power is same and consumed

power but case of alternating current, require active power because generate

wattles power. Both apparent power and active power shows in graph because

apparent power is likely to be used by personal electronics. To get value of

current and voltage in parallel, display link smart outlet and wattmeter2.

Average active power of display in all a day is 37.10W. Other apparent

power is 55.05W that product average of voltage in all a day (110.10V) with

smart outlet and average of current in all a day (0.5A) with arduino shield.

Apparent power is about 1321Wh that product 55.05W and 24 hours and

active power are about 890Wh that product 37.1W and 24 hours in

consumption of all a day. Other Apparent power is about 440Wh that product

55.05W and 8 hours and active power is about 296Wh that product 37.1W and

8 hours in consumption of saving power. 8 hours in saving power calculated

that purse data more than 50% in the rate of working person half-Hourly in

2010 [19] because data of more than 50% define general working time and

define general worker used display while this time. Value of saving power and

consumption power in all a day shows Fig 6.

Figure 6 of point is that compare consumption of saving power and all a day.

Active power reduces 594Wh and apparent power reduces 890Wh and shows

reducing power of 66%. Moreover general working time in Japanese is selected

as No.2 in 2011 [20]. General working time in Japan is longer nevertheless,

consumption power can reduce 66%. In fact, electric power saving is very

important and avail research.

15

Figure 9 The Rate of Working Time in Life Time of Japanese

This figure shows the number of persons which were working every 30

minutes [17]. The figure was a result that 4,905 peoples answered a

questionnaire of the Japan government.

Figure 10 Comparing Power Consumption of Display

This figure shows comparing power consumption between using saving and no

saving of electric power.

saving power

all a day

0

200

400

600

800

1000

1200

1400

activepower apparent

power

296 440

890

1330

saving power

all a day

(Wh)

16

4.2 Preciseness of the Proposed Tracking Human

Behavior

To manage power robustly, this research detected human body direction back

or front using k-means algorithm (k=2). To evaluate detected data of 800,

make a decision based on the following.

If shoulder and head looked to sensor, this image is front.

If shoulder and head looked to sensor against, this image is back.

If computer resulted data and my evaluation is same, computer of decision is

correct. Other is error. And to take a sample detected data, used following.

Definitions

n: Quantities of sampling

N: Largeness of parent population

e: maximum error

z: normal distribution matching fidelity

P: prospective rate of parent population

Number of evaluated data is randomness of 260 from 800 because 5% of

accident error can be allowance. This result is following. Total accuracy of

human body direction is 73.46 5%.

Table 1 Accuracy of Human Body direction

This table shows preciseness of human body direction. Total parameter is total

of category. Correct count is number that evaluated correct data in detecting

human behavior with k-means (k=2). Correct of probability is number that

dividing correct count by total value and add purse and minus five percent.

17

Capter5 Conclusion

Power consumption is increasing yearly because there is one factor what

shipment of computers also is increasing yearly. There exist projects which

include green computing and smart grid for this problem. But both projects

have each problem and not control power consumption which stands to human.

And so this research proposed scheme for reducing electricity power

appropriate to human body direction. This proposed system defined using

home but datacenter and office also can use it. If user’s behavior enabled to

control electric power in machine, it is possible for user to reduce electricity

without intercepting use of system. There are two main approaches of the

proposed scheme to realize reducing power.

First, managing power of personal computer structured system in display

and machine electric current source. This proposed system used windows,

client server model, wake on local area network, active directory,

kinect-sdk-beta1 and can realized to control power.

Second, detecting human body direction used kinect sensor. In the future,

human body direction needs to make energy saving more flexible. Low cost

and existing sensor can detect human position X Y Z but cannot detect human

body direction. So for detecting human sate, this research used 2-means

algorithm and k-Nearest Neighbor method (k=3) to compare between sample

position data of 10 back, front and other selected 273 at random in 800

position data to permit 5% error. In the result, calculated 73.46 5%.

Detecting human body direction in this research can develop a novel method

which individual certification and motion recognition because detecting

human body direction used joint position X Y Z. A lot of join position data

permit individual certification and a lot of human body direction enables to

recognize the human motion. If this goal achieved, it is available various

sectors in security, robot [14], machine management, user customization,

reducing power.

In the future, preciseness of recognizing human body direction needs to

increase because human body direction continues detecting in real time. And

not only human body direction of back and front but also left and right, skew

need to detect to make reducing power flexibility.

18

Reference [1] Community Data of Actual Condition, "PC Penetration in Household in

Japanese", http://www2.ttcn.ne.jp/honkawa/6200.html.

[2] Information Area of Electricity, "Section1 Environment of Energy in World

and Japanese",

http://www.fepc.or.jp/library/publication/pamphlet/nuclear/zumenshu/

[3] Robert Harmon, Haluk Demirkan, Nora Auseklis, Marisa Reinoso,”From

Green Computing to Sustainable IT:Developing a Sustainable Service

Orientation”, Proceedings of the 43rd Hawaii International Conference on

System Sciences – 2010.

[4] Auseklis, N., Demirkan, H. Hrmon, R. and Hefley, B., “Designing IT

Services for Sustainability and Business Values”, 2nd Annual International

Conference on Business and Sustainability: Designing Sustainability, October

15-17, 2008.

[5] ITpro Green IT Group of Repoters, ”Complete Understanding of Green IT

in Japanese”, June 19, 2008.

[6] Gungor, V.C., BinLu, Hancke, G.P, “Opportunities and Challenges of

Wireless Sensor Networks in Smart Grid”, IEEE TRANSACTIONS ON

INDUSTRIAL ELECTRONICS, VOL. 57, NO. 10, OCTOBER 2010.

[7] Cal LaPlace, “Realizing the smart grid of the future through AMI

technology” http://www.energyaxis.com/pdf/RealizingSmartGrid.pdf

[8] Tadahiro Gouda, “Smart Grid Textbook in Japanese”, February 24, 2011.

[9] David Wang,”Meeting Green Computing Challenges”,Electronics

Packaging Technlogy Conference, 2008. EPTC 2008. 10th.

[10] M. Amin and B. F. Wollenberg, “Toward a smart grid,” IEEE Power

Energy Mag., vol. 3, no. 5, pp. 34–41, Sep./Oct. 2005.

[11] R. Krishnan, “Meters of Tomorrow,” IEEE Power and Energy Magazine,

pp. 92–94, Mar. 2008.

[12] Norman Villaroman, Dale Rowe, Bret Swan, “Teaching Natural User

Interaction Using OpenNI and the Microsoft Kinect Sensor”,

http://sigite2011.sigite.org/wp-content/uploads/2011/10/session14-paper02.pdf

[13] Maged N Kamel Boulos, Bryan J Blanchard, Julio Montero, Aalap

Tripathy, Ricardo Gutierrez-Osuna,“Web GIS in practice X: a Microsoft Kinect

natural user interface for Google Earth navigation”, Kamel Boulos et al.

International Journal of Health Geographics 2011.

[14] K.Khoshelham,“ACCURACY ANALYSIS OF KINECT DEPTH DATA”,

http://www.isprs.org/proceedings/XXXVIII/5-W12/Papers/ls2011_submission_

19

40.pdf

[15] Wasim Menesi, Paul Vilchez, Mohammad Usman,”Natural User Interface

for Robot Armature Control”,

http://www.pvilchez.com/content/pdf/robonuiProposal.pdf

[16] Kiri Wagstaff, Claire Cardie, Seth Rogers, Stefan Schroedl,“Constrained

K-means Clustering with Background Knowledge”, Proceedings of the

Eighteenth International Conference on Machine Learning, 2001, p. 577-584.

[17] Wikipedia,”Wake-on-LAN”, http://ja.wikipedia.org/wiki/Wake-on-LAN

[18] Wikipedia,”ActiveDirectory”, http://ja.wikipedia.org/wiki/Active_Directory

[19] Toshiyuki Kobayashi, Emi Morofuji, Yoko Watanabe, Japanese Life

Time 2010 in Japanese ,

http://www.nhk.or.jp/bunken/summary/research/report/2011_04/20110401.pdf

[20] OECE,”Society at a Glance 2011 – OECD Social Indicators”

http://www.oecd.org/document/24/0,3746,en_2649_37419_2671576_1_1_1_374

19,00.html