kifall: detecting falls using an android-based smartphone...

KiFall: Detecting Falls Using an Android-BasedSmartphone and the Kinect Sensor

Final Year Project

Author: Supervisor:

Bruna Girvent Dr. Kaushik ChowdhuryEscola Tecnica Superior d’Enginyeriade Telecomunicacio de Barcelona (ETSETB)

Electrical and ComputerEngineering Department (ECE)

Universitat Politecnica de Catalunya (UPC) Northeastern University (NEU)

Barcelona (Spain) Boston (USA)

October 2, 2013

KiFall: Detecting Falls using an Android and the Kinect

-1-


Abstract

One-third of the elderly population experiences at least one fall each year. For the thispopulation or patients with special mobility needs a fall can have dramatic health conse-quences. Fast and precise detection of falls is critical, and prompt notification to emergencyservice is essential for a short recovery time. Therefore, many fall detection systems have beendeveloped. The existing commercial solutions consists of wearable devices placed on differentlocations on the body. However, these devices can be rather expensive, uncomfortable towear, and the individual have to remember to wear them.

In this thesis we propose a fast, private system using off the shelf components typicallyfound at homes, such as an Android-based smartphone and the Kinect sensor. Our systemis based on detecting changes in motion and body position of an individual by trackingacceleration and orientation changes. The data is continuously acquired and analyzed in thesmartphone to determine when a fall occurs. The system integrates the Kinect for morereliable fall detection. If a fall is detected, the system sends an emergency alert which consistof a picture, the time, and the location of the individual.

We performed a set of experiments to test the fall detection algorithm and set up all theparameters of the system. We implemented a prototype system on the Nexus S connected tothe Kinect through a TCP server. Experimental results show that our system achieves strongdetection performance, fast time response, and little impact on the smartphone’s battery life.

-2-


Resumen (espanol)

Un tercio de las personas mayores de 65 anos sufren, al menos, una caıda al ano. Lasconsecuencias de una caıda en personas de edad avanzada o pacientes con problemas demovilidad pueden ser graves. Por eso es importante detectar rapidamente cuando sucede yalertar los servicios de emergencia. Cuanto mas tiempo el individuo esta en el suelo despuesde una caıda, mas lenta es su recuperacion. La mayorıa de los sistemas actuales consisten enpequenos dispositivos que el usuario debe llevar sujetos en distintas partes del cuerpo, comocollares o brazaletes. Estos dispositivos suelen ser caros e incomodos. Ademas el individuodebe recordar de llevarlo ya que en caso contrario la caıda no sera detectada.

En este proyecto de fin de carrera proponemos un sistemas de deteccion rapida y au-tomatica de caıdas usando componentes de bajo coste como un telefono movil Android y elsensor Kinect, dos componentes faciles de encontrar en la mayorıa de hogares. Nuestro sis-tema se basa en detectar cambios en el movimiento y la posicion del individuo analizando loscambios en la aceleracion y la orientacion del cuerpo. Estos datos se obtienen y analizan entiempo real mediante un telefono movil, permitiendo ası detectar si ha habido una caıda. Elsistema integra el Kinect obteniendo ası mejores resultados en cuanto a fiabilidad y eficiencia.Cuando una caıda es detectada mediante el telefono movil, el sistema alerta a los servicios deemergencia mediante un mensaje multimedia con la alerta, el tiempo en que se ha producidola caıda, una foto del individuo en el suelo y su localizacion.

Varios experimentos han sido realizados para disenar el algoritmo de deteccion de caıdaspropuesto. Todos los parametros han sido optimizados para conseguir los mejores resultadosen cuanto a fiabilidad, eficiencia y tiempo de respuesta. El prototipo ha sido implementadoen un Nexus S conectado a un Kinect mediante un servidor TCP. Los experimentos realizadosdan buenos resultados en cuanto a deteccion fiable de caıdas, rapido tiempo de respuesta ybajo consumo de baterıa del telefono movil.

-3-


´

Resum (catala)

Un terc de les persones majors de 65 anys pateix, al menys, una caiguda a l’any. Lesconsequencies d’una caiguda en persones d’edat avancada o en pacients amb problemes demobilitat poden ser greus. Per aixo es important detectar rapidament quan un individu hacaigut i poder alertar els serveis d’emergencia el mes aviat possible. Quant mes temps lapersona esta al terra despres d’una caiguda, mes difıcil i lenta es la seva recuperacio. Es peraixo que actualment s’estan desenvolupant molts sitemes de deteccio de caigudes. La majoriade sistemes actuals, pero, consisteixen en petits dispositius que l’usuari ha de portar en formade collaret o bracalet. Aquests dispositius solen ser cars i incomodes. A mes, l’individu hade recordar de portar-los ja que sino la caiguda no es detectada.

En aquest projecte de final de carrera proposem un sistema de deteccio rapida i automaticade caigudes utilitzant components de baix cost com ara un tel·lefon mobil Android i el sensorKinect, dos components facils de trobar a la majoria de llars. El sistema es basa en detectarcanvis en el moviment i posicio de la persona analitzant les variacions en l’acceleracio iorientacio del cos. Aquestes dades s’analitzen continuament en el mobil, permetent aixı ladeteccio de caigudes en temps real. El sistema integra el Kinect per obtenir millors resultatsen quan a fiabilitat i eficiencia. Quan una caigua es detectada, el sistema envia un missatged’alerta que consisteix en l’hora i la localitzacio d’on s’ha produit, juntament amb una fotode la persona estirada al terra.

S’han realitzat diversos experiments per a dissenyar l’algoritme de deteccio de caigudes,aixı com optimitzar tots els parametres del sistema. S’ha utilitzat el tel·lefon mobil NexusS conectat remotament al Kinect mitjancant un servidor TCP. Els experiments realitzatsmostren resultats satisfactoris en quant a deteccio fiable de caigudes, rapid temps de resposta,i baix consum de bateria del tel·lefon mobil.

-4-


Contact

Bruna GirventStudent of Computer EngineeringEscola Tecnica Superior d’Enginyeria de Telecomunicacio de Barcelona (ETSETB)Universitat Politecnica de Catalunya (UPC)Mail: [email protected]: http://byteway.wordpress.com

Dr. Kaushik ChowdhuryAssistant ProfessorElectrical and Computer Engineering Department (ECE)Northeastern University (NEU)Mail: [email protected]: http://krc.coe.neu.edu

-5-

[email protected]

http://byteway.wordpress.com

[email protected]

http://krc.coe.neu.edu


Acknowledgments

I would like to thank all the people who contributed in some way to the work described in thisthesis. First and foremost, I thank Professor Kaushik Chowdhury for offering me the opportunityof finishing my studies in Northeastern University. I really appreciate all the contributions of time,ideas and advice to make this thesis experience productive and stimulating.

I also thank all the people from 216 Egan Research Building laboratory for all the hours sharedthere. Thank you guys for the stimulating conversations, helpful support and invaluable assistance.

Special gratitude to all the people who contribute to make this year in USA one of the mostamazing experiences of my life. Thanks to Siana, Sinkari, Ferran, Sathya, Clift, and Elena for allthe time shared during those months. And also to all the other people who I met during this yearin Boston.

Gracies tambe als companys de telecos de la ETSETB-UPC per el suport mutu i les estonescompartides durant la carrera. Tambe als professors que m’han donat suport i ajuda abans, duranti despres d’aquesta experiencia.

I l’agraiment mes especial per a la familia i amics de Catalunya, pel suport i l’energia enviadesdes de casa. En especial a la meva germana Mireia per aguantar-me, encara que sigui a la distancia,i als meus pares per donar-me arrels i ales.

-6-


-7-


Contents

1 Introduction 10

2 State of the Art 112.1 Healthcare . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.1.1 Healthcare and technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.1.2 Telecare products for the elderly . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.2 Smartphones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132.2.1 Mobile operating systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132.2.2 Built-in sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152.2.3 Smartphones in healthcare . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

2.3 Kinect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182.3.1 Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182.3.2 Features and limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202.3.3 Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212.3.4 Kinect in healthcare . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

2.4 Fall detection systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242.4.1 Commercial products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242.4.2 Android apps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242.4.3 Current research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

3 Design 313.1 Smartphone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

3.1.1 Fall detection block . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333.1.2 Emergency alert block . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

3.2 Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353.3 Kinect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

4 Preliminary experiments 37

5 Development 455.1 Smartphone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

5.1.1 Getting started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 455.1.2 Hierarchy of classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 465.1.3 Code implementations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

5.2 Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 535.3 Kinect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

5.3.1 Getting started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 555.3.2 Code implementations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

-8-


6 Results 586.1 System overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 586.2 Performance evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

6.2.1 Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 596.2.2 Speed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 606.2.3 Battery consumption of the smartphone . . . . . . . . . . . . . . . . . . . . 61

6.3 Comparison with other fall detection apps . . . . . . . . . . . . . . . . . . . . . . . 616.3.1 Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 616.3.2 Battery consumption of the smartphone . . . . . . . . . . . . . . . . . . . . 62

7 Discussions 637.1 Point the Kinect: indoor location . . . . . . . . . . . . . . . . . . . . . . . . . . . . 637.2 Verify the fall with the Kinect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

8 Conclusions 658.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 658.2 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

References 67

List of tables, figures and listings 70

Appendices 73

A Getting started with Android SDK 73A.1 Install the Android SDK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73A.2 Setting up Eclipse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73A.3 Programming with Eclipse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74A.4 Installing the app . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76

B Getting started with the Kinect sensor 78B.1 Getting started with the Kinect for Windows SDK . . . . . . . . . . . . . . . . . . 78

B.1.1 Installing the Kinect sensor and the developer tools . . . . . . . . . . . . . . 78B.1.2 Programming the Kinect with Visual Studio . . . . . . . . . . . . . . . . . . 79B.1.3 Programming the Kinect with Matlab . . . . . . . . . . . . . . . . . . . . . . 80

B.2 Getting started with the OpenNI SDK . . . . . . . . . . . . . . . . . . . . . . . . . 81B.2.1 Installing the Kinect sensor and the developer tools on Windows . . . . . . . 81B.2.2 Programming the Kinect with Visual Studio on Windows . . . . . . . . . . . 82B.2.3 Installing the Kinect sensor and the developer tools on MacOS X . . . . . . 83B.2.4 Programming the Kinect with XCode on Mac . . . . . . . . . . . . . . . . . 84B.2.5 Programming the Kinect with Processing+SimpleOpenNI . . . . . . . . . . . 85B.2.6 Programming the Kinect with Matlab . . . . . . . . . . . . . . . . . . . . . . 86

C Published Paper 87

-9-


1 Introduction

The elderly population or patients with special mobility needs experience falls than can havedramatic consequences. About one third of older adults have one or more falls per year, andthe risk of falls increase proportionately with age. Studies show that falls are also common withParkinson’s disease, patients with cognitive problems, dementia, and stroke patients: up to 40%of people who have a stroke have a serious fall within the next year.

It is established that the sooner the fall is reported, the lower the risk of severe injuries ordeath. This reinforces the importance of medical alert system devices. Current monitoring devicestypically requires user input such as pushing a button to alert that a fall or emergency has occurred.The problem with this type of monitoring device is that many times, when a fall has occurred,the individual is rendered unconscious and can’t activate the control. Other solutions includessurveillance using cameras or motion detectors, although the use videocameras carries with itprivacy concerns. Nowadays, the detection of falls is an interesting problem which one can approachusing various methods.

Today’s smartphones are sophisticated computing platforms with complex sensor capabilities,such as detecting user location and sensing motion, among others. The built-in motion sensorsmake it possible to develop reliable fall detection systems based on body acceleration and orien-tation. Moreover, location functions and phone’s connectivity enable the ability to call for helpand get rapid assistance when a fall is detected. Therefore, smartphones are portable and afford-able devices whose self-contained functionalities make them an efficient cost-effective solution fordetecting falls.

Microsoft’s Kinect is a set of sensors developed as a peripheral device to use as a touch-freegaming controller. Nevertheless, Kinect’s potential expands far beyond just gaming. The sensorsoffers great possibilities in the medical field: the motion capture functionalities enables the Kinectto track people’s movements, a different approach for a fall detection system.

We propose a fast, private system using shelf components, such as smartphones and gamingcontrols typically found at home. Our system integrates an Android-based smartphone and Mi-crosoft’s Kinect sensor for fast and reliable fall detection. The phone continuously acquires andanalyze the data from motion and position sensors such as accelerometers and gyroscopes. TheKinect could be used to verify the fall by motion analysis based on skeleton tracking. If a fall isdetected, the emergency alert is sent through the smartphone and consist of a multimedia message(MMS) including the picture, the time and the location of the individual lying down after a fall.

-10-


2 State of the Art

2.1 Healthcare

2.1.1 Healthcare and technology

Healthcare involves the diagnosis, treatment and prevention of diseases and injuries. Typicallyinvolves face to face interaction with a doctor, nurse, or pharmacist. However, when the delivery isvia telecommunications, it is known as telecare or telehealth. On the other hand, if the healthcareis supported by electronic devices, it is known as eHealth. From this form emerge the term mHealthin reference to using mobile devices such as smartphones, tablets or PDAs.

Using smartphones and communication networks it is possible for practitioners and patients tocollect remote aggregate health data, provide care via mobile telemedicine, and real-time monitor-ing of patients. Different systems have been developed, that take advantage of the new technologiesand devices.

2.1.2 Telecare products for the elderly

Personal Emergency Response Systems (PERS)PERS lets users call for help in an emergency by pushing a button. A PERS has three components:a small radio transmitter, a console connected to your telephone, and an emergency response centerwith caregivers that monitors calls. If the person pushes the button, it automatically calls theresponse center. Once the call is established, the caregiver can talk to their client and ascertainwhat help is required.

Figure 1: Personal Emergency System (PER)

Telecare sensorsTelecare Sensors work in conjunction with the PERS system and complement its function byproviding additional passive monitoring around the home. The sensors monitor risks and hazardsthat are associated with independent living. They are wireless, battery operated, and simple toinstall.

-11-


Telecare sensors fall into two categories, those that monitor the environment in the home, andthose that monitor the activities of the person living there. When an emergency is detected, ituses the PERS to call the caregiver and establish communication.

Telecare sensors for daily livingThe next generation of telecare takes the application of sensors even further. The systems useenvironmental sensors and motion sensors to track daily living activities of the individual. Thesystems use all the raw information collected from passive infrared detectors, bed, chair, and doorsensors, plus sensors for detecting the use of fridge, coffeemaker or TV.

Figure 2: Telecare sensors for daily activity

Fall DetectorsFall detectors are small devices that clip onto the wearer’s belt or waistband. They contain sensorsand microprocessors to recognize the change of orientation and impact that occurs during a fall.

The risk of falling for the elderly while getting out of bed or in the bathroom is significant. Abed occupancy sensor is used to provide info on an abnormal event. The device uses a pressuresensor under the mattress to detect when the person is in the bed. It is configured according tothe normal sleep routine: if the person does not get up in the morning, or rises from bed duringnight, it calls for help.

Figure 3: Fall Detection System

-12-


Other solutions includes infrared detectors, video cameras and other motion sensors. Thereis also lot of research in this field. Fall Detection Systems are explained with more detail in thesection Fall Detection Systems.

2.2 Smartphones

A smartphone is a portable device built on a mobile operating system which includes advancedcomputing capability and connectivity. Today’s smartphones include complex sensor capabilitiessuch as detecting user location, recording high-quality video, measuring ambient temperature,sensing geomagnetic strength, and sensing motion or position.

Due to widespread use of smartphones it is now possible to develop applications for almostanything. The possibilities for smartphones applications are nearly endless in number and design.The software development kids (SDKs) and all the resources available for each platform combinedwith the latest smartphone models make easy to imagine and develop new applications and newfeatures.

Figure 4: Samsung Galaxy Nexus S from Google

2.2.1 Mobile operating systems

A mobile operating system (mobile OS) is the operating system who manages both hardware andsoftware. The mobile OS operates any kind of mobile devices including smartphones, tablets andPDAs. The major mobile operating systems today are iOS, Android OS, Windows Phone andBlackberry.

iOS Apple’s mobile OS runs on iPhone, iPod Touch, and iPad devices. The Apple Developerwebsite1 includes the SDK and provides all the resources to start building apps on MacOS.A Developer License, which cost $100, is needed to test the apps on the device and distributethem on the App Store. Currently, iOS is the mobile OS which has more apps developed.

Android This open source operating system was released by Google. Though it is an open sourceOS, carriers and manufactures can design a specific version for their devices. The Android

1Apple Developer website http://developer.apple.com

-13-

http://developer.apple.com


Developer website2 includes all the resources to start developing apps. Nowadays is the mostextended mobile OS.

Windows Phone Developed by Microsoft is the mobile OS running in Nokia and some LGsmartphones and tablets. The Windows Phone Dev Center3 includes all the documentation,references, support forum and SDK to start developing apps on Windows.

Blackberry Mobile OS developed by BlackBerry Ltd for its BlackBerry line of smartphone hand-held devices. This platforms is known for its native support for corporate email but nowadaysit also allows to develop and install third party apps. All the resources to design, developand distribute apps are available on the BlackBerry Developer Center4.

Mobile OS MarketShare

Num.Apps

Devices Store

iOS 39.2% 900.000 iPhone, iPod Touch, iPad Apple StoreAndroid OS 52.0% 750.000 Large number of devices Play StoreWindowsPhone

3.0% 160.000 Nokia and some LG models Windows PhoneStore

BlackBerry 5.1% 120.000 BlackBerry smartphones BlackBerry WorldTable 1: Comparison of today’s smartphones. Source: Gartner (April 2013)

Mobile OS IDE Programminglanguage

Platform Distributionlicense

iOS XCode ObjectiveC MacOS Apple Dev.License($100/year)

Android OS Android StudioEclipseother IDEs

Java WindowsMacOSLinux

Free

WindowsPhone

Visual Studio C# Windows Free (underMicrosoft super-vision)

BlackBerryOS

EclipseBlackBerry JDE

C/C++JavaHTML 5

WindowsMacOSLinux

Free (underBlackberry super-vision)

Table 2: Comparison of mobile operating systems

2Android Developer website http://developer.android.com3Windows Phone Dev Center https://dev.windowsphone.com4BlackBerry Developer Center http://developer.blackberry.com

-14-

http://developer.android.com

https://dev.windowsphone.com

http://developer.blackberry.com


2.2.2 Built-in sensors

Current smartphones have a wide range of built-in sensors including location, motion, positionand environment sensors. They add rich location and motion capabilities to the apps, from GPSlocation to accelerometer, gyroscope, temperature, barometer, and more.

Location sensorsMost of the devices includes an Assisted-GPS (A-GPS) for location. The A-GPS uses differentlocation sources: GPS, cell-ID and Wifi can provide a clue to users location. All of the sourceslocate the smartphone based on triangulation with three or more reference points, wether it issatellites, cellular antennas or wifi spots.

Figure 5: Location by triangulation

• GPS: Good accuracy but long time to find location (more than 10sec).

• Cellular network: Poor accuracy but better location time.

• Wifi network: Poor accuracy in comparison to cellular network but faster location time.

Different strategies can be used to obtain a reliable user location, depending on the location,accuracy and speed requirements. In outdoors, the best choice is to use the GPS. A good strategycould be to use the celular network for the first 10 seconds and refine the location after 10 secondswith the data from GPS. Since the GPS signal is weak indoors, a good strategy would be to usethe cellular or Wifi network.

Motion sensorsMotion sensors are useful for monitoring device movent. Usually smartphones combine hardware-based sensors with software-based. The software-based sensors derive their data from the hardwareones, for example the gravity sensor can derive data from the accelerometer and either magne-tometer or gyroscope.

• Accelerometer (hardware-based): Acceleration force along the x-,y- and z-axis in m/s2.

• Gravity (usually software-based): Force of gravity along the x-,y- and z-axis in m/s2.

-15-


• Gyroscope (hardware-based): Rate of rotation around the x-,y- and z-axis in rad/s.

• Linear acceleration (usually software-based): Acceleration force (excluding gravity) alongthe x-, y-, and z-axis in m/s2.

• Rotation vector (software-based): Rotation vector component along the x- (x ∗ sin(θ/2)) , y-(y ∗ sin(θ/2)), z-axis (z ∗ sin(θ/2)) and scalar component (cos(θ/2)) of the rotation vector.

Position sensorsTwo sensors are used to determine the position of the device: the geomagnetic field and theorientation sensor. Most of the smartphones also includes a proximity sensor to determine howclose the face of the device is to an object.

• Magnetic field (hardware-based): Geomagnetic field strength along the x-,y- and z-axis inµT.

• Orientation (software-based): Azimuth (angle around the z-axis), pitch (angle around thex-axis) and roll (angle around the y-axis) in degrees.

• Proximity (hardware-based): Distance from object in cm.

Environment sensorsSome smartphones provides environment sensors to monitor various environmental properties suchas the humidity, illuminance, ambient pressure and ambient temperature.

• Ambient temperature: Ambient air temperature in ◦C.

• Light: Iluminace in lx.

• Pressure: Ambient air pressure in hPa or mbar.

• Relative humidity: Ambient relative humidity in %.

• Temperature: Device temperature in ◦C.

2.2.3 Smartphones in healthcare

Today’s smartphones are devices with advanced computing capabilities and connectivity which alsohave a wide range buil-in sensors. All of theses features make the smartphones a good platformfor telehealth.

-16-


Autodiagnosis and telemedicine

PokiDoc is like having a doctor in your pocket. It allows you to search for and compare informa-tion about health services and pricing based on location, condition or a doctor’s speciality.

Scanadu (available in 2014) is an app connected with a scanner device packet with sensors tocheck the temperature, heart rate, oximetry, ecg wave, heart’s beats, pulse, urine analysisand stress. The app keeps the report with all the information and it contacts to the doctorin case of emergency.

uCheck analyzes the urine. Users have to purchase a kit containing urine test strips that can bevisually analyzed with the iPhone’s camera.

Mango Health lets patients monitor their use of medications by setting up schedules for tak-ing their medication. When it is time to take the medications, it reminds them throughnotifications.

Figure 6: Scanadu, app and sensor for autodiagnosys

Patient Montiroing

Asthmapolis is connected with a sensor that patients can place on the inhaler. The app collectsall the data and information. It is used by doctors to monitor asthma symptoms and createthe patients report.

Dr. Diabetes provides diabetes awareness, monitoring and management to patients with chronicillness. It provides medical data (via the cloud) to physicians for accurate diagnosis.

AirStrip allows doctors check in on patients and review their vitals, cardiac waveforms, medica-tions, intakes and outputs, and allergies. The phone is connected to a bedside monitor andsend the collected data to doctors or caregivers through cellular or Wi-Fi connection.

-17-


GI Monitor is an app that helps patients with Crohn’s and Ulcerative Colitis track their symp-toms and provide accurate data to physicians for optimal treatment. Data is synchronizedacross all platforms in real-time and users can print out easy-to-read reports for their physi-cians.

RheumaTrack is a patient diary app which record pain on the VAS scale. Patients have to recordtheir pain, use of medication and activities. All this information is useful for the doctors.

Figure 7: Asthmapolis, app and sensor to monitor asthma patients

2.3 Kinect

The Kinect is a motion sensing device developed by Microsoft. It was released to enable users tointeract with the Xbox 360 console using gestures and spoken commands. One year later, Microsoftreleased the SDK to allow developers to write Kinecting apps on Windows. Other open sourceSDKs were also released. Nowadays there is a big community of developers and researchers aroundthe world.

Figure 8: Microsoft Kinect Sensor

2.3.1 Hardware

The Kinect is a device developed by Microsoft that combines different sensors for a touch-freegaming experience among other research uses:

-18-


• RGB camera.

• Depth sensor (also know as depth camera) with an infrared (IR) projector and camera.

• Microphone array with 4 directional microphones.

• Triaxial accelerometer.

Figure 9: Microsoft Kinect Sensor

There are two types of Kinect Hardware: Kinect for Windows and Kinect for XBox 360. Kinectfor Windows is specifically designed to be used with computers. It is licensed for commercial appdistribution, which makes it the best option for development. Kinect for Xbox 360 was designedand tested for the console but can also be used on development with some limitations.

The features unique to the Windows version of Kinect include:

• A shortened USB cable to ensure reliability.

• A small dongle to increase compatibility with other USB devices.

• Expanded skeletal tracking abilities, including seated skeletal tracking that focuses on 10upper body joints.

• A firmware update called Near Mode that allows the depth sensor to accurately track objectsand users seated as close as 40 cm from the device.

• Expanded speech recognition including four new languages: French, Spanish, Italian, andJapanese.

• Language packs improving speech recognition for many other English dialects and accents

• Improved synchronization between color and depth streams.

• Improved face tracking.

-19-


2.3.2 Features and limitations

The Kinect’s image, audio, and depth sensors allow to detect movements, identify faces and rec-ognize speech of players. However they have some physical limitations, like the sensing range.On the other hand, Kinect for Windows SDK frameworks also has some limitations such as thenumber of tracking skeletons.

Figure 10: Features and Limitations

• RGB camera: horizontal angular field of view 57◦ (plus 27◦ up or down with the motorizedpivot) and vertical field of view 43◦.

• Depth sensor: viewing distance range from 0.8 m to 4m. Practical limits are from 1.2m to3.5m.

• Depth near mode: viewing distance from 0.4m to 3m.

• Audio beam forming: angular field to identify the current sound source of 100◦ in intervalsof 10◦.

• Skeleton tracking: normal mode with 20-joints per player and seated mode with 10-joints.Both modes have simultaneously tracking up to six people including two active players (mo-tion analysis and feature extraction of 20 joints per player).

• Interactions: library with basic gestures (e.g. targeting and selecting with a cursor) whichalso supports the definition of gestures.

-20-


• Face tracking: angle limits to track face movements are ±45◦ (yaw), ±90◦ (roll) and ±25◦

(pitch).

2.3.3 Development

The Microsoft Kinect is a set of sensors developed as a peripheral device to use with the Xbox 360gaming console. Since it was released, hackers immediately saw potential in the device far beyondit and created open source libraries to use in other applications. Microsoft also released the officialSDK and the Kinect for Windows, a more powerful device to use in research. Nowadays there isa big community of developers and researchers around the world and several new applications areemerging.

Currently, both SDKs co-exist. It is not clear which SDK is better, the Microsoft SDK isthe official and supported SDK. However, the open source community is very active since theyreleased OpenNI (APIs, libraries, demos) and OpenKinect (drivers). Both have advantages anddisadvantages.

Microsoft SDK 5 OpenNI + OpenKinect (drivers) 6

Adva

nta

ges

-Strong documentation and support -Flexibility. No limitations.-Quick and easy installation. Fast learn-ing curve

-Multiplatform: Windows, MacOS, Linux

-Matlab integration (integrated onR2013)

-Programming languages supported:Phyton, C, C++, C#, Java

-Fast time of response. Small trigger de-lay

-IDEs: Visual Studio, XCode, G/Gcccompiler...

-Support skeleton tracking. Track up to20 joints. Stable.

-Open Source distribution licence

-Kinect’s tilt supported

Dis

adva

nta

ges

-Limitations from Microsoft and flexibil-ity

-Tricky installation and confusing docu-mentation

-Platform dependence: Windows 7 or 8 -Integration with IDEs. Set up needed-Programming languages supported:C/C++, C#

-Integration with Matlab and other li-braries and packages

-IDE: Visual Studio -Slow time of response. Trigger delay.-Microsoft Distribution license -Support skeleton tracking. Track up to

3 joints, less stable than the official SDK-Drivers don’t support the Kinect’s tilt

Table 3: Comparison of the Kinect SDKs

2.3.4 Kinect in healthcare

Microsoft’s Kinect has transformed the way people interact with technology. The combination ofthe camera, the depth sensor and the microphone enables new applications and functionalities.In the medical field it offers great possibilities to the treatment and prevention of disease, illness

-21-


or injury. Most of medical applications, specially in eHealth (healthcare through the use of tech-nology), use the Kinect for tracking patient’s movements in order to perform rehabilitation orpatient’s monitoring.

Physical Therapy, Rehabilitation and Remote Patient Monitoring

Games for Parkinson’s disease Red Hill Studios from University of California developed spe-cialized games to improve the gait and balance of people with Parkinson’s disease.

VirtualRehab This commercial solution implements a virtual platform with games and specificexercises for multiple sclerosis rehabilitation.

Home Training System for Rehab Patients(HTSRP) HTSRP motions and captures the phys-ical exercises of the patient using a Kinect. It presents a 3D version of the person, analyzesthe movements to pre-detect if physical exercises are performed correctly or incorrectly andgives visual feedback on the screen.

Jintronix Like HTRSP and other companies, Jintronix, performs rehabilitation in a virtual en-vironment.

Breathing Monitor ZDNet Health is researching about using the Kinect as a breathing monitor.

Figure 11: Rehabilitation with Kinect

Telemedicine

inAvate Microsoft’s InAVate is a platform which allows group therapy session using avatars. Theplatform allows remote access to patients in rural communities.

Collaboration and Annotation of Medical Images (CAMI) University of Arcasas uses Mi-crosoft Kinect to capture medical images and data, which can be checked on a Windowsdevice such as a tablet, phone or computer.

-22-


Robotic Telemedicine System University of South Florida developed a Robotic TelemedicineSystem using the kinect sensor for mapping the environment and plan a path and trajectory.The telemedicine platform also provides videoconference for doctor-patient communicationthrough the Kinect’s integrated camera and microphone.

Figure 12: InAVate for group therapy sessions

Other medical applications

Hands free tool for X-Ray and MRI control Toronto’s Sunnybrook Hospital uses Kinect asa hands free tool to move and zoom X-Ray and MRI without touching the screen and leavingthe sterile area. This solution is very useful for surgeons and interventional radiologist.

Hands free tool on surgery and radiology Getstix has developed a touchless gestural inter-face for the surgeons and interventional radiologists.

System to avoid obstacles for blind people Researchers from University of Konstanz havemade a step forward with NAVI system. They use the Kinect camera and depth sensor witha laptop, a vibrating bell and a bluetooth headset to avoid obstacles for blind people.

Figure 13: Hands free tool to move and zoom X-Ray and MRI

-23-


2.4 Fall detection systems

2.4.1 Commercial products

Currently, there are very few fall detection products. Today fall detectors are small devices whichclip onto the wearer’s belt or waistband and contains sensors and microprocessors to identifyfalls. Other solutions are based on Personal Emergency Response Systems (PERS) or differentenvironment and motion sensors.

Visonic [6] Amber is the family of systems and accessories to provide personal emergency re-sponse and safety to the user. It combines PERS systems with other monitoring and ambientsensors.

Guardian Medical Monitoring [2] Guardian’s Medical Alarm System is a hands-free, personalhelp button designed to get help for seniors by simply pushing the help button. The systemcan be combined with a monitored camera system.

Tynetec [5] Their solution enables passive monitoring inside homes using a large set of telecaresensors, such as sensors deployed with a chair pad, bed pad or floor mat.

Healthsense [3] They use contact and pressure sensors placed on bed, toilet and chairs for dailyactivity and fall detection.

BioSensics [1] PAMSys is the fall detector solution developed by BioSensics. It consists of anecklace wearable device which contains sensors and microprocessors to recognize the changeof orientation and the impact of the fall.

Figure 14: Visonic’s Personal Emergency Response solution

2.4.2 Android apps

Several apps have been developed in order to detect falls on Android-based smartphones. All theseapps acquire and analyze data from motion sensors to detect falls. Table 4 compares the numberof downloads and rating based on user’s reviews.

-24-


App Company/Developer Num. Downloads RatingSmart Fall Detector Hamideh Kerdegari +1.000 4.0Fall Detector Spantec +1.000 3.8iFall FSU Mobile Solutions +1.000 3.6Fade: Fall detector ITER S.A. +500 -Fall Monitor Eric Curtin +100 -

Table 4: Comparison of Android apps for fall detection.

Figure 15: Smart Fall Detection app for Android

2.4.3 Current research

Besides the commercial products, a variety of research is focused on detecting falls. Existingacademic research bases fall detection systems in different approaches and it can be categorized intofour main categories: wearable devices, ambient analyzers, motion detectors and hybrid systems.

Wearable devicesWearable sensors measure body’s acceleration and orientation as an indicator for falls. Existingsolutions mainly use accelerometers for detection, since falls are usually characterized by largeaccelerations. However, focusing only on large acceleration can result in many false positives fromfall-like activities such as sitting down quickly and jumping. Some fall detection systems usegyroscopes and other motion sensors to detect changes of body orientation, assuming that fallsend with a person lying horizontally on the floor.

Washable Low-Cost Garment Niazmand et al. in [23] system consists of a washable pulloverwith eight integrated accelerometer sensors to detect the motion of the torso and the upperbody. The fall is detected based on certain thresholds in the geometric main value of theacceleration for all sensors and locations.

Triaxial Accelerometer Based Fall Detection Hou et al. in [17] proposed a fall detectionsystem based on features abstracted from a tri-axial accelerometer (signal vector magnitude(SVM) of the acceleration, average, titling angle, etc).

-25-


SAFE (SmArt Fall dEtection system) Ojetola et al. in [24] used two sensor nodes placed onthe torso and thigh. A vector machine algorithm determine if a fall has taken place basedon the accelerometer and gyroscope’s acquired data.

Fall Detection System Based on Wireless Body Area Networks Baek et al in. [8] pro-posed fall detection system combining a tri-axial accelerometer and gyroscope. The deviceis placed on the necklace and the axis don’t rotate. A threshold-based detection algorithmuse the acceleration and angular velocity acquired for each axis.

Triaxial Accelerometer and Barometric Pressure Sensor The Tolkien et al in. [31] systemuses a tri-axial accelerometer and a baromtetric pressure sensor for fall detection and deter-mination of the direction of a fall. Data readings from the pressure sensor are used to verifythe fall based on a pre-defined threshold.

Shimmer Wearable Sensor System Shimmer [4] is a sensor which allows simple capture, trans-mission, processing and display of body sensed data in real-time. It provides equipment,supporting tools and applications for the development of body sensor solutions. It is usedin research to collect data for datasets, test the designed fall detection algorithms and verifythe systems.

Figure 16: SAFE (SmArt Fall dEtection) System with Shimmer sensor nodes

Smarpthones: the next generation of wearable devicesMost of the current smartphones integrate a wide range of motion sensors which makes them aneffective and cost-effectiveness solution for fall detection. The accuracy of the sensors is not asgood as the specialized wearable sensors but it is also possible to detect if a fall has taken placeby integrating the data from different sensors such as the accelerometer, gyroscope or compass.

iFall Sposaro et al in. [29] developed an Android app to detect falls based on the tri-axial ac-celerometer. The fall is considered detected if the magnitude of the acceleration exceeds bothhigh and low thresholds. The thresholds are defined based on user provided parameters suchas heigh, weight, and level of activity.

E-FallD Cao et al in [9] used the same technique of iFall system but the thresholds are based onthe heigh, weight, age and sex. These factors change the sensitivity of the system.

-26-


PerFallD Dai et al in. [12] combined the data from the tri-axial accelerometer and the orientationsensor for most reliable fall detection. The algorithm is based on thresholds on the totalacceleration and the vertical acceleration, calculated with the acceleration and orientationreadings.

Fall Detection and Direction Viet et al in. [33] uses the acceleration readings to detect falls.It also uses the orientation data to determine if the fall was forward, backward or aside. Thisapp requires the phone to be upright for correct calibration.

Figure 17: Smartphone based approach with data from accelerometer and gyroscope

Ambient analyzersAn ambient sensor approach analyzes the data from different environment sensors. These falldetection techniques identify falls by analyzing the data from a wide range of ambient sensors suchas pressure, infrared, vibration, electrical field, magnetic field, etc.

Floor Vibrations and Sound Litvak et al in. [20] proposed a technique based on the vibrationand sound signals from an accelerometer and a microphone connected to the floor by a scotchtape.

Electric Near Field Rimminen et al in. [28] tracks people with a near-field imaging (NFI) floor.The sensor detects locations and patters of people based on the measured impedances andvibrations.

Infrared and Pressure Sensors Ariani et al in. [7] designed an algorithm that process infrarredand pressure sensor’s data. When the person experience a fall and can not move, neithermotion nor pressure is detected, as the subject is motionless on the floor, indicating the fallis detected.

Doppler Sensors Tomii et al in. [32] uses the data from two or three doppler sensors. Theacquired data is used to calculate the spectogram and other useful parameters. A supervector machine (SVM) algorithm determine if the fall occurred.

-27-


Figure 18: Fall detection solution based on infrared and pressure sensors

Motion detectorsVision systems track people’s movements in order to detect falls. Video analysis, visible or infrareeddepending if they use video cameras or depth sensors, are used to analyze the person’s postureand the sudden change of position.

Multiple Cameras Thome et al in. [30] proposed a system with several cameras. The videofrom each camera is analyzed. A fall is considered detected if each independent processingcamera detect the fall.

Low-Cost PC camera Lee et al in. [19] system uses a low-cost PC camera for multiple objectdetection, person tracking and fall detection. The system tracks multiple people and com-putes their heigh and with. If the difference of their heigh and width between two framesexceeds certain threshold, the fall is considered detected.

Multi-omnidirectional Cameras Demiroz et al in. [14] use multi-omnidirectional high-resolutioncameras. It use image segmentation and Bayes methods to detect falls.

Wearable Embedded smart camera Ozcan et al in. [26] use a wearable camera to monitorall the daily activities that the subject is doing. The activity classification and fall detectionalgorithm is based on the histograms of edge orientation and strengths.

Waist-Mounted Sensor and Zigbee Network Chen [11] designed a system based on a Zigbeenetwork. The waist node is connected with the other nodes installed on the room to estimatethe location and position of the individual. If the person is lying down, the fall is detected.

-28-


Figure 19: Multiple people tracking to and fall detection

Kinect sensor: the new motion detectorThe Kinect sensor features a RGB camera, a depth sensor and a multi-array microphone. Thedata collected with the video and/or depth camera could be used to detect falls.

Tracking Joints with the Skeleton mode Kawatsu et al in. [18] developed a sytem to detectfalls by tracking some joints of the individual. The development is based on the skeletonmode (included on the Microsoft SDK) to track joints and suddenly change of their positions.

Human Silouettes with the RGB and Depth Camera Ozcan et al in. [25] utilizes the RGBand depth readings from the Kinect to capture human silouettes. The detection algorithmis based on the silhouette data, depth histogram, bounding box, distribution of average andhighest depth values.

Head Detection using the RGB Camera Nghiem et al in. [22] analyzed the depth readingsto track the head and other joints. The algorithm is based on the speed of the head plus thebody centroid and their distance to the ground to detect falls.

Changes in the Bounding Box using the Depth Sensor Mastorakis et al in. [21] proposeda solution tracking person’s movements using the depth sensor. The algorithm is based ondetect suddenly changes in the heigh and weight of the user’s bounding box.

-29-


Figure 20: Fall detection system using the Kinect sensor

Hybrid systemsHybrid systems combines two or more different techniques in order to provide better solutions interms of accuracy and efficiency.

Motion, Sound and Visual Perceptual Components Doukas et al in. [15] utilizes wearablesensors, video cameras and microphone arrays. The detection of falls is based on audio dataprocessing, sound directionally analysis, motion information, and subject’s visual location.

Motion and Acoustic Detection Popescu et al in. [27] combined a motion detector and thesound height information (array of microphones) to detect falls. If a fall occurred, the motionanalysis gives an alarm which will be confirmed if the sound comes from a source located ata height lower than 60cm.

3D ToF Camera and Wireless Accelerometer Grassi et al in. [16] use 3D Time-of-Fligh(ToF) cameras and wearable wireless accelerometers. The accelerometer sends to a computera possible fall alarm that will be confirmed using the video collected with the ToF camera.

Sensor Network and Home Robots Della Toffola [13] uses different kinds of sensors and arobot for fall detection. All sensors nodes are connected using a wireless radio network. If thefall is detected, the robot enables mobility inside the home environment while communicatingwith the external world.

Figure 21: Sensor network and home robots for fall detection

-30-


3 Design

The proposed fall detection system combines and Android-based smartphone and the MicrosoftKinect sensor. Figure 22 shows the architecture of the system, integrating both devices. Thesmartphone and the Kinect communicate to each other through a server running on the computer.When a fall is detected, the emergency alert is sent to a cellphone using the cellular network.Figure 22 also shows the architecture of the smartphone. Our system uses three sensors (ac-celerometer, gyroscope and magnetometer) to collect the data. It also uses three connectivitymodules: A-GPS (location), WiFi (connection to the laptop and the Kinect), and 3G/4G RFsystem (send the emergency alert).

Figure 22: Architecture of the system and the device.

The phone acquires the data from motion and position sensors such as accelerometers andgyroscopes. This data is processed and analyzed on the phone, where the proposed algorithmdetermines if a fall has occurred. Once the fall is detected, the phone triggers the Kinect through aTCP server running on the computer. The Kinect could be used to verify the fall by motion analysisbased on skeleton tracking. However, our first approach uses the Kinect to take a picture of theperson lying down. This picture is sent to the phone through the server and it will be broadcastedwith the emergency alert. The alert is sent from the phone and consist of a multimedia message(MMS) with the picture of the person lying down, the time, and location (longitude, latitude andlink to google maps) where the fall occurred. Figure 23 shows the flowchart of the system.

-31-


Figure 23: Flowchart of the proposed fall detection system.

-32-


3.1 Smartphone

The smartphone runs several processes which can be divided into two blocks: the fall detectionand the emergency alert.

3.1.1 Fall detection block

This block contains all the processes to read the sensor’s data, process it and analyze it in orderto detect if a fall has taken place.

• Data acquisition: The raw sensor values are collected using the smartphone’s built-in sensorsat 50Hz (see section Preliminary experiments for more information).

– Acceleration (m/s2): the raw acceleration in the x-,y- and z-axis is acquired with thetri-axial accelerometer.

– Rotation (rad/s): the tri-axial gyroscope’s readings give the raw values of the angularvelocity in the x-,y- and z-axis.

– Orientation (rad): The orientation of the phone is determined by the azimuth (anglearound the x-axis), the pitch (angle around the x-axis) and the roll (angle around thez-axis). In our system we use a combination of the magnetic field and the accelerationfor a most reliable orientation readings.

• Data processing and feature extraction: Each sensor readings are filtered and processed toextract:

– Total acceleration (m/s2): the geometric mean value is calculated using (8).

|AT | =√a2x + a2y + a2z (1)

– Total rotation (degree/s): the phone’s coordinate-system is defined relative to the phoneand the axes are not swapped when the device’s orientation changes. Because of that,the geometric mean value is more useful (9).

ωT =√ω2x + ω2

y + ω2z (2)

– Triaxial orientation (degrees): The orientation of the phone, therefore the orientationof the person, is determined by the azimuth, pitch and roll angles eqrefeqOrient.

θx, θy, θz (3)

• Prediction and smoothing: The Kalman filter is a prediction and smoothing filter based ona linear weighed average of the values. The filter computes the average µn of the series giventhe first n values. When a new point is measured it recompute µn using (7), where xn+1

is the new value measured, µn is the old average, µn+1 is the predicted value and K = 1n+1

.Other prediction and smoothing filters were applied giving worse results than the Kalmanfilter (see section Preliminary experiments for more information).

µn+1 =n

n+ 1µ+

1

n+ 1xn+1 = 1 +K(xn+1 − µ) (4)

-33-


Detection algorithmFalls are usually characterized by changes in motion and body position of an individual. Figure 30shows the characteristic pattern of these values collected when an individual is falling. The typicalacceleration pattern when a person falls is a combination of free fall, high impact and dampingeffect. The rotation, collected with a gyroscope, is characterized by large values with a high peakat the moment of impact. The fall usually ends with the person lying horizontally on the floor,which means a change of 90 degrees in the orientation x- or y-axis.

Figure 24: Typical patterns of total acceleration, rotation and orientation.

The proposed detection algorithm uses the acceleration and orientation readings. It assumesthan a fall end with the individual lying on the floor, which means a change of 90 degrees inthe x- or y-axis. The fall is detected if the acceleration readings exceeds both thresholds (highand low) within a triggering window and there is a change of 90 degrees in the orientation. Thefollowing pseudo-code shows the detection algorithm. Different algorithms were proposed but ourfinal choice give us better results in terms of accuracy (see section Preliminary experiments formore information).

Listing 1: Pseudo-code of the proposed detection algorithm

//At = total acceleration

//th_l, th_h = low, hight threshold (different for each phone location (chest,

waist, pocket)

within 2s window {

if (At<th_l)

freeFallDetected=true;

if (At > th_u)

impactDetected=true;

if (maxOrient[axis] - minOrient[axis])

changeOrientDetected = true

}

if (freeFallDetected && impactDetected && changeOrientDetected)

"FALL DETECTED"

-34-


The detection algorithm defines thresholds on AT and a change of 90 degrees in the orientation.The acceleration and orientation readings on the different phone locations are slightly different sowe define different thresholds for each location (chest, waist and pocket). Table 6 shows the highand low thresholds for the position of chest, waist and pocket.

Location Thh Thlchest 1.7G 0.8Gwaist 1.3G 0.7Gpocket 1.4G 0.6G

Table 5: Thresholds for different phone locations.

3.1.2 Emergency alert block

The emergency alert block composes the alert message and sends it to the emergency contact.

• A-GPS: The assisted GPS uses the GPS, cell-ID and Wifi location sources to determine thelocation of the device, therefore the location of the individual who has fallen.

• Compose SMS/MMS: If the Kinect is not connected, it composes a text message (SMS)including the alert message, the time when the fall was detected and the location obtainedwith the A-GPS, which consist on the latitude, the longitude and a link to google maps. Ifthe Kinect is connected, it compose a multimedia message (MMS) also including the pictureof the individual lying down.

• Broadcast alert: The alert message is sent to the emergency contact through the cellularnetwork.

3.2 Server

The server connects the smartphone and the Kinect to allow communication between them. It isa TCP server and runs in the same computer where the Kinect is connected. The phone connectsto it through a WiFi network. Figure 30 shows the processes and message exchange between theAndroid, the server and the Kinect.

As figure 30 shows, the server is always running on the computer. It has a socket opened andlistening until the smartphone connects to it using the IP address and port number. If the serverreceives the message ”fall detected” it triggers the Kinect, which takes a picture of the individuallying down. Once the picture is taken, the server sends it to the phone and closes the connection.The Kinect is also switched off. This message and information exchange includes all the protectionsand security protocols to avoid undesired connections.

-35-


Figure 25: Smartphone-Kinect communication through a TCP server.

3.3 Kinect

The Kinect sensor is used to take a picture of the person lying down, which will be sent withthe alert to the emergency contact. When the server triggers the Kinect, it runs the followingprocesses.

• Start the Kinect: It start the device which will be in standby until the camera (or any othercapability) is enabled.

• Start the RGB camera: It starts the RGB camera and enables the RGB stream.

• Take picture: Once the camera is on and the stream is enable, it captures one of the frames.The frame is stored in a temporary location before it is sent to the smartphone.

• Stop the RGB camera and the Kinect: Once the server receives the picture, it turns off theKinect and disables the RGB stream and the camera.

In our system we only use the Kinect to take a picture of the individual lying down. However,Kinect’s tracking capabilities could be used to verify the fall through motion analysis of somejoints. This verification would improve the accuracy of the system. Other functionalities andcapabilities of the device could be used to improve the system (see section Discussions for moreinformation). This extra functionalities were rejected due to intrinsic limitation of the sensor thateffected the reliability of the system.

-36-


4 Preliminary experiments

Preliminary experiments were performed to set up all the parameters of the fall detection system.

4.1 Which sensors to use?

The Android built-in sensors measures motion, position and environmental conditions. In order todetect falls, we monitored the device movement and positioning. Figure 26 shows the acceleration,linear acceleration, rotation and orientation raw data collected while a person is walking andsuddenly falls down.

5 6 7 8 9 10

0

10

20

time (s)

Acc. (m

/s2)

Acceleration

ax

ay

az

5 6 7 8 9 10

0

10

20

time (s)

Acc. (m

/s2)

Acceleration (geometric mean value)

5 6 7 8 9 100

20

40

time (s)

Acc. (m

/s2)

Linear Acceleration

gx

gy

gz

5 6 7 8 9 100

20

40

time (s)

Acc. (m

/s2)

Linear Acceleration (geometric mean value)

5 6 7 8 9 10−300

−200

−100

0

100

time (s)

Rot. (

degre

es/s

) Rotation

ωx

ωy

ωz

5 6 7 8 9 100

200

time (s)

Rot. (

degre

es/s

) Rotation (geometric mean value)

5 6 7 8 9 100

200

time (s)

Or.

(degre

es)

Orientation

yaw (θz)

pitch (θy)

roll (θx)

5 6 7 8 9 100

500

time (s)

Or.

(degre

es)

Orientation (geometric mean value)

Figure 26: Sensor readings while an individual is walking and suddenly falls down.

Acceleration The raw acceleration data is collected using the tri-axial accelerometer, which is ahardware-based sensor. To detect a fall we calculate the geometric mean value using (8).

|AT | =√a2x + a2y + a2z (5)

Linear acceleration The linear acceleration is the acceleration force that is applied to the deviceon all three physical axes excluding the force of gravity. This software-based sensor derivesthe data from the accelerometer and the gravity sensors which at the same time derivethe data from the accelerometer and the magnetometer or gyroscope. Because of the erroraccumulation from all the sensors, the raw data from the accelerometer is more reliable thanthe linear accelerometer.

-37-


Rotation The rotation, or angular velocity, is collected using the hardware-based tri-axial gyro-scope. Since the phone’s coordinate-system is defined relative to the phone and the axes arenot swapped when the device’s orientation changes, the geometric mean value (6) is moreuseful.

ωT =√ω2x + ω2

y + ω2z (6)

Orientation The orientation of the phone is determined by the azimuth (angle around the z-axis),the pitch (angle around the x-axis) and the roll (angle around the z-axis). The orientationsensor was deprecated in Android 2.2 so there are a few alternatives to acquire device ori-entation. The easiest alternative is by using the rotation vector, a software-based sensorwhich derives the data from the magnetic field and the accelerometer. However, not allAndroid phones have this sensor so an alternative is manually integrating the data from themagnetometer and the accelerometer. Another alternative consist of a low-pass filtering ofthe accelerometer and magnetic field sensor signals and high-pass filtering of the gyroscopesignals. This avoid both gyro drift and noisy orientation obtained with the accelerometerand magnetometer.

4.2 Which prediction mode to use?

Once the data is collected, different prediction modes are applied. Figure 27 shows the effects offour filters on the geometric value of the acceleration.

5 5.5 6 6.5 7 7.5 8 8.5 9 9.5 10 10.5

0

10

20

time (s)

Acc.(

m/s

2)

Polynomial filter on At

Real

Polynomial

5 5.5 6 6.5 7 7.5 8 8.5 9 9.5 10 10.5

0

10

20

time (s)

Acc. (m

/s2)

Median filter on At

Real

Median

5 5.5 6 6.5 7 7.5 8 8.5 9 9.5 10 10.5

0

10

20

time (s)

Acc.(

m/s

2)

Low pass filter on At

Real

Low pass

5 5.5 6 6.5 7 7.5 8 8.5 9 9.5 10 10.5

0

10

20

time (s)

Acc. (m

/s2)

Kalman filter on At

Real

Kalman

Figure 27: Effects on At for different prediction modes.

Median filter The filter predicts new values based on the median value of the series.

-38-


Polynomial filter It predicts new values based on finding the coefficients of a polynomial ofdegree n that fits the data.

Low pass filter It consists of a 3r order Butterword IIR low-pass filter with a cutoff frequency4Hz, since most of the energy contained while walking is below 3Hz.

Kalman filter This prediction and smoothing filter is based on a linear weighed average of thevalues. The filter computes the average µn of the series given the first n values. When a newpoint is measured it recompute µn using (7), where xn+1 is the new value measured, µn isthe old average, µn+1 is the predicted value and K = 1

n+1.

µn+1 =n

n+ 1µ+

1

n+ 1xn+1 = 1 +K(xn+1 − µ) (7)

Based on the results of Figure 27 and Figure 29 we determined the median Filter and Kalmanfilter are good approaches to predict future sensor readings. Figure 28 compares the false alarmsand miss detections of the two prediction filters for different sample frequencies and position ofthe phone. The results for the location of the pocket had been removed to make the plot clear.The obtained results were similar than other locations, although the rate of false alarms and missdetections was higher.

Since we suggest the use of the Kinect to verify falls, lower miss detections are more importantthan higher false alarms. Based on this principe, Kalman filter gives us a better approach in oursolution.

20 40 60 80 1002

4

6

8

10

12

14

16False Alarms

% F

A

sample rate (Hz)

waist (Kalman)

waist (Median)

chest (Kalman)

chest (Median)

20 40 60 80 1002

4

6

8

10

12

14Miss Detections

% M

D

sample rate (Hz)

waist (Kalman)

waist (Median)

chest (Kalman)

chest (Median)

Figure 28: Evaluation of the prediction mode at different sample rates.

4.3 Which sample rate to use?

The election of the sample rate determines the accuracy of the system and also affects on thebattery life of the smartphone. As Figure 29 shows, higher sample rates generate less error duringthe prediction. However, in terms of saving battery, lower sample rates are better: collect datafrom sensors every 40ms (25Hz) waste less battery than every 10ms (100Hz).

In order to determine the best sample rate for our system, we ran the same algorithm atdifferent sample rates. As you can see in figure 28 , higher sample rates generate better results.

-39-


Based on all of these experiments and due to battery concerns, we select a sample rate of 50Hz.This sample rate offers good results in terms of miss detections, false alarms and battery saving.

0 20 40 60 80 1000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Sample Rate (Hz)

Err

or

PolyfitKF

MedianLowpass

Figure 29: Error when applying different filters on At at different sample rates.

4.4 Which algorithm to use

Falls are usually characterized by changes in motion and body position of an individual. The usefuldata to determine is a fall has taken place is the acceleration, orientation and rotation. Figure 30shows the characteristic pattern of these values collected when an individual is falling.

The typical acceleration pattern when a person falls is a combination of free fall, high impactand damping effect. The rotation, collected with a gyroscope, is characterized by large values witha high peak at the moment of impact. The fall usually ends with the person lying horizontallyon the floor, which means a change of 90 degrees in the orientation x- or y-axis. All proposedalgorithms are based on identifying these patterns.

Figure 30: Typical patterns of total acceleration, rotation and orientation.

Algorithm 1 This algorithm was proposed for Dai et al in. [12]. The algorithm uses theacceleration readings in directions of x-, y-, and z-axis to calculate the total acceleration of

-40-


the phone (8).

|AT | =√a2x + a2y + a2z (8)

It also uses the yaw, pitch and roll values collected with the orientation sensor to obtain theamplitude of the acceleration at the absolute vertical direction (9).

|Av| = |Ax sin θz + Ay sin θy + Az cos θy cos θz| (9)

This fall detection algorithm is based on the values of AT and Av . We calculate the amplitudebetween the minimum and the maximum readings of the acceleration within a triggeringwindow. If there is a fall, these values correspond to the free fall and high impact patterns.If this difference exceeds a defined threshold, we calculate the amplitude between the nexttwo peaks while the damping effect is taking place. If this difference is high than anotherthreshold a fall is considered detected. The same rule applies to Av but the defined thresholdsare different.

Algorithm 2 It is a threshold-based algorithm on the total acceleration AT (8) and rotation ωT

(10), collected with the accelerometer and the gyroscope. The fall is considered detected ifthe acceleration reading exceeds the high and low defined values within a triggering windowand the rotation also exceeds its defined threshold.

ωT =√ω2x + ω2

y + ω2z (10)

Algorithm 3 This algorithm uses the acceleration and orientation readings. It assumes than afall ends with the individual lying on the floor, which means a change of 90 degrees in thex- or y-axis. The fall is detected if the acceleration readings exceeds both thresholds (highand low) within a triggering window and there is a change of 90 degrees in the orientation.

Figure 31 shows the evaluation of the three algorithms when the phone is placed in differentlocations: chest, waist and pocket. Algorithm 3 performs better results in terms of false alarmsand miss detections.

waist chest pocket0

5

10

15False alarms

% F

A

A1

A2

A3

waist chest pocket0

5

10

15Miss detections

% M

D

Figure 31: Comparation of the proposed algorithms for different phone locations.

-41-


4.5 Which thresholds to use?

The proposed algorithm (Algorithm 3) defines thresholds on AT and a change of 90 degrees inthe orientation. The rate of false alarms and miss detections is lower than the other proposedalgorithms for all phone locations but these rates can be better. The acceleration and orientationreadings on the different phone locations are slightly different; defining different thresholds for thesethree locations (chest, waist and pocket) the results will be better. We adjusted the thresholdsin order to reduce false negative while simultaneously keep false positive in an acceptance range.Figure 32 shows the ROC curve for different thresholds Thh when the Thl is fixed at 0.6G, whereG = 9.81m/s2. Data is from backward falls and the phone is placed at the position of chest.The same evaluation was done for the position of waist and pocket obtaining the high and lowthresholds showed in Table 6.

Figure 32: ROC curve for different thresholds Thh when the Thl is fixed at 0.6G (G = 9.81m/s2). Phoneis placed at chest.

Location Thh Thlchest 1.7G 0.8Gwaist 1.3G 0.7Gpocket 1.4G 0.6G

Table 6: Thresholds for different phone locations.

4.6 Where to trigger the Kinect?

There are two ways to use Kinect on the computer: the official Microsoft’s SDK and the opensource version OpenNI. The Microsoft SDK includes several features not included in OpenNI andthe installation and support is better. However, OpenNI is multi platform (Windows, MacOS andLinux) and apps can be programmed in several languages, whereas the official SDK only supportsC++, C# and Visual Basic on Windows. Both SDKs can be used on Matlab: OpenNI requiresthe installation of a set of C++ wrapper functions whereas the Microsoft’s SDK is integrated on

-42-


the new Matlab’s release. The official SDK has better response time so it would be a better chancefor our system.

Figure 33 compares the delay of the different SDKs when we use different methods to triggerthe Kinect. The first experiment triggers the sensor from Matlab by starting a Kinect application,which is an executable file. Next experiment triggers the sensor from Matlab using the integratedfunctions. The third approach triggers the device from the server using the compiled file. Basedon these experiments, we determined to trigger the Kinect from the server using the MicrosoftSDK, which give us better results in terms of delay.

Matlab (exe) Matlab (int.) Server(comp.)0

0.5

1

1.5

2

2.5

3

3.5

4

tim

e (

s)

Microsoft SDK

OpenNI

Figure 33: Comparation of the delay when triggering the Kinect.

4.7 Whether compute the data?

The phone acquires the data from the sensors. Once the data is acquired, it can be analyzed inorder to identify fall patterns in both phone or computer.

Thefirst approach computes all the data on the computer. The phone acquires the datafrom the sensors and it sends it to the computer over a TCP server. Matlab reads the values andanalyzes them to identify if a hall has taken place. If a fall is detected, it triggers the Kinect. Basedon preliminary experiments, we use the official SDK from Microsoft and we trigger the Kinect fromMatlab due to delay concerns.

Thesecond approach analyzes the data on the phone. The processing, prediction and algo-rithm are performed on the phone. If the fall is detected, it connects to the computer to triggerthe Kinect. In this case, we trigger the kinect from the server using the Microsoft SDK which giveus better results in terms of delay.

Figure 34 compares the delay of both approaches at different sample rates. The total delay isdivided into the time it takes to analyze the data (tprocess), connect to the server (tconnection), sendthe data (ttransmission) and trigger the Kinect (ttriggerKinect). Based on this results, we decided tocompute the data on the phone, and trigger the Kinect if a fall is detected.

-43-


C S 25Hz C S 50Hz C S 75Hz C S 100Hz0

1

2

3

4

5

6

7

8

9

10

tim

e(s

)

tconnection

ttransmission

tprocess

ttriggerKinect

Figure 34: Comparison of the system delay to determine where to compute the data.

-44-


5 Development

5.1 Smartphone

Currently there are different mobile operating systems such as Android, iOS or Windows Phone.We chose to implement our system for Android due to the number of devices that use it, availabilityof resources and wide availability. We used the Google Nexus S but the app was developed to runon any Android device which has accelerometer, gyroscope and magnetometer regardless the sizeof the screen and other features.

5.1.1 Getting started

Android is an open source operating system released by Google. The Android SDK provides allthe resources necessary to develop applications, available on the official website 7. Android Studiois the new IDE to develop for Android but many other IDEs could be used such as Eclipse, themost used IDE to develop for Android. The section Appendix-2 Android Installation shows how tostart programming Android apps. It includes a tutorial on how to install the Android SDK, howto set up Eclipse and a quick guide to program and run the first app.

Java is the most used language to develop Android apps, although is also possible to codewith C/C++. Java is an object oriented language which incorporates a big set of libraries to helpdevelopers to build applications. The Android SDK includes many standard Java libraries for datastructures, maths, graphics, networking and special Android libraries to help developing Androidapplications.

Figure 35: Developing Android apps in Eclipse.

7Android Developer website http://http://developer.android.com/

-45-

http://http://developer.android.com/


The essential building blocks of an Android application are called application components.There are four types and each type serves a distinct purpose and has a distinct lifecycle:

Activities An activity represents a single screen with a user interface.

Services A service is a component that runs in the background to perform long-running operationsor to perform remote processes. A service does not provide a user interface. A service cantake two forms:

Started Once started, a service can run in the background indefinitely, even if the compo-nent that started is destroyed.

Bound It only runs as long as another application component is bound to it. It allowscomponents to interact with the service, send requests, get results...

Content providers A content provider manages a shared set of application data. The applicationcan access to the stored data (in the file system, a database, on the web...) through a contentprovider. It is also useful for reading and writing data that is private in the application.

Broadcast receivers A broadcast receiver is a component that responds to system-wide broad-cast announcements such as when the battery is low or the screen has turned off. They arealso used to exchange message between classes.

Besides the application components, there are other classes defined in Android very useful todevelop applications.

Intent An intent is an object that launch Activities, Services, BroadcastReceivers. It also can beused to communicate with a background Service.

Event Listener A listener is an object that listens, and takes actions upon certain events, forexample the accelerometer sensor.

Event Handler A handler is an object that handles certain things that client class doesn’t wantto deal with.

5.1.2 Hierarchy of classes

The Fall Detection Android app combines different types of application components to performall the functionalities. Figure 38 shows the hierarchy of classes.

• MainActivity: main user interface which contains the buttons and other visual elements.

• ProviderStatus: class to check if the GPS and the WiFi are enabled. It shows a notificationto enable them if they are disabled. This class makes use of the Location Manager and theConnectivity Manager, which are Android services already implemented.

-46-


Figure 36: Hierarchy of classes of the Android application.

• SensorsService: service that runs on the background and continuously reads the sensorsvalues, filters them with a Kalman filter and runs the algorithm to detect falls. It is anstarted service, which means that it runs in the background indefinitely. When a fall isdetected, the service notice the main activity through a broadcast receiver with the messagefall detected.

• SensorsManager: class that extends the sensor listener (provided by Android) to continuouslyacquire data from the sensors (Accelerometer, Gyroscope and Magnetometer). It process thesensor readings and calculates the total acceleration, rotation and tri-axial orientation.

• KalmanFIlter: class that filters the previous stored values with the new sensor readings topredict the new values through the Kalman algorithm.

• DetectFallsAlgorithm: class that implements the algorithm to detect falls. The algorithmuses the sensor readings after a processing, feature extraction and Kalman prediction.

• ConectToServer: started service that connects to the server through a WiFi network. The

-47-


class uses the IP and port number to connect to the server and start exchanging informationsuch as messages or a picture.

• SendEmergencyAlert: class that implements the functionalities of sending a SMS or MMS tothe emergency phone number. When a fall is detected and the picture is received, this classcompose the SMS/MMS message with the alert message, the current time and the location(including a link to open it on google maps). It uses an Intent to send the SMS or MMSthorough the cellular network.

• GPSTracker: bound service that implements the LocationListener to get the location of thedevice which consist of the longitude and the latitude. The listener uses the A-GPS, whichmeans it can use the GPS, wifi or cellular to get the location.

• PreferencesActivity: screen to set up the preferences and the settings of the system such asthe position of the phone, the IP of the server or the emergency phone number.

• HelpActivity: screen to show how to use the app.

• AboutActivity: screen to show the about section on developer.

5.1.3 Code implementations

The source files with the implementation of the classes are included in the zip file attached to thisthesis. However, the important codes are detailed below.

Read values from sensorsTo get the sensor readings the first step consist of obtain instances of the SensorManager and thesensors we want to access. Then, enable them and register the listeners to start collecting thedata. The following code shows how to register and get values of the accelerometer, gyroscope androtation vector.

Listing 2: Register sensors

private SensorManager mSensorManager;

private Sensor mAccelerometer, mGyros, mRotVector;

//Get instance from SensorManager

SensorManager mSensorManager = (SensorManager)

getSystemService(SENSOR_SERVICE);

//Get instance from sensors

mAccelerometer = mSensorManager.getDefaultSensor(Sensor.TYPE_ACCELEROMETER);

mGyros = mSensorManager.getDefaultSensor(Sensor.TYPE_GYROS);

mRotationVector = mSensorManager.getDefaultSensor(Sensor.TYPE_ROTATION_VECTOR);

//Register sensors

-48-


mSensorManager.registerListener(this, mAccelerometer,

mSensorManager.SENSOR_DELAY_NORMAL);

mSensorManager.registerListener(this, mGyros,


mSensorManager.registerListener(this, mRotationVector,


//Unregister sensors when the app is destroyed (onPause() methods)

mSensorManager.unregisterListener(this);

The onSensorChanged() method is used to get the sensor values, the timestamp, the typeof sensor, etc. In this method we copy the values from the accelerometer, gyroscope and rota-tion vector sensor, a virtual sensor to calculate the orientation from the magnetometer and thegyroscope.

Listing 3: Read sensor values

public void onSensorChanged(SensorEvent event) {

Sensor sensor = event.sensor;

//Accelerometer values

if (sensor.getType()==Sensor.TYPE_ACCELEROMETER) {

Ax = event.values[0]; //Copy each value into a independent variable

Ay = event.values[1];

Az = event.values[2];

}

if (senor.getType() == Sensor.TYPE_GYROS)

W = event.values.clone(); // Copy all values into the array (W[0], W[1],

W[2])

//Rotation vector uses the magnetometer and gyroscope to get the orientation.

if(sensor.getType()==Sensor.TYPE_ROTATION_VECTOR){

float[] rotationMatrix = new float[16];

SensorManager.getRotationMatrixFromVector(rotationMatrix, event.values);

SensorManager.getOrientation(rotationMatrix, mOrientValues);

for(int i=0; i<3; i++)

mOrientValues[i]=(float)

Math.toDegrees(mOrientValues[i])+180.0f;//orientation in degrees

}

Prediction with the Kalman FilterThere are several Java implementations of the Kalman filter. In our application we adapted theOne-Dimensional Kalman Filter included in the math java classes 8. Due to space limitations, the

8Kalman Filter on Java http://commons.apache.org/proper/commons-math/apidocs/apache/math3/

KalmanFilter.html

-49-

http://commons.apache.org/proper/commons-math/apidocs/apache/math3/KalmanFilter.html

http://commons.apache.org/proper/commons-math/apidocs/apache/math3/KalmanFilter.html


code of the Kalman filter is not shown. However, the parameters to adapt the filter in our systemand get a good prediction are detailed in previous sections.

Fall Detection AlgorithmThe detection algorithm defines a high and low thresholds on the total acceleration (At) for thedifferent phone locations. A fall is detected if both thresholds are exceeded and the orientationchanges 90 degrees. Due to space limitations, only a pseudo-code of the algorithm is shown.However, the implementation is quite easy following all the steps of the pseudo-code.

Listing 4: Pseudo-code of the proposed detection algorithm

//At = total acceleration

//th_l, th_h = low, hight threshold (different for each phone location (chest,

waist, pocket)

within 2s window {

if (At<th_l)

freeFallDetected=true;

if (At > th_u)

impactDetected=true;

if (maxOrient[axis] - minOrient[axis])

changeOrientDetected = true

}

if (freeFallDetected && impactDetected && changeOrientDetected)

"FALL DETECTED"

Connect to the serverThe application works as a client that connects to the server through its IP address and portnumber. This functionality is implemented as a started service running on the background. Thefollowing code shows how to open the socket and send/receive message. The code to handle theexceptions is not included but it must be implemented in the application.

Listing 5: Connect to the server and exchange messages

//create a socket to make the connection with the server

Socket socket = new Socket(serverIP, serverPort);

//create the input and output streams

DataOutputStream dataOutputStream = new DataOutputStream(socket.getOutputStream());

DataInputStream dataInputStream = new DataInputStream(socket.getInputStream());

//ACK

//Send a message

dataOutputStream.writeUTF("Fall Detected");

dataOutputStream.flush();

-50-


//Receive a message

message = inputStreamToString(dataInputStream);

//Close socket and streams

socket.close();

dataInputStream.close();

dataOutputStream.close();

//Convert input stream to string

private String inputStreamToString(InputStream is) {

BufferedReader r = new BufferedReader(new InputStreamReader(inputStream));

StringBuilder total = new StringBuilder();

String line;

while ((line = r.readLine()) != null)

total.append(line);

return total.toString();

}

To transfer a file, such as an image, from the server to the smartphone, a buffered stream haveto be created in order to copy the received bits to the stored file.

Listing 6: Transfer a picture

//Create the image file on imagePath location (sd card or internal storage)

FileOutputStream fOutputStream = new FileOutputStream(imagePath);

//Create stream to write the received bytes of the image on the created file

BufferedOutputStream BufOutputStream = new BufferedOutputStream(fOutputStream);

byte[] aByte = new byte[SizeFile];

int bytesRead;

int totalBytes=0;

//Receive the image

while (totalBytes < SizeFile) {

bytesRead = dataInputStream.read(aByte, 0, SizeFile);

totalBytes += bytesRead;

//Write to file

BufOutputStream.write(aByte, 0, bytesRead);

}

//Close the streams

BufOutputStream.flush();

BufOutputStream.close();

-51-


Send an alert SMS/MMSTo compose and send an SMS we use the class SmsManager, included in the Andorid API. TheIntent allows to perform the background activity to send an SMS or MMS.

Listing 7: Send a SMS or MMS message

private void sendSMS()

{

PendingIntent pi = PendingIntent.getActivity(mContext, 0,

new Intent(mContext, MainActivity.class), 0);

SmsManager sms = SmsManager.getDefault();

sms.sendTextMessage(phoneNumber, null, message, pi, null);

}

private void sendMMS()

{

String imagePath = Environment.getExternalStorageDirectory().getAbsolutePath()

+ "/KiFall/Image1.jpg";

Intent i = new Intent(Intent.ACTION_SEND);

i.putExtra("address",phoneNumber);

i.putExtra("sms_body",message);

i.putExtra(Intent.EXTRA_STREAM, imagePath);

i.setType("image/png");

mContext.startActivity(i);

}

Get the location from the A-GPSThe locationManager class allows to get the approximate location of the phone using the A-GPS(GPS, WiFi or cellular positioning). The first step consist of checking if the GPS or the WiFilocation are enabled through the content providers and then get the location. The following codeshows how to get the location. The parts to handle the exceptions and control the errors is notshown but it must be implemented.

Listing 8: Get the location

public Location getLocation() {

// Get location from Network Provider

if (isNetworkEnabled) {

locationManager.requestLocationUpdates( LocationManager.NETWORK_PROVIDER,

MIN_TIME_BW_UPDATES, MIN_DISTANCE_CHANGE_FOR_UPDATES, this);

// change LocationManager.GPS_PROVIDER to get the location from the GPS

provider

location =

locationManager.getLastKnownLocation(LocationManager.NETWORK_PROVIDER);

}

latitude = location.getLatitude();

-52-


longitude = location.getLongitude();

return location;

}

5.2 Server

The server runs on the computer where the Kinect is connected in order to allow interactionbetween the Android smartphone and the Kinect. The server is coded in C# to allow interactionwith the Kinect and uses the TCP protocol to communicate with the phone.

The server accepts an unlimited number of connections and it spawns a thread for each clientconnected. The server class creates a new TCPListener to accept socket communications at port3200. It creates a new thread for each client connection.

Listing 9: Start the listener

class Server

{

private TcpListener tcpListener;

private Thread listenThread;

public Server()

{

this.tcpListener = new TcpListener(IPAddress.Any, 3200);

this.listenThread = new Thread(new ThreadStart(ListenForClients));

this.listenThread.Start();

}

The server blocks until the smartphone connects. When the client is connected it creates a newthread to handle communication with it:

Listing 10: Listen for clients

private void ListenForClients()

{

this.tcpListener.Start();

while (true)

{

//blocks until a client has connected to the server

TcpClient client = this.tcpListener.AcceptTcpClient();

//create a thread to handle communication with connected client

Thread clientThread = new Thread(new

ParameterizedThreadStart(HandleClientComm));

clientThread.Start(client);

}

-53-


}

The following function allows the communicate the smartphone and the server. The serverwaits until the phone sends a message and then it sends a new message to it. After that, it closesthe communication. In our application, the smartphone and the server exchange several messages.

Listing 11: Send/receive a message (code to handle the errors has to be implemented)

//Read the data from the client

private void HandleClientComm(object client)

{

TcpClient tcpClient = (TcpClient)client; //start the client

NetworkStream clientStream = tcpClient.GetStream(); //get the stream of

data for network access

byte[] message = new byte[4096];

int bytesRead;

while (true)

{

bytesRead = 0;

//blocks until a client sends a message

bytesRead = clientStream.Read(message, 0, 4096);

//Write the message on the screen

ASCIIEncoding encoder = new ASCIIEncoding();

System.Console.WriteLine(encoder.GetString(message, 0, bytesRead));

//Reply

byte[] buffer = encoder.GetBytes("Hello Android!");

clientStream.Write(buffer, 0, buffer.Length);

clientStream.Flush();

}

tcpClient.Close();

}

}

The Kinect is used to take a picture which will be send to the smartphone through the server.The code below shows how to send a picture from the server to the phone.

Listing 12: Send a picture

//Send the image size

FileInfo FileInfo = new FileInfo(imagepath); //get info of the picture (size,

date, format...)

String ImageSizeStr = FileInfo.Length.ToString(); //image size

-54-


sw.Write(ImageSizeStr); //send: image size

sw.Flush();

//Send the picture

byte[] fBuffer = new byte[FileInfo.Length];

FileStream fs = new FileStream(imagepath, FileMode.Open, FileAccess.Read); //open

the file

fs.Read(fBuffer, 0, fBuffer.Length); //read byte from image

fs.Flush();

fs.Close();

clientStream.Write(fBuffer, 0, fBuffer.Length); //send the picture

5.3 Kinect

The Kinect sensor was developed by Microsoft. Altought it was a gaming peripheral, hackers sawpotential in the device far beyond it and created open source drivers and libraries. Currently, theofficial and the open source SDK coexists and they can be used in different IDEs and programminglanguages.

5.3.1 Getting started

Currently there are two Kinect SDKs: the Microsoft SDK and OpenNI. The section State ofthe Art, Kinect compares both SDKs, detailing the platforms, IDEs and programming languagessupported.

The Microsoft SDK requires Visual Studio to develop applications on C# language. It is onlysupported on Windows. The new Matlab’s release (R2013a) enables to use the sensor on Matlabwithoud installing additional libraries. Previous versions required to install and set up specificlibraries.

OpenNI is a multiplatform (Windows, MacOS, Linux) open source SDK. Several IDEs can beused to develop applications such as Visual Studio, XCode, GCC or Processing. There is also ahuge range of libraries and resources available online. The list of supported languages includesC/C++, #C, Java, Phyton and Ruby, among others. It is also possible to use the sensor on Matlabbut it requires to install a set of wrappers.

The section Appendix 2-Kinect installation details how to install the device and start program-ming using the Microsoft SDK and the OpenNI. It shows the steps to install the drivers using theMicrosoft SDK, how to set up Visual Studio and Matlab. It also shows how to install the device inother platforms using OpenNI and set up different IDEs such as XCode, Visual Studio, Processingand Matlab.

5.3.2 Code implementations

Taking into account the advantages and disadvantages of both SDKs and after performing someexperiments, we decided to use the official SDK from Microsoft. This SDK gives us better results

-55-


Figure 37: Programming the Kinect with Visual Studio

in terms of accuracy and delay (see section Preliminary Experiments).

Initialize the sensorBefore the Kinect begins producing data for the application it must be initialized. The initializingprocess consist of three steps: enumerate the Kinect sensors, enable data streaming and start theKinect.

Listing 13: Start the sensor

//Instantiate the sensor instance

KinectSensor sensor = KinectSensor.KinectSensors[0];

//Initialize the RGB camera

sensor.ColorStream.Enable(); //Enable camera

sensor.ColorFrameReady += Kinect_ColorFrameReady; //create event when there is

a camera stream (30fps)

//Start the Kinect

sensor.Start();

Process the streamOnce the Kinect is started and a stream is enabled, the application extracts frame data from thestream and processes it as long as a new stream is available. For each frame of data: registers anevent which fires when data is ready, implements the event handler, allocates storage for the data,and gets the data.

-56-


Listing 14: Process the stream

//create new bitmap

this._ColorImageBitmap = new WriteableBitmap(colorStream.FrameWidth,

colorStream.FrameHeight, 96, 96, PixelFormats.Bgr32, null);

//bitmap at the image (visual)

ColorImageElement.Source = this._ColorImageBitmap; //in xamp <Image

x:Name="ColorImageElement" />

private void Kinect_ColorFrameReady(object sender, ColorImageFrameReadyEventArgs e) {

using (ColorImageFrame imageFrame = e.OpenColorImageFrame()) {

if (imageFrame != null) {

byte[] pixelData = new byte[imageFrame.PixelDataLength]; //bytearray to

hold the data

imageFrame.CopyPixelDataTo(pixelData); //copy the data to the array

pixeldata

//updates the bitmap (we are going to use it for the snapshoot)

this._ColorImageBitmap.WritePixels(this._ColorImageBitmapRect,

pixelData, this._ColorImageStride, 0);

} } }

Take a pictureTo take a picture, we take an screenshot of one of the frames acquired with the RGB camera. Thisframe is stored in the directory Images.

Listing 15: Take and save a picture

//Create the new directory

pathFolder = System.IO.Path.Combine(Environment.CurrentDirectory, @"Images");

System.IO.Directory.CreateDirectory(pathFolder); //create the new directory

//Create a png bitmap encoder which knows how to save a .png file

BitmapEncoder encoder = new PngBitmapEncoder();

//Create frame from the writable bitmap and add to encoder

encoder.Frames.Add(BitmapFrame.Create(this._ColorImageBitmap));

//Create the file

pathFile =

System.IO.Path.Combine(Environment.CurrentDirectory,@"Image\ImageFall.png");

// Write the new file to disk

using (FileStream fs = new FileStream(pathFile, FileMode.Create))

encoder.Save(fs);

-57-


6 Results

6.1 System overview

The Android application has a minimalist design which makes it easy to use. It has four differentscreens, the main screen, the settings, a help and about. The help includes a visual tutorial whichexplains how to use it with, or without the Kinect.

Figure 38: Main screen of the KiFall app. This app is available on Google Play.

Figure 39: Other screens of the KiFall app: a) Settings b) Help c) About

-58-


The Kinect block consist on a executable file (.exe) which contains the server and the Kinectapplication. The user starts the server by double click on the file and the Kinect applicationautomatically starts when the fall is detected on the phone.

Figure 40: Server after the fall is detected and the Android connected.

A video demo of the server has been uploaded on youtube 9. The app is also available onGoogle Play. The server and the Kinect application can be downloaded from the web. The readmefile and the Android’s application help shows how to set up the IP address and the port numberto run the server on the computer.

6.2 Performance evaluation

To validate the system we used a dataset created from the German Aerospace Center [10] with thehelp of MIT. The dataset contains human activities such as sitting, standing, waking, jumping,and falling. The data was collected from ten different people and the phone was placed in threedifferent locations: in a shirt pocket (chest), on the belt (waist), and in the pants pocket (pocket).The data was acquired with a Samsung Galaxy S2 and consist on the acceleration, orientation,and rotation readings while ten different people is performing all the activities. We used the NexusS from Google to analyze these sensor readings.

The first approach of the system only uses the proposed algorithm which runs on the Androidphone. Further versions will use the Kinect to verify if the fall has taken place, as we alreadydiscussed. However, the Kinect is used to take a picture and send it to the phone, to broadcast itwith the emergency alert. The Kinect doesn’t affects the accuracy but it does affect the speed ofthe system.

6.2.1 Accuracy

We measured the accuracy in terms of false alarms (FA) and miss detections (MD). False alarmshappens when the systems senses a fall but it did not occur. Miss detections happens when a

9Video demo of the Fall Detection System on youtube http://www.youtube.com/user/skiria8

-59-

http://www.youtube.com/user/skiria8


fall occurs but the system misses it. In general, the lower both FA and MD are, the better theperformance is.

After adjusting all the parameters such as the sample rate, the prediction mode, the algorithm,the thresholds, etc. we obtain false alarm and miss detection rates below 5% for all phone locations.Figure 41 details the rates obtained with the dataset.

waist chest pocket0

1

2

3

4

5

6

7False alarms

% F

A

waist chest pocket0

1

2

3

4

5

6

7Miss detections

% M

DFigure 41: Accuracy of the system.

6.2.2 Speed

Our system integrates an Android-based smartphone and the Microsoft Kinect, which are con-nected over a TCP/IP network. Some of the activities significantly effect the speed of the system:the analysis of the data on the phone, the connection to the Kinect over TCP, and the time whilethe Kinect starts. Preliminary experiments were done to reduce the delay of these activities. Wedetermined, for example, to use the phone to analyze the data and connect to the Kinect if thefall is detected. All parameters and applications were designed to minimize the delay between thefall is detected and reported.

We evaluated the speed of the system in a 5Mb/s ADSL network, which is an standard speedfor home networks. We ran the system several times and we calculated the mean values of thedelays for all processes of the system. Table 7 details this delay. The system runs in less than 7seconds, but if we use without the Kinect it needs less than 4 seconds. Both times are acceptabledelays to report a fall.

Process time (sec)Smartphone (data acquisition and analysis) 1.53Connection to Server 1.28Start Kinect 3.46Send data to phone 0.83Get location and send SMS/MMS 2.23

Table 7: Delay of the main processes

-60-


6.2.3 Battery consumption of the smartphone

To test power consumption of the Android app, we monitored the power states in two differentscenarios: when the fall detection app was running and when it was not. In both experiments thephone is fully charged. The app can run 15 hours before consuming all the battery, which meansthe user has to charge the phone every day. Figure 42 shows the two curves of battery level statesduring 8 hours: after 8 hours of acquiring the data from the sensors and analyzing it, the appconsumes 48% of the battery.

0 1 2 3 4 5 6 7 80

10

20

30

40

50

60

70

80

90

100

time (h)

Battery

level (%

)

not running KiFall

running KiFall

Figure 42: Comparison of the power consumption when KiFall runs on the phone and when the app isnot running.

6.3 Comparison with other fall detection apps

After validating the system with the dataset created by the German Aerospace Center with thehelp of MIT, we compared our app with other apps available on Google Play. The apps that werecompared were chosen based on the number of reviews and user’s rating.

6.3.1 Accuracy

We installed all apps on the phone and we ran every app for 12 hours to track the number of falsealarms while using the phone in the author daily life. To compare the number of miss detections,4 different subjects pretended to fall down while walking. We tracked 4 falls (forward, backward,left-side and right-side). Figure 43 compares the number of false alarms detected in 12 hours andthe number of miss detections for 16 falls.

-61-


SmartFD FallD iFall kiFall0

5

10

15

False alarms

# F

A


1

2

3

4

5

6

7

8

9

10

Miss detections

# M

D

Figure 43: Comparison of false alarms and miss detections for 4 different apps.

Based on these tests, our app achieves better results than other available apps in terms ofaccuracy. After the subjects pretended to fall down while walking, only one fall was not detected.However, 6 false alarms were also detected when the app was running for 12 hours. Our systemintegrates a Kinect so these false alarms are mitigated since the emergency contact will receive apicture of the subject walking instead of lying on the floor.

6.3.2 Battery consumption of the smartphone

Battery life is very important in portable devices. We compared the battery consumption ofdifferent apps when they were running 8 hours. The phone was fully charged for all the experiments.Figure 44 compares the different battery levels for all the fall detection apps.

0 1 2 3 4 5 6 7 80

10

20

30

40

50

60

70

80

90

100

time (h)

Battery

level (%

)

no apps

Smart Fall Detector

Fall Detector

iFall

KiFall

Figure 44: Comparison of the battery consumption for different fall detection apps.

As figure 44 shows, our app consumes 48% of battery after 8 hours running. These resultsare similar than other apps: the consumption of battery is due to the data acquisition with theaccelerometer, gyroscope and other sensors.

-62-


7 Discussions

The Microsoft’s Kinect is a powerful sensor with motion tracking capabilities. In theory it cantrack motion of the user with the depth camera, which might make the device useful to detectfalls. In order to do that, the device must be activated and collecting data from the RGB or depthcamera, and the individual must remain in the Kinect’s field of view. Due to all these limitations,it makes sense the use of an smartphone with build-in motion sensors to detect falls and only startthe Kinect to verify them. The fall is considered detected if both devices determines that the falloccurred. Using both devices, we could get better results in terms of accuracy and reliability.

7.1 Point the Kinect: indoor location

Once the fall is detected on the smartphone, the Kinect can verify it. To do that, the user mustbe in the field of view of the sensor. The angular field of view of the RGB camera can go up to 57◦

horizontal and 43◦ vertically. The viewing distance range form the depth sensor goes from 0.8mto 3.5m. These limitations can be overcomed by placing the sensor in an appropriate location inthe room and using the motorized pivot. The motorized pivot can tilt the sensor 27◦ either up ordown and it is possible to integrate a pivot to turn the Kinect in the horizontal direction.

Pointing the Kinect to the individual it is not difficult; the problem resides in how to deter-mine the position of the individual. Reliable indoor positioning is the main challenge. Differenttechniques could be used taking advantage of the smartphone’s capabilities.

The first approach consist on determine the distance and bearing (angle between the forwarddirection) between two points. The distance and bearing can be calculated through the A-GPS, aset of location sensors that combines GPS location, network positioning and cellular positioning.Figure 45 shows how to determine the angle between two points. This approach is not useful topoint the Kinect in indoors because of the accuracy of location in indoors (10m).

Figure 45: Distance and bearing between two points.

The second approach uses the double integration of the linear acceleration (acceleration of thedevice minus the acceleration due to gravity) to measure position (11). However, double integration

-63-


amplifies acceleration noise so fast. The acceleration readings from the smartphone’s sensors arevery noisy, making this approach useless in our system.

x =

∫v · dt =

∫ ∫a · dt (11)

The third approach consists of the development of algorithms based on the detection of stepsand heading directions while walking. The positioning might be reliable using the accelerometerand gyroscope sensors, and with an accurate estimation of the step length.

7.2 Verify the fall with the Kinect

Although the proposed algorithm running on an Android phone presents high accuracies, the fallverification with the Kinect could improve the results. The Kinect could provide an accuratecontrol of some issues, specially the fall-like events such as jumping or sitting.

The fall verification could consist of detect some joints (head and hands, for example) using theskeleton tracking and calculates the distance from the floor. The fall will be considered detectedif the distance from the floor is almost 0 from all joints. We performed several experiments usingboth Microsoft and OpenNI SDKs but the main challenge is to detect the joints when the Kinectturn on and the person is already lying on the floor. The algorithm gives good results after smallmovements of the individual, but sometimes the person remains unconscious after a fall whichmakes this approach not useful.

We also performed some experiments using OpenNI and open source libraries. The fall verifica-tion consist of detect the individual using the depth data. The libraries allows to segment the userfrom the background. Once the individual is selected, we checked if the individual’s bounding boxis less tall than a threshold value and the position of the highest point is lower than a threshold,which means than the user is lying on the floor. This approach has the same limitation: pickupthe person if is already lying on the floor when the Kinect turns on.

Figure 46: Fall verification with a) skeleton tracking (Microsoft SDK) b) skeleton tracking + depth(OpenNI) c) user selection using depth data (OpenNI).

-64-


using

8 Conclusions

8.1 Conclusions

In this thesis, we propose a fall detection system using an Android-based smartphone and theKinect sensor. The technique is based on the principle of detecting changes in motion and bodyposition of an individual. The user is required to wear a smartphone which tracks acceleration andorientation changes. The data is continuously analyzed on the smartphone to determine wetherthe individual’s body is falling or not. If an individual falls, the smartphone triggers the Kinectto take a picture, and employs GPS to determine the location. An emergency alert is issued inorder to get assistance. The alert is sent through the phone and consist of a multimedia message(MMS) including the location (coordinates and a link to google maps), the picture, and the timewhen the fall occurred.

We designed the detection algorithm and implemented it on a Nexus S running Android 4.1.2.The smartphone can be placed on different locations -pocket, waist and chest- presenting highaccuracies for all of them. The algorithm was validated with the dataset created from the GermanAerospace Center [10], obtaining false alarm and miss detection rates below 5% for all phonelocations. Experimental results show that the Android app achieves good detection performanceand battery consumption. The developed Android app was compared with other fall detection appsavailable on Google Play, obtaining better results in terms of accuracy and battery consumption.

This is the first system which integrates a smartphone and the Kinect sensor for reliable falldetection. The system, using both devices, needs 7 seconds to report a fall. Without the useof the Kinect, it takes less than 4 seconds since the individual fall and the emergency alert issent. However, the integration of a smartphone and the Kinect sensor presents many excitingopportunities for the future of fall detections and emergency reportedly.

-65-


8.2 Future work

The proposed system integrates an Android-based smartphone and the Microsoft Kinect for reliablefall detection. However, as we discussed in the section Discussions the Kinect could also be usedfor most reliable fall detection. The future work includes the points we already discussed it andnew ideas for the advancement of the project and the Kinect development.

• Report of daily activities and detected falls: the accelerometer and gyroscope’s readingscould be combined with other sensor readings to identify basic daily activities such as sitting,standing, jumping, running, walking... It might be useful for doctors and caregivers to havea report of all daily activities in order to detect future problems and prevent illness, injuriesor falls.

• Point the Kinect: the field of view angles for the Kinect are 43 vertically (±27 with the mo-torized pivot) and 57 horizontally. These are enough for a small room and a good placementof the sensor but not for big rooms. Because of that, it will be useful to point the Kinect tothe individual using the phone’s location. There are several approaches that could be usedbut the most reliable for indoor positioning might be an algorithm based on the detection ofthe steps and heading directions while walking. The Android’s accelerometer and gyroscopemight be useful for accurate estimation.

• Verify the fall with the Kinect: the Kinect could be integrated with the phone for morereliable results, specially to handle fall-like events such as jumping or sitting. The fallverification could consist of detect some joints (head and hands, for example) using theskeleton tracking and calculates the distance from the floor. Other approaches could bealso tested, using both Microsoft’s SDK and OpenNI and comparing the results in terms ofaccuracy and response time.

• Raspberry Pi (RaspPi) with the Kinect: currently the Kinect must be connected to a com-puter or the Xbox 360 to run applications. The RaspPI is a credit-card-sized single boardcomputer with limited processing and storage capabilities. The most powerful Raspberry Piin terms of CPU might be used to run Kinect applications. However, performed experimentswere not successful. The integration of the Kinect with a RaspPI and a mobile platformwould be a fantastic platform for our system. It only needs the adaptation of the Kinect’sdrivers, which have open source versions, to enable and acquire data from the RGB cameraor depth sensor.

-66-


References

[1] Biosensic’s fall detector. http://www.biosensics.com/pamsys-overview/, 2013.

[2] Guardian medical monitoring. http://guardianmedicalmonitoring.com/solutions/

personal-emergency-response-system, 2013.

[3] Healthsence system for daily activity and fall detection. http://healthsense.com/index.

php/products/remote-monitoring/eneighbor, 2013.

[4] Shimmer wearable sensor system. http://www.shimmersensing.com/

research-and-education/applications/#applications-tab, 2013.

[5] Tynetec telecare sensors. http://www.tynetec.co.uk/products/products/

telecare-sensors, 2013.

[6] Visonic’s device. http://www.visonic.com/Products/Wireless-Emergency-Response-Systems/Fall-detector-mct-241md-pers-wer, 2013.

[7] A. Ariani, S.J. Redmond, D. Chang, and N.H. Lovell. Software simulation of unobtrusive fallsdetection at night-time using passive infrared and pressure mat sensors. In Engineering inMedicine and Biology Society (EMBC), 2010 Annual International Conference of the IEEE,pages 2115–2118, 2010.

[8] Woon-Sung Baek, Dong-Min Kim, F. Bashir, and Jae-Young Pyun. Real life applicable falldetection system based on wireless body area network. In Consumer Communications andNetworking Conference (CCNC), 2013 IEEE, pages 62–67, 2013.

[9] Yabo Cao, Yujiu Yang, and Wenhuang Liu. E-falld: A fall detection system using android-based smartphone. In Fuzzy Systems and Knowledge Discovery (FSKD), 2012 9th Interna-tional Conference on, pages 1509–1513, 2012.

[10] German Aerospace Center. Human activity recognition dataset with inertial sensors. http:

//www.dlr.de/kn/en/desktopdefault.aspx/tabid-8500/, 2012.

[11] Tso-Cho Chen. Fall detection and location using zigbee sensor network. In Cross Strait Quad-Regional Radio Science and Wireless Technology Conference (CSQRWC), 2011, volume 2,pages 937–941, 2011.

[12] Jiangpeng Dai, Xiaole Bai, Zhimin Yang, Zhaohui Shen, and Dong Xuan. Perfalld: A pervasivefall detection system using mobile phones. In Pervasive Computing and CommunicationsWorkshops (PERCOM Workshops), 2010 8th IEEE International Conference on, pages 292–297, 2010.

[13] L. Della Toffola, S. Patel, Bor rong Chen, Y.M. Ozsecen, A. Puiatti, and P. Bonato. Develop-ment of a platform to combine sensor networks and home robots to improve fall detection inthe home environment. In Engineering in Medicine and Biology Society,EMBC, 2011 AnnualInternational Conference of the IEEE, pages 5331–5334, 2011.

-67-

http://www.biosensics.com/pamsys-overview/

http://guardianmedicalmonitoring.com/solutions/personal-emergency-response-system

http://guardianmedicalmonitoring.com/solutions/personal-emergency-response-system

http://healthsense.com/index.php/products/remote-monitoring/eneighbor

http://healthsense.com/index.php/products/remote-monitoring/eneighbor

http://www.shimmersensing.com/research-and-education/applications/#applications-tab

http://www.shimmersensing.com/research-and-education/applications/#applications-tab

http://www.tynetec.co.uk/products/products/telecare-sensors

http://www.tynetec.co.uk/products/products/telecare-sensors

http://www.visonic.com/Products/Wireless-Emergency-Response-Systems/Fall-detector-mct-241md-pers-wer

http://www.visonic.com/Products/Wireless-Emergency-Response-Systems/Fall-detector-mct-241md-pers-wer

http://www.dlr.de/kn/en/desktopdefault.aspx/tabid-8500/

http://www.dlr.de/kn/en/desktopdefault.aspx/tabid-8500/


[14] B.E. Demiroz, A.A. Salah, and L. Akarun. Fall detection using multi-omnidirectional cameras.In Signal Processing and Communications Applications Conference (SIU), 2013 21st, pages1–4, 2013.

[15] C.N. Doukas and I. Maglogiannis. Emergency fall incidents detection in assisted living envi-ronments utilizing motion, sound, and visual perceptual components. 15(2):277–289, 2011.

[16] M. Grassi, A. Lombardi, G. Rescio, M. Ferri, P. Malcovati, A. Leone, G. Diraco, P. Siciliano,M. Malfatti, and L. Gonzo. An integrated system for people fall-detection with data fusioncapabilities based on 3d tof camera and wireless accelerometer. In Sensors, 2010 IEEE, pages1016–1019, 2010.

[17] Yibin Hou, Na Li, and Zhangqin Huang. Triaxial accelerometer-based real time fall eventdetection. In Information Society (i-Society), 2012 International Conference on, pages 386–390, 2012.

[18] Christopher Kawatsu, Jiaxing Li, and C.J. Chung. Development of a fall detection system withmicrosoft kinect. In Jong-Hwan Kim, Eric T. Matson, Hyun Myung, and Peter Xu, editors,Robot Intelligence Technology and Applications 2012, volume 208 of Advances in IntelligentSystems and Computing, pages 623–630. Springer Berlin Heidelberg, 2013.

[19] Young-Sook Lee and HoonJae Lee. Multiple object tracking for fall detection in real-timesurveillance system. In Advanced Communication Technology, 2009. ICACT 2009. 11th In-ternational Conference on, volume 03, pages 2308–2312, 2009.

[20] D. Litvak, Y. Zigel, and Israel Gannot. Fall detection of elderly through floor vibrationsand sound. In Engineering in Medicine and Biology Society, 2008. EMBS 2008. 30th AnnualInternational Conference of the IEEE, pages 4632–4635, 2008.

[21] Georgios Mastorakis and Dimitrios Makris. Fall detection system using kinect’s infraredsensor. Journal of Real-Time Image Processing, pages 1–12, 2012.

[22] Anh Tuan Nghiem, E. Auvinet, and J. Meunier. Head detection using kinect camera and itsapplication to fall detection. In Information Science, Signal Processing and their Applications(ISSPA), 2012 11th International Conference on, pages 164–169, 2012.

[23] K. Niazmand, C. Jehle, L.T. D’Angekoray13lo, and T.C. Lueth. A new washable low-costgarment for everyday fall detection. In Engineering in Medicine and Biology Society (EMBC),2010 Annual International Conference of the IEEE, pages 6377–6380, 2010.

[24] O. Ojetola, E.I. Gaura, and J. Brusey. Fall detection with wearable sensors–safe (smart falldetection). In Intelligent Environments (IE), 2011 7th International Conference on, pages318–321, 2011.

[25] D. Ozcan and O.O. Ergun. Developing an occlusion-resistant automatic fall detection systemfor smart environments. In Signal Processing and Communications Applications Conference(SIU), 2012 20th, pages 1–4, 2012.

-68-


[26] K. Ozcan, A.K. Mahabalagiri, M. Casares, and S. Velipasalar. Automatic fall detection andactivity classification by a wearable embedded smart camera. Emerging and Selected Topicsin Circuits and Systems, IEEE Journal on, 3(2):125–136, 2013.

[27] M. Popescu, Yun Li, M. Skubic, and M. Rantz. An acoustic fall detector system that usessound height information to reduce the false alarm rate. In Engineering in Medicine andBiology Society, 2008. EMBS 2008. 30th Annual International Conference of the IEEE, pages4628–4631, 2008.

[28] H. Rimminen, J. Lindstrom, M. Linnavuo, and R. Sepponen. Detection of falls among theelderly by a floor sensor using the electric near field. Information Technology in Biomedicine,IEEE Transactions on, 14(6):1475–1476, 2010.

[29] F. Sposaro and G. Tyson. ifall: An android application for fall monitoring and response.In Engineering in Medicine and Biology Society, 2009. EMBC 2009. Annual InternationalConference of the IEEE, pages 6119–6122, 2009.

[30] N. Thome, S. Miguet, and S. Ambellouis. A real-time, multiview fall detection system: Alhmm-based approach. Circuits and Systems for Video Technology, IEEE Transactions on,18(11):1522–1532, 2008.

[31] Marie Tolkiehn, L. Atallah, B. Lo, and Guang-Zhong Yang. Direction sensitive fall detectionusing a triaxial accelerometer and a barometric pressure sensor. In Engineering in Medicineand Biology Society,EMBC, 2011 Annual International Conference of the IEEE, pages 369–372, 2011.

[32] S. Tomii and T. Ohtsuki. Falling detection using multiple doppler sensors. In e-Health Net-working, Applications and Services (Healthcom), 2012 IEEE 14th International Conferenceon, pages 196–201, 2012.

[33] Vo Quang Viet, Gueesang Lee, and Deokjai Choi. Fall detection based on movement and smartphone technology. In Computing and Communication Technologies, Research, Innovation, andVision for the Future (RIVF), 2012 IEEE RIVF International Conference on, pages 1–4, 2012.

-69-


List of Tables

1 Comparison of today’s smartphones. Source: Gartner (April 2013) . . . . . . . . . . 142 Comparison of mobile operating systems . . . . . . . . . . . . . . . . . . . . . . . . 143 Comparison of the Kinect SDKs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214 Comparison of Android apps for fall detection. . . . . . . . . . . . . . . . . . . . . . 255 Thresholds for different phone locations. . . . . . . . . . . . . . . . . . . . . . . . . 356 Thresholds for different phone locations. . . . . . . . . . . . . . . . . . . . . . . . . 427 Delay of the main processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

List of Figures

1 Personal Emergency System (PER) . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 Telecare sensors for daily activity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 Fall Detection System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 Samsung Galaxy Nexus S from Google . . . . . . . . . . . . . . . . . . . . . . . . . 135 Location by triangulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156 Scanadu, app and sensor for autodiagnosys . . . . . . . . . . . . . . . . . . . . . . . 177 Asthmapolis, app and sensor to monitor asthma patients . . . . . . . . . . . . . . . 188 Microsoft Kinect Sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189 Microsoft Kinect Sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1910 Features and Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2011 Rehabilitation with Kinect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2212 InAVate for group therapy sessions . . . . . . . . . . . . . . . . . . . . . . . . . . . 2313 Hands free tool to move and zoom X-Ray and MRI . . . . . . . . . . . . . . . . . . 2314 Visonic’s Personal Emergency Response solution . . . . . . . . . . . . . . . . . . . . 2415 Smart Fall Detection app for Android . . . . . . . . . . . . . . . . . . . . . . . . . . 2516 SAFE (SmArt Fall dEtection) System with Shimmer sensor nodes . . . . . . . . . . 2617 Smartphone based approach with data from accelerometer and gyroscope . . . . . . 2718 Fall detection solution based on infrared and pressure sensors . . . . . . . . . . . . . 2819 Multiple people tracking to and fall detection . . . . . . . . . . . . . . . . . . . . . 2920 Fall detection system using the Kinect sensor . . . . . . . . . . . . . . . . . . . . . . 3021 Sensor network and home robots for fall detection . . . . . . . . . . . . . . . . . . . 3022 Architecture of the system and the device. . . . . . . . . . . . . . . . . . . . . . . . 3123 Flowchart of the proposed fall detection system. . . . . . . . . . . . . . . . . . . . . 3224 Typical patterns of total acceleration, rotation and orientation. . . . . . . . . . . . . 3425 Smartphone-Kinect communication through a TCP server. . . . . . . . . . . . . . . 3626 Sensor readings while an individual is walking and suddenly falls down. . . . . . . . 3727 Effects on At for different prediction modes. . . . . . . . . . . . . . . . . . . . . . . 3828 Evaluation of the prediction mode at different sample rates. . . . . . . . . . . . . . 3929 Error when applying different filters on At at different sample rates. . . . . . . . . . 4030 Typical patterns of total acceleration, rotation and orientation. . . . . . . . . . . . . 40

-70-


31 Comparation of the proposed algorithms for different phone locations. . . . . . . . . 4132 ROC curve for different thresholds Thh when the Thl is fixed at 0.6G (G = 9.81m/s2).

Phone is placed at chest. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4233 Comparation of the delay when triggering the Kinect. . . . . . . . . . . . . . . . . . 4334 Comparison of the system delay to determine where to compute the data. . . . . . . 4435 Developing Android apps in Eclipse. . . . . . . . . . . . . . . . . . . . . . . . . . . 4536 Hierarchy of classes of the Android application. . . . . . . . . . . . . . . . . . . . . 4737 Programming the Kinect with Visual Studio . . . . . . . . . . . . . . . . . . . . . . 5638 Main screen of the KiFall app. This app is available on Google Play. . . . . . . . . . 5839 Other screens of the KiFall app: a) Settings b) Help c) About . . . . . . . . . . . . 5840 Server after the fall is detected and the Android connected. . . . . . . . . . . . . . . 5941 Accuracy of the system. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6042 Comparison of the power consumption when KiFall runs on the phone and when

the app is not running. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6143 Comparison of false alarms and miss detections for 4 different apps. . . . . . . . . . 6244 Comparison of the battery consumption for different fall detection apps. . . . . . . . 6245 Distance and bearing between two points. . . . . . . . . . . . . . . . . . . . . . . . 6346 Fall verification with a) skeleton tracking (Microsoft SDK) b) skeleton tracking +

depth (OpenNI) c) user selection using depth data (OpenNI). . . . . . . . . . . . . 6447 Android SDK Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7348 Install the Eclipse plugging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7449 Create an Android App on Eclipse . . . . . . . . . . . . . . . . . . . . . . . . . . . 7550 Eclipse workspace after creating a new project . . . . . . . . . . . . . . . . . . . . . 7651 App running on the Android Virtual Device . . . . . . . . . . . . . . . . . . . . . . 7752 App running on the Android Device . . . . . . . . . . . . . . . . . . . . . . . . . . . 7753 Kinect Explorer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7854 Programming the Kinect with Visual Studio on Windows . . . . . . . . . . . . . . . 8055 Acquire data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8156 SimpleViewer (OpenNI sample) to test if the installation was correct . . . . . . . . 8257 Programming the Kinect with Visual Studio on Windows . . . . . . . . . . . . . . . 8358 OpenNI pre-compiled sample to test the installation of the Kinect on MacOS . . . . 8459 Programming the Kinect with XCode+OpenNI on Mac . . . . . . . . . . . . . . . . 8560 Programming the Kinect with Processing+SimpleOpenNI . . . . . . . . . . . . . . . 8661 Programming the Kinect with Matlab+OpenNI . . . . . . . . . . . . . . . . . . . . 87

Listings

1 Pseudo-code of the proposed detection algorithm . . . . . . . . . . . . . . . . . . . 342 Register sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 483 Read sensor values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494 Pseudo-code of the proposed detection algorithm . . . . . . . . . . . . . . . . . . . 505 Connect to the server and exchange messages . . . . . . . . . . . . . . . . . . . . . 50

-71-


6 Transfer a picture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 517 Send a SMS or MMS message . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 528 Get the location . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 529 Start the listener . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5310 Listen for clients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5311 Send/receive a message (code to handle the errors has to be implemented) . . . . . 5412 Send a picture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5413 Start the sensor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5614 Process the stream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5615 Take and save a picture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

-72-


A Getting started with Android SDK

The Android SDK provides the API libraries and developer tools necessary to build, test, anddebug apps for Android. All the resources are available on the official website 10.

Currently it is possible to develop apps on different IDEs although Eclipse is the most used. Toset up the IDE it is necessary to install a plugging and the SDK tools, which include all platformsand packages needed. The SDK is multiplatform and the steps to install and run an applicationare the same for Windows, MacOS and Linux.

A.1 Install the Android SDK

The SDK tools include the Android SDK Manager which allows to download and install all neededtools and packages. Available packages on the SDK Manager are: tools for debugging and testingapps, documentation for the Android platform APIs, the SDK platform, system images fore eachversion of Android, a collection of samples, the Google APIs, and support libraries.

1. Download and install the SDK tools.

2. Launch the SDK Manager and install the desired packages. Recommended packages are thelatest tools packages, the latest version of Android and the Android Support Library.

Figure 47: Android SDK Manager

A.2 Setting up Eclipse

The Eclipse IDE for Java Developers contains all the resources to build Java applications. Howeverit requires to install the ADT Plugin to build Android apps.

10Android Developer website http://developer.android.com/

-73-

http://developer.android.com/


1. Open Eclipse and install the ADT Plugin on Help > Install New Software.

2. Add the repository from the following URL location: http://dl-ssl.google.com/android/eclipse/).

3. Once the installation completes, restore Eclipse.

Figure 48: Install the Eclipse plugging

A.3 Programming with Eclipse

Eclipse is a multiplatform IDE so the steps to program and run a new Android app are the samefor all platforms (Windows, MacOs and Linux).

1. Create a New Android Project by clicking New > Project > Android Project.

2. Set the Application Name (name that appears to users), the Project Name (name of theproject visible in Eclipse), Package Name (package namespace for the app), the SDK versionagains which the app will compile, and the Minimum SDK required.

3. Create a launcher icon and select a template from which begging building the app.

-74-


Figure 49: Create an Android App on Eclipse

The Android project created includes a default set of source files:

• src/: Directory for the app’s main source files coded in Java.

• drawable/: Drawable objects such as bitmaps.

• layout/: Files that define the user interface.

• values/: XML files that contains a collection of resources (strings, color definitions, padding...).

• Androidmanifest.xml: Defines all components of the app and describes the fundamentalcharacteristics.

-75-


Figure 50: Eclipse workspace after creating a new project

A.4 Installing the app

Once the app is developed, it could be installed on an Android device or on the emulator.To run the app on the emulator:

1. Launch the Virtual Device Manager from Window > AVD Manager in Eclipse.

2. Create a new Android Virtual Device and get it a name, platform target and SD card size.

3. Click the Run button from the toolbar in Eclipse. In case there are more emulators runningor an Android device plugged, Eclipse will ask where to run the app.

-76-


Figure 51: App running on the Android Virtual Device

To run the app on a real device:

1. Plug in the Android device to the computer with the USB cable.

2. Install the USB drivers if it is needed.

3. Enable USB debugging on the device by clicking Settings > Developer Options.

4. Click the Run button from the toolbar on Eclipse. In case there are other Android devicesplugged, or emulators running, Eclipse will ask where to run the app.

Figure 52: App running on the Android Device

-77-


B Getting started with the Kinect sensor

B.1 Getting started with the Kinect for Windows SDK

B.1.1 Installing the Kinect sensor and the developer tools

The Kinect for Windows Software Development Kit (SDK) enables developers to use C++, C#,or Visual Basic to create applications by using the Kinect for Windows sensor connected on acomputer. All drivers, software and resources to install the Kinect and start developing apps areon the Microsoft official webpage11.

1. Download and install the Kinect for Windows SDK which includes the drivers for using theKinect sensor on computers and other devices running Windows.

2. Download and install the Windows Developer Toolkit which includes all the resources andsamples to simplify developing applications.

3. Download and install the .NET Framework. This framework is a developer platform to buildapps for Windows.

4. Download and install the Microsoft Visual Studio Express 2012. This is the official IDEused for developing on Windows. The new Matlab’s release enables the use the Kinect fromMatlab, in this case you should install Matlab instead of Visual Studio. The settings to startdeveloping in both platforms are detailed in the following sections.

5. Test if the installation was correct running the Kinect Explorer. This software is included onthe Developer Toolkit which also includes applications, documentation, tools and samples toprovide a starting point for working with the SDK.

Figure 53: Kinect Explorer

11Kinect for Windows developer website: www.microsoft.com/visualstudio/eng/products/

visual-studio-express-products#2010-Visual-CPP

-78-

www.microsoft.com/visualstudio/eng/products/visual-studio-express-products#2010-Visual-CPP

www.microsoft.com/visualstudio/eng/products/visual-studio-express-products#2010-Visual-CPP


B.1.2 Programming the Kinect with Visual Studio

Visual Studio is the official IDE to develop for Windows Desktop, Windows Phone and to developKinect applications.

1. Create a new Windows Based application project.

2. Add references to the Kinect library by right-click on the References menu located in theExplorer Panel. Add the Microsoft Kinect, which is the official driver for the Kinect.

3. Program the application.

(a) Initialize and start the Kinect, which consist on three steps: initialize the sensor in-stance, enable the data streaming (camera, depth, skeleton tracking or speech recogni-tion streams) and start the kinect.

Initialize Kinect

(b) Once the Kinect is started and a stream is enabled, the appication extracts frame datafrom the stream and processes it as long as a new stream is available. For each frameof data: register an event which fires when data is ready, implement an event handler,allocate storage for the data and get the data.

Implement the event for the RGB camera

-79-


4. Compile the project, build the application and run it on a new window.

Figure 54: Programming the Kinect with Visual Studio on Windows

B.1.3 Programming the Kinect with Matlab

The new Matlab release (R2013a) enables the use the Kinect sensor on Matlab without installingadditional libraries. Previous versions requires to install and set up a specific library. With thenew release Matlab can obtain the data from the Kinet using the Image Acquisition Toolbox whichcan be downloaded from the Mathworks website12.

1. Add the path that includes utility functions which accepts the skeleton data, color imageand depth maps.

Add functions

2. Initialize and start the Kinect by enabling the sensors (color sensor and depth sensor), ac-quiring color and depth data. Finally start the acquisition or the preview.

12Image Acquisition Toolbox: www.mathworks.com/products/imaq/

-80-

www.mathworks.com/products/imaq/


Initialize and start the Kinect

3. Once the Kinect and the streams are enabled it is really easy to retrieve the acquired datain order to use in your program, script or Matlab application.

Figure 55: Acquire data

B.2 Getting started with the OpenNI SDK

B.2.1 Installing the Kinect sensor and the developer tools on Windows

The OpenNI framework is an open source SDK used for the development of Kinect applications andlibraries. There is a big community of developers using this framework and the OpenNI website13

provides all the tools, resources and support to develop applications using this framework. Alltools are available for using on Windows, Linux and MacOS.

13OpenNI website: www.openni.org

-81-

www.openni.org


1. Download and install the OpenNI SDK which installs the SDK and the resources to developOpenNI applications.

2. Download and install the NITE library used for body tracking such as hand tracking, sceneanalyzer, skeleton joint tracking, and gesture recognition.

3. Download and install SensorKinect 14 which is the sensor driver for OpenNI 1.5.

4. There are several options to develop applications for Windows, MacOS or Linux. On MacOSand Linux, Kinect can be used from Matlab, Processing or the GCC/GNU compiler. OnWindows, Kinect can also be used from Matlab, Processing and Visual Studio. Downloadand install the IDE version accordingly with your needs and operating system.

5. Plug the Kinect into the computer and check out some of the example demos provided. Allthe pre-compiled samples are located in the OpenNI/Samples/Bin and NITE/Samples/Binfolders. Test if the installation was correct by running the SimpleViewer which displays thedepth image captured with the Kinect.

Figure 56: SimpleViewer (OpenNI sample) to test if the installation was correct

B.2.2 Programming the Kinect with Visual Studio on Windows

Visual Studio is the official IDE to develop Kinect apps on Windows. However, it can also be usedto develop apps using OpenNI with the following configuration:

1. Create a new C++ Win32 Console Application Project.

2. Go to Project > MyProject Properties. Do the following steps for both Debug and Re-lease configurations, which can be selected on the upper-left side of the properties window.

In C/C++ > General:

(a) Add ($OPENNI_INCLUDE) or ($OPENNI_INCLUDE64) for 64bits version to AdditionalInclude Directories.

14SensorKinect: https://github.com/avin2/SensorKinect

-82-

https://github.com/avin2/SensorKinect


(b) Add ($OPENNI_LIB) or ($OPENNI_LIB64) for 64bits version to Additional LibraryDirectories.

In C/C++> Linker:

(a) Add openNI.lib in Additional Dependencies.

3. Copy all the files from OpenNI redist directory (C:\Program Files (x86)\OpenNI\Redist)

to the working directory. The working directory is the directory where the executable canbe found if you run Visual Studio from command line. If you run Visual Studio by clickingthe icon, the working directory is where the project files can be found.

Figure 57: Programming the Kinect with Visual Studio on Windows

B.2.3 Installing the Kinect sensor and the developer tools on MacOS X

One of the advantages of OpenNI is that can be installed in Windows, MacOS and Linux. OnMacOS.

1. Download and Install XCode from the Apple Developer website 15. Once installed installCommand Line Tool from the Downloads tab on the XCode Preferences.

2. Download and install Xquartz 16, MacPorts 17 and CMake18

15XCode website: http://developer.apple.com/xcode/16XQuartz website: http://xquartz.macosforge.org/landing/17MacPorts website: www.macports.org/install.php18CMake website: www.cmake.org/cmake

-83-

http://developer.apple.com/xcode/

http://xquartz.macosforge.org/landing/

www.macports.org/install.php

www.cmake.org/cmake


3. Open the terminal which is located on Applications > Utilities > Terminal.

4. Install Libtool running the following command: sudo port install libtool

5. Install Libusb running the following command: sudo port install libusb +universal

6. Download OpenNI from the official website 19 and install it. To install from terminal, navigateinside the directory and run the command: sudo ./install.sh

7. Download SensorKinect from the github place 20 . To install from terminal, navigate insidethe directory and run the command: sudo ./install.sh

8. Download NITE from the OpenNI website and install it. To install from terminal, navigateinside the directory and run the command: sudo ./install.sh

9. Plug the Kinect into the Mac and check out some of the example demos provided. Allthe pre-compiled samples are located in the OpenNI/Samples/Bin and NITE/Samples/Binfolders. Test if the installation was correct by running the SimpleViewer which displays thedepth image captured with the Kinect.

Figure 58: OpenNI pre-compiled sample to test the installation of the Kinect on MacOS

B.2.4 Programming the Kinect with XCode on Mac

XCode is the IDE most used on MacOS. It can be used to program on several languages andplatforms such as iOS development. The steps to set up the environment to compile OpenNI andNITE are:

1. Go to Project > Edit Project Settings to point the project to necessary files and li-braries:

19OpenNI website: www.openni.org20SensorKinect: https://github.com/avin2/SensorKinect

-84-

www.openni.org

https://github.com/avin2/SensorKinect


(a) In Search Paths > Header Search Paths, recursively add /usr/include/ni and /us-r/include/NITE.

(b) In Linking > Other Linker Flags, add /usr/lib/libnimCodecs.dylib, /usr/lib/libnim-MockNodes.dylib, /usr/lib/libnimRecorder.dylib, /usr/lib/libOpenNI.dylib, /usr/lib/libXn-VFeatures.dylib, /usr/lib/libXnVHandGenerator.dylib .

(c) In Architectures > Architecture, set project to 32 or 64 bit.

2. Right click on Project and select Add > Existing Frameworks... to point the project toadditional needed frameworks:

(a) In OpenGL.framework, add /System/Library/Frameworks/GLUT.framework.

Figure 59: Programming the Kinect with XCode+OpenNI on Mac

B.2.5 Programming the Kinect with Processing+SimpleOpenNI

Processing is a programming language and development environment. It can be used to developKinect apps by using the libraries SimpleNI and OpenKinect, among others.

1. Download the libraries needed for OpenNI from the official website. SimpleOpenNI is themost used library 21.

21Kinect libraries to use on Processing: http://code.google.com/p/simple-openni/

-85-

http://code.google.com/p/simple-openni/


2. Copy the library into the Processing working directory. For the SimpleOpenNI library:WorkingDirectory/libraries/library

3. Open Processing, start a new project and add the library.

Figure 60: Programming the Kinect with Processing+SimpleOpenNI

B.2.6 Programming the Kinect with Matlab

OpenNI is not officialy supported on Matlab but it is possible to download a set of c++ wrapperfunctions for the Kinect with OpenNI. These functions are compatible with Matlab 32 and 64bitson Windows, MacOS and Linux but requires Visual Studio or Gcc compiler.

1. Download 22 the zip-file which contains the c++ wrapper functions for the Kinect.

2. Download Visual Studio or a Gcc C++ compiler.

3. Start Matlab and go to the directory where the wrapper functions are located.

4. Execute compile_cpp_files to compile the mex-files. Once done the mex-files are readyto use.

5. Run one of the examples. ExampleIR show a high-res IR image, ExampleSK show skeletontracking on recorded Kinect movie and ExampleCP capture the Kinect streams to a file.Explore all the examples to understand how to use OpenNI and Kinect on Matlab.

22Kinect OpenNI on Matlab: /urlwww.mathworks.com/matlabcentral/fileexchange/30242-kinect-matlab

-86-


Figure 61: Programming the Kinect with Matlab+OpenNI

-87-


1

KiFall: Detecting Falls Using an Android-BasedSmartphone and the Kinect Sensor

Bruna Girvent, Kaushik R. ChowdhuryNortheastern University, Boston (MA), USA

Email: [email protected]

Abstract—One-third of the elderly population experiences atleast one fall each year. For the this population or patientswith special mobility needs a fall can have dramatic healthconsequences. Fast and precise detection of falls is critical, andprompt notification to emergency service is essential for a shortrecovery time. Therefore, many fall detection systems have beendeveloped. The existing commercial solutions consists of wearabledevices placed on different locations on the body. However, thesedevices can be rather expensive, uncomfortable to wear, and theindividual have to remember to wear them.

In this thesis we propose a fast, private system using offthe shelf components typically found at homes, such as anAndroid-based smartphone and the Kinect sensor. Our systemis based on detecting changes in motion and body position ofan individual by tracking the acceleration and orientation. Thedata is continuously acquired and analyzed in the smartphone todetermine when a fall occurs. The system integrates the Kinectfor more reliable fall detection. If a fall is detected, the systemsends an emergency alert which consist of a picture, the time,and the location of the individual.

We performed a set of experiments to test the fall detectionalgorithm and set up all the parameters of the system. Weimplemented a prototype system on the Nexus S connected tothe Kinect through a TCP server. Experimental results showthat our system achieves strong detection performance, fast timeresponse, and little impact on the smartphone’s battery life.

I. INTRODUCTION

The elderly population or patients with special mobilityneeds experience falls than can have dramatic consequences.About one third of older adults have one or more falls peryear, and the risk of falls increase proportionately with age.Studies show that falls are also common with Parkinson’sdisease, patients with cognitive problems, dementia, and strokepatients.

It is established that the sooner the fall is reported, the lowerthe risk of severe injuries or death. This reinforces the impor-tance of medical alert system devices. Current monitoring de-vices typically requires user input such as pushing a button toalert that a fall or emergency has occurred. The problem withthis type of monitoring device is that many times, when a fallhas occurred, the individual is rendered unconscious and can’tactivate the control. Other solutions includes surveillance usingcameras or motion detectors, although the use videocamerascarries with it privacy concerns.

Nowadays, there are very few fall detection products. To-day’s fall detectors are small devices which clip onto thewearer’s belt or waistband and it contains sensors and mi-croprocessors to identify falls. However, the detection of the

fall is also an interesting problem which one can approachusing various methods.

II. MOTIVATION

The considerable risks of falls on the elderly and patients,and the lack of monitoring systems motivate the developmentof fall detection systems. Fall detectors must be automatic,reliable and easy to use. Getting help fast after a fall improvesthe change of survival and the health outcome. Therefore, wepropose a fast, private system using shelf components typicallyfound at homes, such as smartphones and gaming controls.

Today’s smartphones are sophisticated computing platformswith complex sensor capabilities, such as detecting user loca-tion or sensing motion, among others. The built-in motion sen-sors make possible to develop reliable fall detection systemsbased on the body acceleration and orientation. Moreover,location functions and phone’s connectivity enable to call forhelp and get rapid assistance when a fall is detected. Therefore,smartphones are portable and affordable devices whose self-contained functionalities make them an efficient cost-effectivesolution for detecting falls.

Microsoft’s Kinect is a set of sensors developed as aperipheral device to use as a touch-free gaming controller.Nevertheless, Kinect’s potential expands far beyond just gam-ing. The sensors offers great possibilities in the medical field:the motion capture functionalities enables the Kinect to trackpeople’s movements, a different approach for a fall detectionsystem.

Fig. 1: Architecture of the system and the device.

Our contribution combines and Android-based smartphoneand the Microsoft’s Kinect sensor. Figure 1 shows the ar-

-88-


2

chitecture of the system, integrating both devices. It alsoshows the device architecture, which integrates the sensorsand the connectivity modules. The phone acquires the datafrom motion and position sensors such as accelerometers andgyroscopes. This data is processed and analyzed on the phone,where the proposed algorithm determines if a fall has occurred.Once the fall is detected, the phone triggers the Kinect througha TCP server running on the computer. The Kinect could beused to verify the fall by motion analysis based on skeletontracking. However, our first approach uses the Kinect to takea picture of the person lying down. This picture is sent to thephone through the server and it will be broadcasted with theemergency alert. The alert is sent from the phone and consistof a multimedia message (MMS) with the picture of the personlying down, the time, and location (longitude, latitude and linkto google maps) where the fall occurred. Figure 2 shows theflowchart of the system.

Fig. 2: Flowchart of the proposed fall detection system.

III. RELATED WORK

A variety of research is focused on detecting falls. Existingacademic research bases fall detection system in differentapproaches which can be categorized into four categories:wearable devices, ambient analyzers, motion detectors andhybrid systems.

A. Wearable devices

Wearable sensors measure body’s acceleration and orienta-tion as an indicator for falls. Existing solutions mainly useaccelerometers for detection, since falls are usually character-ized by large accelerations. Niazmand et al. in [17] designeda washable pullover with eight integrated accelerometers todetect the motion of the torso and the upper body. Houet al. in [11] proposed a fall detection system based onfeatures abstracted from a tri-axial accelerometer. Focusingonly on large acceleration can result in may false positivesfrom fall-like activities. To solve that, Ojetola et al. in [18]and Baek et al in. [2] proposed a system combining a tri-axialaccelerometer and a tri-axial gyroscope. The Tolkiehn et alin [25] system uses a barometric pressure sensor instead ofa gyroscope: the tri-axial accelerometer detects possible fallsand the pressure sensor is used to verify them.

New generation of smartphones have buil-in motion andposition sensors which make them useful to detect falls.Sposaro et al in. [23], and Cao et al in. [3] use the datafrom the accelerometer and a threshold based algorithm todetermine when a fall occurred. The problem of these systemsis the elevated number of false alarms, since the accuracyof smartphone’s sensors is not as good as the specializedwearable devices. Dai et al in. [6] combine the data from thetri-axial accelerometer and the orientation sensor for a mostreliable fall detection based on the total acceleration and thevertical acceleration. Viet et al in. [27] went one step furtherand uses the orientation data to determine if the fall wasforward, backward or aside. This app requires the phone tobe upright for correct calibration.

B. Ambient analyzers

The ambient sensor approach analyzes the data from dif-ferent environment sensors. Litvak et al in. [14] designed asolution based on floor vibration and sound signals. Rimmi-nen’s et al in. [22] solution also uses a floor sensor but inthis case, it tracks the near electrical field. Ariani et al in. [1]uses the infrared and pressure sensor data whereas Tomii et alin. [26] uses the data from two doppler sensors.

C. Motion detectors

Video analysis, visible (using video cameras) or infrared(using depth sensor), analyzes the person’s posture and thesudden change of position as an indicator of fall. Thome etal in. [24] proposed a Real Time Multiview Fall DetectionSystem. Lee et al in. [13] use low-cost PC cameras whereasDemiroz et al in. [8] use multi-omnidirectional high-resolutioncameras. Ozcan et al in. [20] proposed a fall detection andactivity classification system by a wearable embedded smartcamera which monitors all daily activities. Chen [5] developeda Fall Detection and Location system using ZigBee sensornetworks.

Recently, new solutions using the Kinect sensor have beendeveloped. Kawatsu et al in. [12] developed a sytem to detectfalls by tracking some joints of the individual. It uses skeletonmode include in the Microsoft SDK. Ozcan et al in. [19]

-89-


3

utilizes the RGB and depth data from the Kinect to capturehuman silhouettes in order to detect falls. Nghiem et al in. [16]proposed an algorithm to detect falls by tracking and analyzingthe head position captured with the RGB camera. Mastorakiset al in. [15] proposed a solution tracking individual’s move-ments using the infrared sensor.

D. Hybrid systems

Hybrid systems combine two or more techniques in orderto provide better solutions in terms of accuracy and efficiency.Doukas et al in. [9] utilize video, audio, and motion datacaptured with wearable sensors, cameras, and microphonearrays. Popescu et al in. [21] combine a motion detectorand the sound heigh-information. Grassi et al in. [10] useToF cameras and a wearable wireless accelerometer. DellaToffola [7] developed a platform which integrates on-bodyand environment sensors connected to a home robot. In casea fall is detected, an alert is relayed to the robot whichenables mobility inside the home while communicating withthe external world.

IV. PRELIMINARY EXPERIMENTS

The proposed fall detection system integrates an Androidphone and the Kinect sensor. The detection of the fall takesplace on the phone. Preliminary experiments were performedto set up the parameters of the fall detection system.

A. Which sensors to use?

The Android built-in sensors measures motion, position andenvironmental conditions. In order to detect falls, we moni-tored the device movement and positioning. Figure 3 showsthe acceleration, linear acceleration, rotation and orientationraw data collected while a person is walking and suddenlyfalls down.

5 6 7 8 9 10

0

10

20

time (s)

Acc. (m

/s2)

Acceleration

ax

ay

az

5 6 7 8 9 10

0

10

20

time (s)

Acc. (m

/s2)

Acceleration (geometric mean value)

5 6 7 8 9 100

20

40

time (s)

Acc. (m

/s2)

Linear Acceleration

gx

gy

gz

5 6 7 8 9 100

20

40

time (s)

Acc. (m

/s2)

Linear Acceleration (geometric mean value)

5 6 7 8 9 10−300

−200

−100

0

100

time (s)

Rot. (

degre

es/s

) Rotation

ωx

ωy

ωz

5 6 7 8 9 100

200

time (s)

Rot. (

degre

es/s

) Rotation (geometric mean value)

5 6 7 8 9 100

200

time (s)

Or.

(degre

es)

Orientation

yaw (θz)

pitch (θy)

roll (θx)

5 6 7 8 9 100

500

time (s)

Or.

(degre

es)

Orientation (geometric mean value)

Fig. 3: Sensor readings while an individual is walking andsuddenly falls down.

The raw acceleration data is collected using the tri-axialaccelerometer, which is a hardware-based sensor. To detect afall we calculate the geometric mean value (total acceleration)using (1).

|AT | =√a2x + a2y + a2z (1)

The linear acceleration is the acceleration force that isapplied to the device on all three physical axes excludingthe force of gravity. This software-based sensor derives thedata from the accelerometer and the gravity sensors whichat the same time derive the data from the accelerometerand the magnetometer or gyroscope. Because of the erroraccumulation from all the sensors, the raw data from theaccelerometer is more reliable than the linear accelerometer.

The rotation, or angular velocity, is collected usingthe hardware-based tri-axial gyroscope. Since the phone’scoordinate-system is defined relative to the phone and the axesare not swapped when the device’s orientation changes, thegeometric mean value (2) is more useful.

ωT =√ω2x + ω2

y + ω2z (2)

The orientation of the phone is determined by the azimuth(angle around the z-axis), the pitch (angle around the x-axis)and the roll (angle around the z-axis). The orientation sensorwas deprecated in Android 2.2 so there are a few alternativesto acquire the device orientation. The easiest alternative isby using the rotation vector, a software-based sensor whichderives the data from the magnetic field and the accelerometer.However, not all Android phones have this sensor so an alter-native is manually integrating the data from the magnetometerand the accelerometer. Another alternative consist on a low-pass filtering of the accelerometer and magnetic field sensorsignals and high-pass filtering of the gyroscope signals. Thisavoid both gyro drift and noisy orientation obtained with theaccelerometer and magnetometer.

B. Which prediction mode to use?

Once the data is collected, different prediction modes areapplied. Figure 4 shows the effects of four filters on thegeometric value of the acceleration.

The median filter refers to the replacement of a point bythe median value of the series. The polynomial filter predictsnew values based on finding the coefficients of a polynomialof degree n that fits the data. The low pass filter consists of a3r order Butterword IIR low-pass filter with a cutoff frequency4Hz, since most of the energy contained while walking isbelow 3Hz. The Kalman filter is a prediction and smoothingfilter based on a linear weighed average of the values. Thefilter computes the average µn of the series given the first nvalues. When a new point is measured it recomputes µn using(3), where xn+1 is the new value measured, µn is the oldaverage, µn+1 is the predicted value and K = 1

n+1 .

µn+1 =n

n+ 1µ+

1

n+ 1xn+1 = 1 +K(xn+1 − µ) (3)

Based on the results of Figure 4 and Figure 6 we determinedthe median filter and Kalman filter are good approaches to

-90-


4

5 5.5 6 6.5 7 7.5 8 8.5 9 9.5 10 10.5

0

10

20

time (s)

Acc.(

m/s

2)

Polynomial filter on At

Real

Polynomial

5 5.5 6 6.5 7 7.5 8 8.5 9 9.5 10 10.5

0

10

20

time (s)

Acc. (m

/s2)

Median filter on At

Real

Median

5 5.5 6 6.5 7 7.5 8 8.5 9 9.5 10 10.5

0

10

20

time (s)

Acc.(

m/s

2)

Low pass filter on At

Real

Low pass

5 5.5 6 6.5 7 7.5 8 8.5 9 9.5 10 10.5

0

10

20

time (s)

Acc. (m

/s2)

Kalman filter on At

Real

Kalman

Fig. 4: Effects on At for different prediction modes.

predict future sensor readings. Figure 5 compares the numberof false alarms and miss detections of the two prediction filtersfor different sample rates and positions of the phone. Theresults for the location of the pocket were removed to makethe plot clear, but the obtained results were similar than otherlocations.

Since we suggested the use of the Kinect to verify falls,lower miss detections are more important than higher falsealarms. Based on this principe, Kalman filter gives us betterresults for our system.

20 40 60 80 1002

4

6

8

10

12

14

16False Alarms

% F

A

sample rate (Hz)

waist (Kalman)

waist (Median)

chest (Kalman)

chest (Median)

20 40 60 80 1002

4

6

8

10

12

14Miss Detections

% M

D

sample rate (Hz)

waist (Kalman)

waist (Median)

chest (Kalman)

chest (Median)

Fig. 5: Evaluation of the prediction mode at different samplerates.

C. Which sample rate to use?

The election of the sample rate determines the accuracy ofthe system. It also affects the battery life of the smartphone. AsFigure 6 shows, higher sample rates generate less error duringthe prediction. However, in terms of battery consumption,lower sample rates are better: collect data from sensors every40ms (25Hz) uses less battery than every 10ms (100Hz). Inorder to determine the best sample rate for our system, we ranthe same algorithm at different sample rates. As you can seein Figure 5, higher sample rates generate better results. Based

on all of these experiments and due to battery concerns, wechose a sample rate of 50Hz. This sample rate offers goodresults in terms of miss detections, false alarms and batteryconsumption.

0 20 40 60 80 1000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Sample Rate (Hz)

Err

or

PolyfitKF

MedianLowpass

Fig. 6: Error when applying different filters on At at differentsample rates.

D. Where to trigger the Kinect?

There are two ways to use Kinect on the computer: theofficial SDK from Microsoft and OpenNI, the open sourceversion. The Microsoft SDK includes several features notincluded in OpenNI and the installation and support is better.However, OpenNI is multi platform (Windows, MacOS andLinux) and apps can be programmed in several languages,whereas the official SDK only supports C++, C# and VisualBasic on Windows. Both SDKs can be used on Matlab:OpenNI requires the installation of a set of C++ wrapperfunctions whereas the Microsoft’s SDK is integrated on thenew Matlab’s release.

Figure 7 compares the delay of the different SDKs whenwe use different methods to trigger the Kinect. The first ex-periment triggers the sensor from Matlab by starting a Kinectapplication, which is an executable file. Next experimenttriggers the sensor from Matlab using the integrated functions.Third approach triggers the device from the server using thecompiled file. Based on these experiments, we determined totrigger the Kinect from the server using the Microsoft SDK,which give us better results in terms of delay.

E. Whether compute the data?

The phone acquires the data from the sensors. These sensorreadings are processed and analyzed to identify falls. Bothcomputer or smartphone can be used to analyze the data.

First approach computes all the data on the computer. Thephone acquires the data from the sensors and it sends it tothe computer over a TCP server. Matlab reads the valuesand analyzes them to identify if a hall has taken place. Ifa fall is detected, it triggers the Kinect. Based on preliminaryexperiments, we use the official SDK from Microsoft and wetrigger the Kinect from Matlab due to delay concerns.

-91-


5

Matlab (exe) Matlab (int.) Server(comp.)0

0.5

1

1.5

2

2.5

3

3.5

4

tim

e (

s)

Microsoft SDK

OpenNI

Fig. 7: Comparison of the delay when triggering the Kinect.

The second approach analyzes the data on the phone. All theprocesses, including the detection algorithm, are performed onthe phone. If the fall is detected, it connects to the computer totrigger the Kinect. In this case, we trigger the kinect from theserver using the Microsoft SDK which give us better resultsin terms of delay.

Figure 8 compares the delay of both approaches at differentsample rates. The total delay is divided into the time it takesto analyze the data (tprocess), connect to the server (tconnection),send the data (ttransmission) and trigger the Kinect (ttriggerKinect).Based on this results, we decided to compute the data on thephone and trigger the Kinect if a fall is detected.

C S 25Hz C S 50Hz C S 75Hz C S 100Hz0

1

2

3

4

5

6

7

8

9

10

tim

e(s

)

tconnection

ttransmission

tprocess

ttriggerKinect

Fig. 8: Comparison of the delay of the system to determinewhere to compute the data.

V. ALGORITHM

Falls are usually characterized by changes in motion andbody position of an individual. The useful data to determineis a fall occurred is the acceleration, orientation and rotation.Figure 9 shows the characteristic pattern of these valuescollected when an individual falls down.

The typical acceleration pattern when a person falls is acombination of free fall, high impact and damping effect. Therotation, collected with a gyroscope, is characterized by largevalues with a high peak at the moment of impact. The fallusually ends with the person lying horizontally on the floor,which means a change of 90 degrees in the orientation x- or

y-axis. All proposed algorithms are based on identifying thesepatterns.

Fig. 9: Typical patterns of total acceleration, rotation andorientation when an individual falls down.

A. Partial combinations

Algorithm 1 was proposed for Dai et al in. [6]. Thealgorithm uses the acceleration readings in directions of x-,y-, and z-axis to calculate the total acceleration of the phone(4).

|AT | =√a2x + a2y + a2z (4)

It also uses the yaw, pitch and roll values acquired with theorientation sensor to obtain the amplitude of the accelerationat the absolute vertical direction (5).

|Av| = |Ax sin θz +Ay sin θy +Az cos θy cos θz| (5)

-92-


6

This fall detection algorithm is based on the values of AT andAv . We calculate the amplitude between the minimum andthe maximum readings of the acceleration within a triggeringwindow. If there is a fall, these values correspond to thefree fall and high impact patterns. If this difference exceedsa defined threshold, we calculate the amplitude between thenext two peaks while the damping effect is taking place.If this difference is high than another threshold a fall isconsidered detected. The same rule applies to Av but thedefined thresholds are different.

Algorithm 2 is a threshold-based algorithm on the totalacceleration AT (4) and rotation ωT (6), collected withthe accelerometer and the gyroscope. The fall is considereddetected if the acceleration reading exceeds the high and lowdefined thresholds within a triggering window. The rotationalso has to exceed a defined threshold.

ωT =√ω2x + ω2

y + ω2z (6)

Algorithm 3 uses the acceleration and orientation readings.It assumes than a fall ends with the individual lying on thefloor, which means a change of 90 degrees in the x- or y-axis. The fall is detected if the acceleration readings exceedsboth thresholds (high and low) within a triggering window andthere is a change of 90 degrees in the orientation.

Figure 10 shows the evaluation of the three algorithms whenthe phone is placed in different locations: chest, waist andpocket. Algorithm 3 performs better results in terms of falsealarms and miss detections.

waist chest pocket0

5

10

15False alarms

% F

A

A1

A2

A3

waist chest pocket0

5

10

15Miss detections

% M

D

Fig. 10: Comparison of the proposed algorithms for differentphone locations.

B. Our contribution

The proposed algorithm (Algorithm 3) defines thresholds onAT and a change of 90 degrees in the orientation. The rateof false alarms and miss detections is lower than the otherproposed algorithms for all phone locations but these ratescan be better. The acceleration and orientation readings onthe different phone locations are slightly different; definingdifferent thresholds for these three locations (chest, waist andpocket) the results will be better. We adjusted the thresholds

in order to reduce false negative while simultaneously keepfalse positive in an acceptance range. Figure 11 shows theROC curve for different thresholds Thh when the Thl isfixed at 0.6G, where G = 9.81m/s2. Data is from backwardfalls and the phone is placed at the position of chest. Thesame evaluation was done for the position of waist and pocketobtaining the high and low thresholds showed in Table I.

Location Thh Thl

chest 1.7G 0.8Gwaist 1.3G 0.7Gpocket 1.4G 0.6G

TABLE I: Thresholds for different phone locations.

Fig. 11: ROC curve for different thresholds Thh when the Thlis fixed at 0.6G (G = 9.81m/s2). Phone is placed at chest.

VI. PERFORMANCE EVALUATION

To validate the system we used a dataset created from theGerman Aerospace Center [4] with the help of MIT. Thedataset contains human activities such as sitting, standing,waking, jumping, and falling. The data was collected from tendifferent people and the phone was placed in three differentlocations: in a shirt pocket (chest), on the belt (waist), and inthe pants pocket (pocket). The data was acquired with a Sam-sung Galaxy S2 and consist on the acceleration, orientation,and rotation readings while ten different people is performingall the activities. We used the Nexus S from Google to analyzethese sensor readings.

The first approach of the system only uses the proposedalgorithm which runs on the Android phone. Further versionswill use the Kinect to verify if the fall has taken place, aswe already discussed. However, the Kinect is used to takea picture and send it to the phone, to broadcast it with theemergency alert. The Kinect doesn’t affects the accuracy butit does affect the speed of the system.

A. Accuracy

We measured the accuracy in terms of false alarms (FA) andmiss detections (MD). False alarms happens when the systems

-93-


7

senses a fall but it did not occur. Miss detections happens whena fall occurs but the system misses it. In general, the lowerboth FA and MD are, the better the performance is.

After adjusting all the parameters such as the sample rate,the prediction mode, the algorithm, the thresholds, etc. weobtain false alarm and miss detection rates below 5% for allphone locations. Figure 12 details the rates obtained with thedataset.

waist chest pocket0

1

2

3

4

5

6

7False alarms

% F

A

waist chest pocket0

1

2

3

4

5

6

7Miss detections

% M

D

Fig. 12: Accuracy of the system.

B. Speed

Our system integrates an Android-based smartphone and theMicrosoft Kinect, which are connected over a TCP/IP network.Some of the activities significantly effect the speed of thesystem: the analysis of the data on the phone, the connectionto the Kinect over TCP, and the time while the Kinect starts.Preliminary experiments were done to reduce the delay ofthese activities. We determined, for example, to use the phoneto analyze the data and connect to the Kinect if the fall isdetected. All parameters and applications were designed tominimize the delay between the fall is detected and reported.

We evaluated the speed of the system in a 5Mb/s ADSLnetwork, which is an standard speed for home networks. Weran the system several times and we calculated the mean valuesof the delays for all processes of the system. Table II detailsthis delay. The system runs in less than 7 seconds, but if weuse without the Kinect it needs less than 4 seconds. Both timesare acceptable delays to report a fall.

Process time (sec)Smartphone (data acquisition and analysis) 1.53Connection to Server 1.28Start Kinect 3.46Send data to phone 0.83Get location and send SMS/MMS 2.23

TABLE II: Delay of the main processes

VII. COMPARISON WITH OTHER FALL DETECTION APPS

After validating the system with the dataset, we comparedour app with other apps available on Google Play. The appsthat were compared were chosen based on the number ofreviews and user’s rating.

A. Accuracy

We installed all apps on the phone and we ran every app for12 hours to track the number of false alarms while using thephone in the author daily life. To compare the number of missdetections, 4 different subjects pretended to fall down whilewalking. We tracked 4 falls (forward, backward, left-side andright-side). Figure 13 compares the number of false alarmsdetected in 12 hours and the number of miss detections for 16falls.


5

10

15

False alarms

# F

A


1

2

3

4

5

6

7

8

9

10

Miss detections

# M

D

Fig. 13: Comparison of the accuracy of KiFall and three otherfall detection apps. The results corresponds to the number offalse alarms detected in 12 hours and the number of missdetections for 16 falls.

Based on these tests, our app achieves better results thanother available apps in terms of accuracy. After the subjectspretended to fall down while walking, only one fall was notdetected. However, 6 false alarms were also detected when theapp was running for 12 hours. Our system integrates a Kinectso these false alarms are mitigated since the emergency contactwill receive a picture of the subject walking instead of lyingon the floor.

B. Battery consumption of the smartphone

Battery life is very important in portable devices. To testthe battery consumption of the different Android apps, wemonitored the power states in two different scenarios: wheneach fall detection app was running and when it was not. Thephone was fully charged for all the experiments. Figure 14compares the curves of battery level states during 8 hours.

As Figure 14 shows, our app consumes 48% of battery after8 hours running. The app can run 15 hours before consumingall the battery, which means the user has to charge the phoneevery day. These results are similar than other apps becausethe consumption of battery is due to the data acquisition withthe accelerometer, gyroscope and other sensors.

VIII. DISCUSSIONS

The Microsoft’s Kinect is a powerful sensor with motiontracking capabilities. In theory it can track motion of the userwith the depth camera, which might make the device useful todetect falls. In order to do that, the device must be activated

-94-


8

0 1 2 3 4 5 6 7 80

10

20

30

40

50

60

70

80

90

100

time (h)

Ba

tte

ry le

ve

l (%

)

no apps

Smart Fall Detector

Fall Detector

iFall

KiFall

Fig. 14: Comparison of the battery consumption of kiFall andthree other fall detection apps.

and collecting data from the RGB or depth camera, and theindividual must remain in the Kinect’s field of view. Due to allthese limitations, it makes sense the use of an smartphone withbuild-in motion sensors to detect falls and only start the Kinectto verify them. The fall is considered detected if both devicesdetermines that the fall occurred. Integrating both devices, wecould get better results in terms of accuracy and reliability.

A. Point the Kinect: indoor location

Once the fall is detected on the smartphone, the Kinect canverify it. To do that, the user must be in the field of viewof the sensor. The angular field of view of the RGB cameracan go up to 57◦ horizontal and 43◦ vertically. The viewingdistance range form the depth sensor goes from 0.8m to 3.5m.These limitations can be overcomed by placing the sensor inan appropriate location in the room and using the motorizedpivot. The motorized pivot can tilt the sensor 27◦ either up ordown and it is possible to integrate a pivot to turn the Kinectin the horizontal direction.

Pointing the Kinect to the individual it is not difficult;the problem resides in how to determine the position of theindividual. Reliable indoor positioning is the main challenge.Different techniques could be used taking advantage of thesmartphone’s capabilities.

The first approach consist on determine the distance andbearing (angle between the forward direction) between twopoints. The distance and bearing can be calculated through theA-GPS, a set of location sensors that combines GPS location,network positioning and cellular positioning. Figure15 showshow to determine the angle between two points. This approachis not useful to point the Kinect in indoors because of theaccuracy of the location in indoors (10m).

The second approach uses the double integration of thelinear acceleration (acceleration of the device minus the ac-celeration due to gravity) to measure position (7). However,double integration amplifies acceleration noise so fast. The

Fig. 15: Distance and bearing between two points.

acceleration readings from the smartphone’s sensors are verynoisy, making this approach useless in our system.

x =

∫v · dt =

∫ ∫a · dt (7)

The third approach consists of the development of algo-rithms based on the detection of steps and heading directionswhile walking. The positioning might be reliable using theaccelerometer and gyroscope sensors, and with an accurateestimation of the step length.

B. Verify the fall with the Kinect

Although the proposed algorithm running on an Androidphone presents high accuracies, verify the fall with the Kinectcould improve the results. The Kinect could provide an accu-rate control of some issues, specially the fall-like events suchas jumping or sitting.

The fall verification could consist of detect some joints(head and hands, for example) using the skeleton trackingand calculate the distance from the floor. The fall will beconsidered detected if the distance from the floor is almost0 for all joints. We performed several experiments using bothMicrosoft and OpenNI SDKs but the main challenge is todetect the joints when the Kinect turn on and the person isalready lying on the floor. The algorithm gives good resultsafter small movements of the individual, but sometimes theperson remains unconscious after a fall which makes thisapproach not useful.

We also performed some experiments using OpenNI andopen source libraries. The fall verification consists of detect theindividual using the depth data. The libraries allow to segmentthe user from the background. Once the individual is selected,we checked if the individual’s bounding box is less tall than athreshold value and the position of the highest point is lowerthan a threshold, which means than the user is lying on thefloor. This approach has the same limitation: pickup the personif is already lying on the floor when the Kinect turns on.

IX. CONCLUSION

In this paper, we propose a fall detection system usingan Android-based smartphone and the Kinect sensor. Thetechnique is based on the principle of detecting changes inmotion and body position of an individual. The user is requiredto wear a smartphone which tracks acceleration and orientationchanges. The data is continuously analyzed on the smartphone

-95-


9

to determine wether the individual’s body is falling or not. Ifan individual falls, the smartphone triggers the Kinect to takea picture, and employs GPS to determine the location. Anemergency alert is issued in order to get assistance. The alertis sent through the phone and consist of a multimedia message(MMS) including the location (coordinates and a link to googlemaps), the picture, and the time when the fall occurred.

We designed the detection algorithm and implemented iton a Nexus S running Android 4.1.2. The smartphone canbe placed on different locations -pocket, waist and chest-presenting high accuracies for all of them. The algorithm wasvalidated with the dataset created from the German AerospaceCenter with the help of MIT [4], obtaining false alarmand miss detection rates below 5% for all phone locations.Experimental results show that the Android app achievesgood detection performance and battery consumption. Thedeveloped Android app was compared with other fall detectionapps available on Google Play, obtaining better results in termsof accuracy and battery consumption.

This is the first system which integrates a smartphone andthe Kinect sensor for reliable fall detection. The system, usingboth devices, needs 7 seconds to report a fall. Without the useof the Kinect, it takes less than 4 seconds since the individualfall and the emergency alert is sent. However, the integrationof a smartphone and the Kinect sensor presents many excitingopportunities for the future of fall detections and emergencyreportedly.

REFERENCES

[1] A. Ariani, S.J. Redmond, D. Chang, and N.H. Lovell. Softwaresimulation of unobtrusive falls detection at night-time using passiveinfrared and pressure mat sensors. In Engineering in Medicine andBiology Society (EMBC), 2010 Annual International Conference of theIEEE, pages 2115–2118, 2010.

[2] Woon-Sung Baek, Dong-Min Kim, F. Bashir, and Jae-Young Pyun.Real life applicable fall detection system based on wireless body areanetwork. In Consumer Communications and Networking Conference(CCNC), 2013 IEEE, pages 62–67, 2013.

[3] Yabo Cao, Yujiu Yang, and Wenhuang Liu. E-falld: A fall detectionsystem using android-based smartphone. In Fuzzy Systems and Knowl-edge Discovery (FSKD), 2012 9th International Conference on, pages1509–1513, 2012.

[4] German Aerospace Center. Human activity recognition dataset with in-ertial sensors. http://www.dlr.de/kn/en/desktopdefault.aspx/tabid-8500/,2012.

[5] Tso-Cho Chen. Fall detection and location using zigbee sensor network.In Cross Strait Quad-Regional Radio Science and Wireless TechnologyConference (CSQRWC), 2011, volume 2, pages 937–941, 2011.

[6] Jiangpeng Dai, Xiaole Bai, Zhimin Yang, Zhaohui Shen, and DongXuan. Perfalld: A pervasive fall detection system using mobile phones.In Pervasive Computing and Communications Workshops (PERCOMWorkshops), 2010 8th IEEE International Conference on, pages 292–297, 2010.

[7] L. Della Toffola, S. Patel, Bor rong Chen, Y.M. Ozsecen, A. Puiatti,and P. Bonato. Development of a platform to combine sensor networksand home robots to improve fall detection in the home environment.In Engineering in Medicine and Biology Society,EMBC, 2011 AnnualInternational Conference of the IEEE, pages 5331–5334, 2011.

[8] B.E. Demiroz, A.A. Salah, and L. Akarun. Fall detection using multi-omnidirectional cameras. In Signal Processing and CommunicationsApplications Conference (SIU), 2013 21st, pages 1–4, 2013.

[9] C.N. Doukas and I. Maglogiannis. Emergency fall incidents detectionin assisted living environments utilizing motion, sound, and visualperceptual components. Information Technology in Biomedicine, IEEETransactions on, 15(2):277–289, 2011.

[10] M. Grassi, A. Lombardi, G. Rescio, M. Ferri, P. Malcovati, A. Leone,G. Diraco, P. Siciliano, M. Malfatti, and L. Gonzo. An integrated systemfor people fall-detection with data fusion capabilities based on 3d tofcamera and wireless accelerometer. In Sensors, 2010 IEEE, pages 1016–1019, 2010.

[11] Yibin Hou, Na Li, and Zhangqin Huang. Triaxial accelerometer-basedreal time fall event detection. In Information Society (i-Society), 2012International Conference on, pages 386–390, 2012.

[12] Christopher Kawatsu, Jiaxing Li, and C.J. Chung. Development of afall detection system with microsoft kinect. In Jong-Hwan Kim, Eric T.Matson, Hyun Myung, and Peter Xu, editors, Robot Intelligence Tech-nology and Applications 2012, volume 208 of Advances in IntelligentSystems and Computing, pages 623–630. Springer Berlin Heidelberg,2013.

[13] Young-Sook Lee and HoonJae Lee. Multiple object tracking for falldetection in real-time surveillance system. In Advanced CommunicationTechnology, 2009. ICACT 2009. 11th International Conference on,volume 03, pages 2308–2312, 2009.

[14] D. Litvak, Y. Zigel, and Israel Gannot. Fall detection of elderly throughfloor vibrations and sound. In Engineering in Medicine and BiologySociety, 2008. EMBS 2008. 30th Annual International Conference ofthe IEEE, pages 4632–4635, 2008.

[15] Georgios Mastorakis and Dimitrios Makris. Fall detection system usingkinects infrared sensor. Journal of Real-Time Image Processing, pages1–12, 2012.

[16] Anh Tuan Nghiem, E. Auvinet, and J. Meunier. Head detection usingkinect camera and its application to fall detection. In InformationScience, Signal Processing and their Applications (ISSPA), 2012 11thInternational Conference on, pages 164–169, 2012.

[17] K. Niazmand, C. Jehle, L.T. D’Angekoray13lo, and T.C. Lueth. A newwashable low-cost garment for everyday fall detection. In Engineeringin Medicine and Biology Society (EMBC), 2010 Annual InternationalConference of the IEEE, pages 6377–6380, 2010.

[18] O. Ojetola, E.I. Gaura, and J. Brusey. Fall detection with wearablesensors–safe (smart fall detection). In Intelligent Environments (IE),2011 7th International Conference on, pages 318–321, 2011.

[19] D. Ozcan and O.O. Ergun. Developing an occlusion-resistant automaticfall detection system for smart environments. In Signal Processing andCommunications Applications Conference (SIU), 2012 20th, pages 1–4,2012.

[20] K. Ozcan, A.K. Mahabalagiri, M. Casares, and S. Velipasalar. Automaticfall detection and activity classification by a wearable embedded smartcamera. Emerging and Selected Topics in Circuits and Systems, IEEEJournal on, 3(2):125–136, 2013.

[21] M. Popescu, Yun Li, M. Skubic, and M. Rantz. An acoustic fall detectorsystem that uses sound height information to reduce the false alarm rate.In Engineering in Medicine and Biology Society, 2008. EMBS 2008. 30thAnnual International Conference of the IEEE, pages 4628–4631, 2008.

[22] H. Rimminen, J. Lindstrm, M. Linnavuo, and R. Sepponen. Detectionof falls among the elderly by a floor sensor using the electric nearfield. Information Technology in Biomedicine, IEEE Transactions on,14(6):1475–1476, 2010.

[23] F. Sposaro and G. Tyson. ifall: An android application for fall monitoringand response. In Engineering in Medicine and Biology Society, 2009.EMBC 2009. Annual International Conference of the IEEE, pages 6119–6122, 2009.

[24] N. Thome, S. Miguet, and S. Ambellouis. A real-time, multiview falldetection system: A lhmm-based approach. Circuits and Systems forVideo Technology, IEEE Transactions on, 18(11):1522–1532, 2008.

[25] Marie Tolkiehn, L. Atallah, B. Lo, and Guang-Zhong Yang. Directionsensitive fall detection using a triaxial accelerometer and a barometricpressure sensor. In Engineering in Medicine and Biology Society,EMBC,2011 Annual International Conference of the IEEE, pages 369–372,2011.

[26] S. Tomii and T. Ohtsuki. Falling detection using multiple doppler sen-sors. In e-Health Networking, Applications and Services (Healthcom),2012 IEEE 14th International Conference on, pages 196–201, 2012.

[27] Vo Quang Viet, Gueesang Lee, and Deokjai Choi. Fall detectionbased on movement and smart phone technology. In Computing andCommunication Technologies, Research, Innovation, and Vision for theFuture (RIVF), 2012 IEEE RIVF International Conference on, pages1–4, 2012.

-96-

kifall: detecting falls using an android-based smartphone...

Documents