i

Real Time Wireless Robotic Arm System by Using Leap Motion

Controller

ONG TING YONG

This Report Is Submitted In Partial Fulfillment Of Requirements For

The Bachelor Degree of Electronic Engineering (Computer

Engineering)

Faculty of Electronic and Computer Engineering

University Technical Malaysia Malacca

ii

JUNE 2016

iii

iv

v

Dedicated to my beloved father and mother

vi

ACKNOWLEDGEMENT

First of all, I would like to express my greatest gratitude to Dr. Nurulfajar Bin Abd

Manap, my thesis advisor and also my project supervisor for his guidance and advice

throughout this project. He helped me out with his greatest knowledge in order to solve

some of the problem I faced for my final year project and encourage me to participate

some of the competition in order to gain more valuable experience. Next, I would like to

thanks to my panel examiners Dr. Lim Kim Chuan and Dr. Syafeeza Binti Ahmad Radzi

for spending their valuable time in order to listen my project presentation.

Besides, I would like to thanks to all the staff of the Department of Electronic and

Computer Engineering (FKEKK), UTeM for their guidance and valuable comments.

Nonetheless, I would like to thanks to my teammate supervised under Dr. Nurulfajar Bin

Abd Manap as well for sharing their knowledge and experiences throughout this project.

Last but not least, I would like to thanks to my family and my friends for giving me

support throughout the implementation of this project and I would like to apology for all

the inconvenience that I might had made during this project implementation.

Thanks.

vii

ABSTRACT

This report describe the design and the integration of a 5 Degree Of Freedom

(DOF) robotic arm into a wireless robotic arm system. Nowadays, the evolution of the

technology from time to time bring lots of changes to the human life. One of the

evolution until nowadays is the wireless system. A wireless system is a system that can

convey the information between two or more points and it is not physically connected by

any electrical conductors. The advantages to go wireless can increase the mobility of the

technology, increase the scalability, save cost and also save spaces. In this project, a 5

DOF robotic arm controlled by the Leap Motion controller in wireless system are

proposed. Generally, a robotic arm have the same function as a normal human arm. It

can be used to perform some high-precision task and also handle some hazardous

situation for us. However, most of the robotic arm are wire connected to the system

which limit the mobility of the robotic system and took up more spaces compared with

the wireless system. Hence, the approach to integrate the robotic arm into wireless

system are proposed. All the process to integrate the robotic arm into a wireless system

will be presented in this thesis. Besides, the method to integrate the robotic arm into a

wireless system will be discussed and demonstrated. Furthermore at the end of this

project, a prototype robotic arm in wireless system will be presented and the results for

this project will be recorded into this thesis.

viii

ABSTRAK

Laporan ini menerangkan reka bentuk dan integrasi ke atas robotik lengan yang

mempunyai 5 darjah kebebasan kepada sistem wayarles. Pada zaman sekarang, evolusi

teknologi dari semasa ke semasa telah membawa banyak perubahan kepada kehidupan

manusia. Salah satu evolusi adalah sistem wayarles. Sistem wayarles ini merupakan

salah satu sistem yang menyampaikan maklumat diantara dua atau lebih sistem dan ia

tidak disambungkan secara fizikal oleh konduktor elektrik. Evolusi teknologi dalam

sistem wayarles ini membawa manfaat seperti meningkatkan mobiliti, meningkatkan

produktiviti, menjimatkan kos dan juga menjimatkan ruang. Dalam projek ini, robotic

lengan yang mempunyai 5 darjah kebebasan, dikawal melalui Leap Motion controller

secara sistem wayarless akan dibentangkan. Secara umumnya, robotic lengan ini

mempunyai fungsi seperti lengan manusia biasa. Ia boleh digunakan untuk

melaksanakan tugas yang memerlukan ketepatan tinggi dan juga digunkan untuk

menangani keadaan yang berbahaya untuk kita. Walau bagaimanapun, kebanyakan

robotic lengan yang digunakan adalah dalam sistem berwayar. Ia menghadkan mobiliti

sistem robotik yang digunakan dan ia mengambil ruang yang lebih berbanding dengan

sistem wayarles. Oleh itu, cara untuk mengintegrasikan robotic lengan kepada sistem

wayarles dicadangkan. Proses mengintegrasikan robotic lengan kepada sistem wayarles

akan dibentangkan dalam tesis ini. Selain itu, kaedah- kaedah yang digunakan untuk

mengintegrasikan robotic lengan ini akan dibincangkan. Pada keakhiran projek ini, satu

prototaip robotic lengan dalam sistem wayarles akan dibentangkan dan hasilnya akan

direkodkan ke dalam tesis ini.

ix

TABLE OF CONTENT

CHAPTER CONTENT PAGE

TITLE i

VERIFICATION FORM ii

AUTHOR RECOGNITION iii

SUPERVISOR RECOGNITION iv

DEDICATION v

ACKNOWLEDGEMEN vi

ABSTRACT vii

ABSTRAK viii

TABLE OF CONTENT ix

LIST OF TABLE xii

LIST OF FIGURE xiii

LIST OF ABBREVIATIONS xvii

1 INTRODUCTION

1.1 Background 1

1.2 Problem Statement 3

1.3 Objectives 4

1.4 Scope of Project 4

1.5 Report Structure 5

x

2 LITERATURE REVIEW

2.1 Introduction 7

2.2 Robotic Arm 7

2.3 Leap Motion Controller 9

2.4 NRF24L01 Transceiver Module 11

2.5 Wireless System 12

2.6 Reviews on Related Works 14

2.6.1 Wireless Robotic Arm via Bluetooth Connection 14

2.6.2 Wireless Robotic Arm via Wi-Fi Connection 16

2.6.3 Wireless Robotic Arm via ZigBee Connection 16

2.6.4 Wireless Robotic Arm via Radio Frequency Connection 17

2.6.5 Comparison between Related Works 19

2.7 Summary 19

3 METHODOLOGY

3.1 Introduction 20

3.2 Project Overview 20

3.2.1 Description 20

3.2.2 Block Diagram 22

3.2.3 Project Implementation 23

3.3 Hardware Part 24

3.3.1 Components and Hardware 24

3.3.2 Integration Process 26

3.4 Software Part 29

3.4.1 Leap Motion 29

3.4.2 Node.js 31

3.4.3 Arduino IDE 32

3.4.4 Notepad++ 34

3.5 Programming Part 35

3.6 Summary 36

xi

4 RESULTS AND DISCUSSION

4.1 Introduction 37

4.2 Input Data from Leap Motion Controller 37

4.3 From JavaScript to Arduino 51

4.4 From Arduino Transmitter to Arduino Receiver 60

4.5 Arduino and Circuitry for Both Transmitter and Receiver Module 62

4.6 From Arduino to Robotic Arm 65

4.7 Visualizer 73

4.8 Summary 74

5 CONCLUSION AND RECOMMENDATION

5.1 Conclusion 75

5.2 Recommendation 76

REFRENCES 78

xii

LIST OF TABLES

NO. TITLE PAGE

2.1 NRF24L01 transceiver pin and pin description 12

2.2 Comparison of related work 19

3.1 Connection for NRF24L01 RF module and Arduino Mega 2560 board 27

3.2 Pin connection between Arduino Uno and Arduino Mega 2560 28

3.3 Connection between Arduino Mega 2560 (receiver) and servo robotic arm 28

3.4 Details for EmBoss EB-500W power supply 29

xiii

LIST OF FIGURES

NO. TITLE PAGE

2.1 Robotic Arm with 6 Degree of Freedom (DOF) 9

2.2 Schematic view of the Leap Motion controller 10

2.3 Leap Motion coordinate system 10

2.4 NRF24L01 transceiver module 11

2.5 Robotic system to handle dangerous task 14

2.6 HC-05 Bluetooth module 15

2.7 ZigBee module 17

2.8 NRF24L01 module 18

3.1 Block diagram of the project 22

3.2 PCB board, female header pin and jumper wires 24

3.3 Leap Motion controller 24

3.4 Arduino Mega 2560 and Arduino Uno board 25

3.5 NRF24L01 transceiver module 25

3.6 The hardware for the robotic arm 26

3.7 Leap Motion Control Panel 30

3.8 Leap Motion Visualizer 30

3.9 Node.js command prompt 31

3.10 Standard Firmata code from Firmata library 33

3.11 Receiver code (left) and transmitter code (right) 33

3.12 Notepad++ 34

3.13 Executing the JavaScript coding 34

xiv

4.1 Sample data obtained from the Leap Motion controller when the hand is in

center position 38

4.2 Sample data obtained when the hand is positioned towards the left end 39

4.3 Sample data acquired when the hand moved towards the right end 39

4.4 Sample data acquired when the hand is positioned towards the upper end 40

4.5 Sample data obtained when the hand moved towards the lower end 40

4.6 Sample data acquired when the hand is positioned towards the front end 41

4.7 Sample data obtained when the hand moved towards the back end 41

4.8 Leap Motion controller tracking data in x, y and z axis 42

4.9 X-axis rotational data obtained from Leap Motion Controller 43

4.10 Sample data obtained when the hand's position is in the center of Z rotation 43

4.11 Sample data acquired when the hand rotated upward according X-axis 44

4.12 Sample data obtained when the hand spin downward according X-axis 44

4.13 Z-rotational data obtained when the hand rotated rightward 45

4.14 Z-rotational data acquired when the hand spin leftward 45

4.15 Leap Motion controller tracking data in x rotation and z rotation 46

4.16 Sample data obtained for the middle finger and ring finger 47

4.17 Sample data obtained when the fingers in x-axis opened 48

4.18 Sample data obtained when the fingers opened according y-axis 48

4.19 Sample data obtained when the fingers in z-axis opened 49

4.20 Sample data for finger distance when the fingers are closed 50

4.21 Sample data obtained when the fingers are opened 50

4.22 Sample data obtained when the hand is in center position 52

4.23 Sample data obtained when the hand moved to the left end 53

4.24 Sample data acquired when the hand is in the right end 53

4.25 Sample data acquired when the hand moved to the upper end 54

4.26 Sample data obtained when the hand is in the lower end 54

4.27 Sample data obtained when the hand rotated to the right end 55

4.28 Sample data acquired when the hand rotated to the left end 55

4.29 Sample data obtained when the hand rotated to the lower end 56

4.30 Sample data acquired when the hand rotated to the upper end 56

xv

4.31 Sample data for fingers in normal position 57

4.32 Sample data obtained when the fingers are in closed position 57

4.33 Sample data acquired when the fingers are in opened position 58

4.34 Sample of data when the data are grouped as struct data 58

4.35 Sample data obtained when the hand moved to the left end and the palm of

the hand rotated toward the right end with fingers closed in Leap Motion

controller 59

4.36 Sample data acquired when the hand stand still in middle position and the

palm of the hand rotated toward the left end with fingers opened in Leap

Motion controller 59

4.37 Schematic diagram designed for both transmitter and receiver module in

Arduino board 62

4.38 PCB layout routing for both transmitter and receiver module in Arduino

board 62

4.39 Printed PCB layout for both transmitter and receiver module in Arduino

board 63

4.40 Transmitter board (Arduino Mega 2560+ Arduino Uno + NRF24L01

transceiver module) connected with host PC 64

4.41 Receiver board (Arduino Mega 2560 + NRF24L01 transceiver module)

connected with robotic arm 64

4.42 Orange (Signal pin), Red (VCC pin), Black (ground pin) 65

4.43 Overall connection for the wireless robotic arm system 67

4.44 The robotic arm moved rightward according to the hand's position detected

in Leap Motion controller 68

4.45 The robotic arm moved leftward according to the hand's position detected in

Leap Motion controller 68

4.46 The robotic arm moved upward according to the hand's position detected in

Leap Motion controller 69

4.47 The robotic arm moved downward according to the hand's position detected

in Leap Motion controller 69

xvi

4.48 The robotic palm rotated leftward according to the palm's position detected

in Leap Motion controller 70

4.49 The robotic palm rotated rightward according to the palm's position detected

in Leap Motion controller 70

4.50 The robotic palm moved upward according to the palm's position detected in

Leap Motion controller 71

4.51 The robotic palm moved downward according to the hand's position detected

in Leap Motion controller 71

4.52 The robotic gripper opened according to the fingers detected in Leap Motion

controller 72

4.53 The robotic gripper closed according to the fingers detected in Leap Motion

controller 72

4.54 Visualizer by Leap Motion App Home software 73

4.55 Visualizer using HTML and JavaScript coding 73

xvii

LIST OF ABBREVIATIONS

COM - Communication port

DOF - Degree of Freedom

FIFO - First In First Out

GFSK - Gaussian Frequency Shift Keying

IDE - Integrated Development Environment

ISM - Industrial, Scientific and Medical

PC - Personal Computer

PCB - Printed Circuit Board

PWM - Pulse Width Modulation

RF - Radio Frequency

RX - Receiver

SPI - Serial Peripheral Interface

TX - Transmitter

USB - Universal Serial Bus

Wi-Fi - Wireless Fidelity

1

CHAPTER 1

INTRODUCTION

In this chapter, all the project introduction, background, problem statement,

objectives, scope of project and report structure will be discussed. This project is all

about controlling a real time robotic arm in wireless system by using the NRF24L01

transceiver module and Leap Motion controller. The robotic arm moved according to the

hand detected by the Leap Motion controller while NRF24L01 transceiver module act as

the transmitter and receiver module for the robotic arm system. The background of this

project will be discussed next, continue with problem statement, objectives, scope of

project and the structure of this report.

1.1 Background

Nowadays, the invention of the robotic system grows rapidly from time to time

to ease human in daily activities and serve the world for a better future. In order to build

out a robotic system, the three main branch to develop a good robotic system are

mechanical engineering, electronic engineering and computer science engineering. The

mechanical engineering will handle the hardware part, the computer science engineering

will handle the software part while electronic engineering will handle both hardware and

2

software part. The hardware developer will cover up the design and build up the robotic

system while the software developer will focus on how to function the robotic system.

Different robotic system are designed according to the requirement of the user to

perform specific tasks. The evolution of the robotic system serve human in wide range

of areas such as performing hazardous task for human, environmental interaction and

navigation, resemble human in appearance, education, trainings and much more.

Generally, most of the robotic system are used as industrial robots to perform

human task. It can be used to aid human in performing specific task which could bring

long term or short term harm for humans, aid in hazardous task, dirty task or even boring

task for humans. Besides, robots are widely used in many fields such as underwater

exploration, space exploration, military services, fight against crime, car production,

duct cleaning, fix oil spills, investigate hazardous environment and so on. To design and

construct a robotic system, the most important element to power up a robotic system is

the power source. The power source can be batteries, solar power, pneumatic, nuclear

energy, hydroelectricity and so on. Without the power sources, it is impossible to

function the robotic system. Next, the second element to build up a robotic system is the

actuator. Actuator is the control system for the robotic system and it act as a motor to

rotate and move the robotic system accordingly. Normally, the actuators are powered up

by using the power sources and rotate according to the input signal from the system.

Next, the third element to function the robotic system is the computer programming

code. Programming code are the most important element for the robotic system to move.

Depend on what robot designed, programming code are the essential element to program

a robot on how the robot move according to the input signal. It can be C programming,

C# programming, VB programming, HTML programming, JavaScript programming and

so on.

In the past few years, the most common controller for a robotic system are

keypad, mouse, keyboard or joystick. However, with the growth of the technologies,

numerous of the controller has been invented especially in human-computer interaction

technologies. Besides, most of the robotic system are interconnected via wired

3

connection. With the latest technologies nowadays, the interaction between the human

and the robotic system in wired system can be done as well by using wireless system.

With the integration of wireless communication over the wired communication, the

advantages are greatly increase in the mobility for the robotic system, no cables to mess

up the system, save cost and controllable from far to handle tasks.

The main purpose of this project is to build a wireless robotic arm system

controlled by using a hand gestures controller to control the robotic arm simultaneously.

This robotic arm system will perform according to our hand gesture movement and the

connection between the controller and the robotic arm will be connected via wireless

connection.

1.2 Problem Statement

Although there are various methods to control a robotic arm system, not all

controller are as simplest as possible to control a robotic arm system. Some common

controller to control a robotic arm system include mouse controller, keyboard controller,

joystick controller, keypad controller and so on. Although all these controller are the

most common controller to control a robotic arm system, there are still some difficulties

to use these controller. The difficulties to use these controller include lack of the

understanding about functionality of each button, time consuming to think and move the

robotic arm according ideal movement and hard to control the robotic arm precisely and

accurately. In order to overcome these difficulties, a better solution to control the robotic

arm can be implemented by using the Leap Motion controller. Leap Motion controller, a

hand gestures sensor controller is a new technology that tracks the user's hand

simultaneously with accuracy approximately 0.01mm and a frame rate up to 300 frame

per second. It can detect the user's hand within its surface area of 24 cm² and field of

view up to 150°. It is the best solution to control the robotic arm since it is user friendly,

easy to understand, easy to use and no trainings are required.

4

Besides, the robotic arm nowadays are fixed in a workplace and connected with

the controller via wired connection. If the robotic arm are fixed in a place and connected

to the controller via cable connection, it limit the mobility of the robotic arm to move

from place to place and control it from far distance. Furthermore, exposed cable without

cares could easily damage by the rats, cleaners or even the workers. It may cause power

failure, system failure or even worst is sending someone to the emergency room. With

the integration of the robotic arm into a wireless system, it allow the users to control the

robotic arm from far or handle some dangerous activities such as handling toxic

substances, bomb disposal or exploring hazardous environments. Besides, it can be used

for asepsis preservation in operating room, military service, deep sea discovery or space

discovery to discover the existing and educate the future about the discovery.

1.3 Objectives

The purpose of this project is to build up a robotic arm in wireless system by using the

Leap Motion controller to detect our hand gesture movement and move the robotic arm

simultaneously. In short, the objectives are:

1.3.1 To assemble the robotic arm with Leap Motion controller in wired system.

1.3.2 To integrate the robotic arm system into a wireless robotic arm system.

1.3.3 To develop a real time wireless robotic arm system.

1.4 Scope of Project

The scope of work in this project are divided into three main parts where the first

part of this project will focus on how to get the Leap Motion data from the Leap Motion

controller while the second part of this project will focus on how to utilize the Leap

Motion data to control a 5 DOF robotic arm in wired connection. If the robotic arm in

the second part moved simultaneously according to the hand detected by the Leap

Motion controller, the third part of this project will be the integration of this robotic arm

system into a wireless robotic arm system.

5

For Leap Motion controller, the Leap Motion controller is a hand gesture sensor

in built with two monochromatic IR cameras and three infrared LEDs to interpret the

user's hand accurately and precisely. The Leap Motion data can be obtained via some

programming language and perform some formula calculation to convert it into PWM

signal from 0 to 255 corresponds to the angular range of 0º to 180º [16] for the servo

motors. Next, the PWM signal are feed into a microcontroller to control each servo

motor in the robotic arm simultaneously. When the robotic arm move according to the

hand detected by the Leap Motion controller, NRF24L01 transceiver module can be

added to integrate the robotic arm into a wireless robotic arm system. One will be added

to the microcontroller attached with the robotic arm while another one will be added to

another microcontroller connected with the PC. NRF24L01 transceiver module were

chosen for this project because this module are the cheapest module with high data rate,

low power consumption and support up to 100 meter for wireless communication.

1.5 Report Structure

For the structure of this report, this report consists of five different chapters. The

first chapter of this report will be the introduction of this project and the second chapter

will be the literature review of this project. The methodology used for this project falls

under Chapter 3 while result and discussion of this project will be included into Chapter

4. Lastly, the last part which is the conclusion and the recommendation of this project

will be described in Chapter 5.

In Chapter 1, this chapter will introduce about the introduction of this project.

The background of this project and the problem statement to be solved in this project

were stated. Besides, the objectives of this project and the scope of this project were

mentioned in this chapter. For Chapter 2, this chapter will fully discuss about the

literature reviews related to this project. The literature review gathered from various

sources are divided into two main parts, the first part will be the basic theory and the

principles for this project while the second part will be the overall review from some

journals related to this project.

6

For Chapter 3, this chapter discuss about the methodology used for this project.

In order to achieve the objectives of this project, some problem solving methods and

work are stated here. Moreover, the process to construct and the implementation for the

wireless robotic arm system will be further discussed in this chapter. For Chapter 4, this

chapter identifies the results obtained from this project. Besides, this chapter will present

and explain about the project setup and some sort of problem encountered throughout

the project. Lastly for Chapter 5, this chapter will be the conclusion for this project.

Some future work and recommendations are stated here in order to improve this project.

7

CHAPTER 2

LITERATURE REVIEW

2.1 Introduction

This chapter discusses about some published information regarding this project

and it will cover with some basic theories together with its principles for a better

understanding about this project. Besides, there will be a section discusses about some

published project related to this project and a short comparison between each project.

Furthermore, this part will conclude the entire summary for the literature review and the

alternative to be used throughout this project.

2.2 Robotic Arm

Generally, robotic arm are similar as the human arm and it is programmable

according to the requirement of the users. It is not easy to build up a robotic arm system

because it required the branch of electronic engineering to identify the chips or

component used in the robotic arm, mechanical engineering to design and to construct

the robotic arm, electrical engineering to control the power input and the power output

for each servo motor and computer science to program the operation of the robotic arm

system. Generally, a robotic arm system are classified into two main categories. The first