Building a Thought-Controlled Drone

Jim McKeeth, Embarcadero Technologies jim.mckeeth@embarcadero.com

@JimMcKeeth

Presented at InterDrone Conference

Sept 9th, 2015

About Jim McKeeth• Lead World Wide Developer Evangelist

• Longtime developer

• Object Pascal, Java, C#, JavaScript, Objective-C, etc.

• Invented and patented swipe to unlock in 2000

• US Patent # 8352745 & 6766456, etc.

• Improvisational performer with ComedySportz Boise

• I am not a neuroscientist, but I find this all fascinating!

• Founded 1993

• Provides Architect, Database and Developer Tools

• 3.2 Million Customers including 97% of Fortune 2000

• 35+ Product Awards – Consistent Innovation

• 500+ Employees in 29 Countries

• www.embarcadero.com

Agenda• Drone technical details

• Headset technical details

• Drone API

• Headset SDK

• Concerns, Challenges and the Future

More Information• My email: jim.mckeeth@embarcadero.com

• My blog: delphi.org

• This Slide Deck http://www.slideshare.net/jimmckeeth/build-brain-controlled-drone

• Overview Slide Deck http://www.slideshare.net/jimmckeeth/jim-mc-keeth-wearable-thought-input

• The Code: https://github.com/jimmckeeth/Delphi-Emotiv-EPOC

“In five years, science fiction will be real!”

— Claus Torp Jensen (@ClausTorpJensen) Senior Technical Staff Member and Chief Architect

API Economy at IBM (@ibmapimgt)

What will the world be like in five years?

The PiecesEmotiv EPOC

Brain-Computer Interface Parrot.AR Drone 2 Simi-Autonomous

Connected App Platform

Drone Technical Details

The Drone: Parrot AR.Drone 2.0• Price: $299.99

• Runs Linux internally, but not user accessible

• Contains it’s own WiFi access point

• Connect via UDP & TCP

• Send control strings

• Maintain transmission rate to keep alive

• Can also receive status and video back

Completely RebuildableHD Battery $60

Gears+Shafts $15

Tools $20

Cross bars $30

Screws $7

Propellers $12

4x Engines $200

Bottom structure $60

Main board $110

Nav board $70

Charger $40

Outdoor haul $25

————————

Total $650

New contender: Parrot Bebop• Price: $499.99

• Lots of upgrades, especially camera

• Also contains own WiFi access point

• Whole new SDK

• iOS, Android & Linux

• Published Nov 2014

• Incompatible with AR.Drone 2.0

SDK vs. API• Parrot AR.Drone 2.0 provides both an API and SDK (Bebop has just an

SDK at this point.)

• An SDK is a software library that does additional work locally - makes it easier to handle video streaming, etc.

• An API is a specification of how to connect and control.

• SDK works with specific tools and platforms.

• Typically SDK uses API.

• My examples use the API so it works across platforms.

Brain Computer Interface (BCI) Headset Options

Electroencephalography (EEG)

• Measures electrical activity along the scalp.

• Detects voltage fluctuations resulting from ionic current flows within neurons of brain.

Image source: https://commons.wikimedia.org/wiki/File:EEG_cap.jpg

Neural Oscillation / Brain WavesBe

ta

15-3

0 H

z

Waking Consciousness and Reasoning

Alph

a 7-

14 H

z

Deep Relaxation

Del

ta

4 H

z

Deep Sleep

Thet

a 4-

7 H

z

Light Meditation And Sleeping

Mu

8-13

Hz

Voluntary Movement

Gam

ma

30-1

00 H

z

Conscious Perception

Waveforms by Hugo Gamboa

NeuroSky’s MindWave (also in XWave headsets)

• EEG with 1 dry electrode

• Measures “attention,” “meditation” and eye-blinks

• Bluetooth communication

• SDK for iOS, Android, PC & Mac

• $129.99 to $79.95 retail neurosky.com

• Hacking: http://makezine.com/2011/05/10/behind-the-brain-blinker/

InteraXon’s Muse• 7 dry EEG sensors via Bluetooth

• Focuses on “brain fitness”

• 4 hour battery

• Android, Mac & Windows SDK now, iOS coming soon

• Works with Windows, OS X, iOS, Android, etc.

• $299 at choosemuse.com

OpenBCI• Open source hardware & software

• Bluetooth, Arduino, EEG

• 8 wet Electrodes per board (daisy chainable)

• Full open EEG & BCI platform

• SDK provides EEG data

• www.openbci.com starting at $449

• No iOS or Android yet

Emotiv EPOC• 16 wet electrodes

• 14 EEG electrodes + 2 reference electrodes • Gyroscope to read head movements • 4 mental states, 13 conscious thoughts & facial

expressions – 4 Processing suites • Wireless (encrypted) for Windows, Linux or Mac • $399 for Desktop • $499 for Bluetooth Smart (EPOC+) • www.emotiv.com

Emotiv Insight• Expected eventually

• Starting at $299 (estimate)

• 5 dry sensors + 2 reference

• Bluetooth 4.0 LE (Smart)

• Battery Life: 4+ hours

• SDK: Android, iOS, Mac, Linux and Windows Platforms

• www.emotiv.com

• Comparison http://bit.ly/1cfliTn

Emotiv EPOC SDK Overview

Expressiv Suite• Detect facial expressions

• Eyelid & eyebrow positions

• Wink, blink, furrow

• Horizontal eye movement

• Smile, laugh, clenching & smirking

• EEG sensors picking up signals to muscles (not brain waves)

• Very fast (10ms)

Image Source: https://commons.wikimedia.org/wiki/File:Duchenne-FacialExpressions.jpg

Uses for Facial Detection

• Anything requiring fast input

• Facial gesture detection

• Basic eye tracking

• Fast input & responses

Image Source: https://commons.wikimedia.org/wiki/File:Duchenne-FacialExpressions.jpg

Affectiv Suite• Detect emotions / mental states

• Excitement • Engagement or Boredom • Meditation • Frustration

• Passive detection • Variable levels of each • Short & long term tracking

Image Source: https://commons.wikimedia.org/wiki/File:Drama-icon.svg

Uses for Emotion Tracking• Realtime media evaluation

• Stress analysis and management

• Mood monitoring

• For more accuracy combine with other data sources

• Eye tracking, heart rate, etc.

Image Source: https://commons.wikimedia.org/wiki/File:Drama-icon.svg

Cognitiv Suite• Detect conscious thoughts

• Requires training

• Detect 13 thought patterns with intensity level (4 at a time)

• Left, right, up, down, forward, pull, CW, CCW, left, right, sway-back, sway-forward, Disappear

• Not as fast as Expresiv

Image Source: https://commons.wikimedia.org/wiki/File:ThinkingMan_Rodin.jpg

Uses for Cognitive Thought Identification

• Whole new input method

• Variable intensity for fine control

• Observe subconscious responses

• More training improves accuracy

• Improves the model

• Trains your brain to focus

Image Source: https://commons.wikimedia.org/wiki/File:ThinkingMan_Rodin.jpg

Brain-Computer Interface

Internet of Things

Autonomous Vehicles

expectations

Speech-to-Speech Translation

time

Plateau ofProductivity

Slope of EnlightenmentTrough ofDisillusionment

Peak ofInflated

Expectations

InnovationTrigger

Plateau will be reached in:more than 10 years5 to 10 years2 to 5 yearsless than 2 years

obsoletebefore plateau

Gartner's 2014 Hype Cycle for

Emerging Technologies August 11, 2014

Smart AdvisorsData Science

Prescriptive AnalyticsNeurobusiness

Biochips

Affective ComputingSmart Robots

3D Bioprinting SystemsVolumetric and Holographic Displays

Software-Defined AnythingQuantum Computing

Human Augmentation

Connected Home

Quantified Self

Smart WorkspaceVirtual Personal AssistantsDigital Security

Bioacoustic Sensing

Natural-Language Question Answering

Consumer 3D PrintingCryptocurrenciesComplex-Event Processing

Big DataIn-Memory Database ManagementContent Analytics

Hybrid Cloud ComputingGamificationAugmented Reality

Machine-to-Machine Communication ServicesMobile Health Monitoring

Cloud ComputingNFC

Virtual RealityGesture ControlIn-Memory Analytics

Activity StreamsEnterprise 3D Printing

3D ScannersConsumer Telematics

Speech Recognition

Wearable User Interfaces

Using the AR.Drone API

Command Strings• Connect via WiFi access point to 192.168.1.1 UDP port 5556

• Command string format: AT*[cmd]=[seq],[arg array]

• Sequence number must not repeat

• Argument array is comma separated typically

• Floating point numbers must be encoded as decimals (more later)

• Send command every 30 ms (>2 seconds = session dropped)

Command String Examples• FlatTrim: AT*FTROM=[seq]

• Movement: AT*PCMD=[seq],1,Roll, Pitch, Vert, Yaw

• Values of 1.0 to -1.0

• Use 0 in all values to hover

• Takeoff: AT*REF=[seq],290718208

• Land: AT*REF=[seq],290717696

• Emergency: AT*REF=[seq],290717952

Float as Integer

• Single precision floating point number is 32 bits

• Integer is 32 bits

• Map IEEE-754 bit value from float to integer

• -0.8 = 0xBF4CCCCD = 10000001011001100110011001 = 1085485875

Drone Concerns• Manual override - keyboard override BCI, just in case

• Send emergency command to clear error state

• Start with FlatTrims to level drone

• Use state machine to send command 30 times a second 1. Keyboard override

2. BCI Input

3. Hover

ToDo with Drone• Process NavData (sent back on UDP port 5554)

• Get battery data to know remaining flight time, errors, etc.

• Process Video stream (sent back on UDP port 5555)

• Display on Google Glass for HUD

• Find use for drone “Animations” like flips

• Maybe use LED animations when receiving BCI input

Using Emotiv EDK

SDK Format• On Windows a Visual C++ DLL provides the interface

• Examples in VC++, C#.NET, Java, Python & Matlab

• All my code is in Object Pascal

• On OS X a dylib provides the interface

• Examples in Qt & Xcode

Emotiv EDK Workflow• Call EE_EngineConnect to connect

• Create event handler EE_EmoEngineEventCreate

• Create state structure EE_EmoStateCreate

• Define actions to listen for and then train those actions (more later)

• Every 10 ms call EE_EngineGetNextEvent (more later)

Handle Cognitive Actions• If state is EDK_OK then call EE_EmoEngineEventGetType

• If EventType is not EE_EmulatorError then call EE_EmoEngineEventGetUserId

• If EventType is EE_CognitivEvent then it was a training event (next slide)

• If EventType is EE_EmoStateUpdated then get EE_EmoEngineEventGetEmoState

• Read ES_CognitivGetCurrentAction and ES_CognitivGetCurrentActionPower

Listen & Train• Listening:

• Call EE_CognitivSetActiveActions with mask of Actions

• Call EE_CognitivSetTrainingAction with the ID of the action to train

• Call EE_CognitivSetTrainingControl with COG_START

• On EE_CognitivEventGetType of EE_CognitivTrainingSucceeded call EE_CognitivSetTrainingControl with COG_ACCEPT

New Emotiv Xavier SDK

New Emotiv Xavier SDK

• EPOC+ and Insight are Bluetooth Smart

• Connect to mobile devices - iOS & Android

• Allows to additional user definable cognitive actions

• Still only listens for 4 cognitive actions at a time

Thought Controlled Drone

Thoughts to Movement• Moves while thought active, otherwise hover in position

• Speed of movement = intensity of thought

• How to allow more than 4 directions of movement?

• Use Expressiv facial gesture to switch between sets

• Future consideration: Use strobing grid of controls

Typing via BCIJonathan WolpawWadsworth Center

Albany, NY

~15 seconds / key

. . . . . . . .

. A B C D E F .

. G H I J K L .

. M N O P Q R .

. S T U V W X .

. Y Z 1 2 3 4 .

. 5 6 7 8 9 0 .

. . . . . . . .

BCI Control PanelExample

Purpose of Feedback

• Using a visual feedback gives the wearer something to focus on

• Otherwise they often change their thought pattern when the drone is moving

• Also gives them immediate awareness of what is picked up

Moverio BT-200 Product Overview Integrated Sensors

Accelerometer

Compass

Gyro

Head Set

Camera

Earbuds with

Microphone

+x

+y +z

Slides h"ps://goo.gl/S3uU7V1

Epson Moverio BT-200 Specs

Epson BT-200 Retail Price $699 Display Type Transparent Resolution 960×540 QHD Android v4.0.4 API Level 15 - Ice Cream Sandwich Camera VGA, 640x480 WLAN 802.11b/g/n 2.4GHz Bluetooth 3.0 micro USB v2.0 (On-The-Go) GPS P-GPS

CPU TI OMAP 4460 (ARM Cortex A9 Dual Core 1.2GHz)

RAM 1GB Flash 8GB microSD Max 32GB

Laptop

EDKMy  App

Data Flow

Electrical  

Signals

Wireless  Signal

AT*REF=1,2

90718208

Provides  WiFi    

Hotspot

Feedback

Q & A

More Information• My email: jim.mckeeth@embarcadero.com

• My blog: delphi.org

• This Slide Deck http://www.slideshare.net/jimmckeeth/build-brain-controlled-drone

• Overview Slide Deck http://www.slideshare.net/jimmckeeth/jim-mc-keeth-wearable-thought-input

• The Code: https://github.com/jimmckeeth/Delphi-Emotiv-EPOC