exploring the brain computer interface
DESCRIPTION
Computer-brain interface is a mainstay of science fiction, and devices are available today to use our brainwaves as a computer input. But is it practical? How far away is it? Will "Big Brother" read our thoughts and hack our brains? In this class, we will dive into the future of thought as input for wearable devices with real-world examples and code. Demonstrations will be shown using the Emotiv EPOC headset, a revolutionary high resolution, neuro-signal acquisition and processing wireless neuroheadset that uses a set of sensors to tune into electric signals produced by the brain to detect thoughts, feelings and expressions. You will see the EEG neuroheadset and computer interface with examples of interfacing with desktop, mobile and wearable apps. We will dive into the roots of the technology, showing code and examples along with big pictures of the technology. You will walk away with an understanding of how this still evolving and largely unknown technology really works, how it can be used, as well as longer-term implications.TRANSCRIPT
Android and the Brain-Computer Interface
JIM MCKEETH EMBARCADERO TECHNOLOGIES
[email protected] @JIMMCKEETH
‹#›
About Jim McKeeth• Lead World Wide Developer Evangelist • Host of Podcast at Delphi.org • Longtime developer ◦ Object Pascal, Java, C#, JavaScript, Objective-C, etc.
• Invented and patented swipe to unlock in 2000 ◦ US Patent # 8352745 & 6766456, etc.
• Improvisational performer • I am not a neuroscientist, but I find this all fascinating!
2
‹#›3
• Founded 1993 •Provides Architect, Database and Developer Tools • 3.2 Million Customers including 97% of Fortune 2000 • 35+ Product Awards – Consistent Innovation • 500+ Employees in 29 Countries •www.embarcadero.com
Embarcadero Technologies
‹#›
Agenda•The Science of Reading the Brain
•The Science of Direct Brain Input
•Consumer Brain-Computer Interfaces
•Demonstration - Volunteer flying the Quadricopter
• Summary & MindTunes
4Images from Wikimedia Commons or Product websites.
‹#›
“In five years, science fiction will be real!”
5
-Claus Torp Jensen (@ClausTorpJensen) Senior Technical Staff Member and Chief Architect API Economy at IBM (@ibmapimgt)
What will the world be like in five years?
‹#›
Brain-Computer Interface
6ST:TNG Episode 93 (4x19): The Nth Degree (1991)
‹#›
Brain Reading and Writing
XSQUIDS from Strange Days (1995)
Reading the BrainThe Science of
‹#›
Vocabulary and Roots• Encephalon = Latin. The brain. • -graphy (sufix) = Greek. One who draws or writes. • Encephalography = Drawing (or picture) of the brain • Poly- (prefix) = Greek. Many • Tomos = Greek. Cut into parts or sections. • Neuro- (prefix) = Greek. Relating to nerves or nervous system • Hemo- (prefix) = Greek. Pertaining to blood. • Dynamics = Greek. Active; in motion
X
‹#›
Understanding the Brain
X
Hemodynamics (Blood flow)De
lta
4 Hz
Theta
4-‐7 Hz
Alph
a 7-‐14 Hz
Mu
8-‐13 Hz
Beta
15-‐30 Hz
Gam
ma
30-‐100 Hz
Neu
ral O
scillation
‹#›
Polygraph (aka Lie Detector)•Measures physical manifestations
of stress that come with lying •Blood pressure, pulse, respiration,
skin conductivity, etc.
• Looks for changes.
•Requires specialized training.
• Invented in 1921
8
‹#›
Heart Rate Monitors on Android•Bluetooth GATT Profiles supports
all Bluetooth Smart Heart Rate Monitors.
•Your app doesn’t need to support individual heart rate monitors.
•Assigned Number: 0x180D
9
https://developer.bluetooth.org/TechnologyOverview/Pages/HRP.aspx
https://developer.bluetooth.org/gatt/profiles/Pages/ProfilesHome.aspx
‹#›
Magnetic Resonance Imaging (MRI)• Uses oscillating magnetic fields at
different resonant frequencies. • Functional MRI (fMRI) uses
software to analyze images. • Shows activity (blood flow) in
different parts of the brain. • Combined with brain mapping can
reveal thought information.
10
‹#›
CT / CAT Scan (Computed Tomography X-ray)• Faster & cheaper than an MRI
•Uses ionizing X-ray radiation
• Lower detail images
• Sometimes combined with Positron Emission Tomography (PET) scans which use gamma radiation.
X
‹#›
Near-Infrared Spectroscopy (NIRS)• Spectroscopic use of near-
infrared (700-900nm) light through the skull and detecting degree remerging light is attenuated.
• Detects blood oxygenation levels in different areas of the brain.
• More portable than fMRI but only scans cortical tissue.
X
‹#›
Electroencephalography (EEG)•Measures electrical activity along the scalp.
•Detects voltage fluctuations resulting from ionic current flows within neurons of brain.
•Millisecond-range resolution (faster than CT or MRI)
•Typical clinical usage has 19+ input electrodes as well as ground and reference.
• Invented between 1875 and 1924
11
‹#›
Electroencephalography (continued)• Series of electrodes placed next
to scalp.
•Difference in voltage is measured between each electrode and the reference.
X
‹#›
Magnetoencephalography (MEG)
12
Uses Arrays of SQUIDs (Superconducting QUantum Interference Devices)
Direct Brain InputThe Science of
‹#›
Change Perception of Reality
14The “headjack” from The Matrix movies -‐ 1999
‹#›
Transcranial Direct-Current Stimulation (tDCS)•Constant, low current delivered via electrodes. •Results: ◦ Increase cognitive performance ◦ Enhance language & math ability ◦ Improve attention span ◦ Aid problem solving, memory & coordination •Recent DIY “Brain Hacking” Craze ◦ http://www.reddit.com/r/tDCS
15
Accelerated Learning tdcsplacements.com
http://thebrainstimulator.net/
‹#›
tDCS with Mobile• foc.us just launched new tDCS
headsets with Bluetooth API
•Developer devices are available for preorder
•Targeting athletes and gamers
• http://www.foc.us/index.php/v2-tdcs-developer-edition
• http://www.foc.us/bluetooth-api16
‹#›
Transcranial Magnetic Stimulation (TMS)• Similar strength to MRI •Magnetic induction of electrical
currents •Used to treat neurological conditions •Produces: ◦ Flashes of lights in vision (phosphene) ◦Muscular activity
17
‹#›
Ultrasonic Transcranial Stimulation Device• 2000 patent from Sony (US6584357 B1) ◦ Finer targeting than TMS ◦ Theoretically invoking sensory experiences ◦Continued research and patents (US7350522 B2)
18
http://www.newscientist.com/article/mg18624944.600
Speculative work,
“There were not any
experiments done.”
‹#›
Ultrasonic Neuromodulation for Spatial Resolution• Virginia Tech Carilion
Research Institute study published in Jan 2014 Nature Neuroscience
• Used “targeted ultrasound” to boost “sensory perception”
• Targeted an area of the brain as small as the size of an “M&M”
19http://www.eurekalert.org/pub_releases/2014-‐01/vt-‐udt011014.php
‹#›X
"In neuroscience, it's easy to disrupt things. We can distract you, make you feel numb, trick you with optical illusions. It's easy to make things worse, but it's hard to make them better. These findings make us believe we're on the right path."
-- William "Jamie " Tyler, Assistant Professor Virginia Tech Carilion Research Institute
‹#›20
Using More than 10% of Our Brain?
LUCY2014
We all use 100% of our
Brain Capacity
Interesting MilestonesScience Fiction Becoming Reality
‹#›
Brain-to-Brain Input
22The “Hat” from Brainstorm -‐ 1983
‹#›
Brain-to-Brain Input at University of Washington in 2013• Input from Computer
• EEG from brain
•Transmitted over campus internet
•Replayed through TMS to another brain
•Output to computer
23More information http://bit.ly/EEG2TMS & http://bit.ly/bci2cbi
‹#›
Cybernetic Limbs
XCaptain America: The Winter Soldier -‐ 2014
‹#›
DARPA’s Brain Controlled Prosthetic Arm•Part of DARPA’s Innovation
Pathway.
•Uses microchip implanted in brain.
• 27 different fine grained controls to arm, hand and fingers.
• Expected on market by 2016
Xhttp://www.fastcompany.com/1725799/darpas-‐mind-‐controlled-‐
prosthetic-‐arm-‐could-‐be-‐market-‐four-‐years
‹#›
See What You See - Hear What You Hear
24The “Monitor” from Ender’s Game – 2013
‹#›
See What You See• In 2010 another team lead by Shinji
Nishimoto partially reconstructed YouTube videos based on brain recordings.
• Used fMRI • Overlaid similar videos to create composite video.
• http://youtu.be/nsjDnYxJ0bo & http://youtu.be/EdGjB6XCXOA
25http://bit.ly/ReconstructingVisual
‹#›
See What You See
X
• In 2008 IBM used a feedback loop based on the fusiform face area of the observer’s brain for facial recognition.
•Patent # US20100049076 A1
‹#›
Hear What You Hear• 2012 - UC Berkeley & Brian Pasley decoded
internal neural processing of auditory information.
•Reconstructed sound on computer from electrical signals in observer’s brains.
•Resulting sound was recognizable when compared to original.
26http://bit.ly/ReconstructingSpeech
‹#›
Identifying Thoughts• 2008 & 2009 Using fMRI recognized thoughts
within the human brain. • Discovered similar thoughts appear similar is
different subject’s brains.
• Was able to discern which of 10 objects subject was thinking about.
• Also predicted subject intentions up to 10 seconds before subject expressed them.
Xhttp://bit.ly/IdentifyThoughts
‹#›
Data Entry via BCI
27ST:TNG Episode 93 (4x19): The Nth Degree (1991)
‹#›
Typing via BCI•Brain researcher Jonathan Wolpaw
at Wadsworth Center in Albany, NY
•Wear EEG while looking at a flashing matrix of characters
• Each character flashes with a different pattern
•~15 seconds to type a character
•The Verge http://embt.co/typebci
28
. . . . . . . .
. A B C D E F .
. G H I J K L .
. M N O P Q R .
. S T U V W X .
. Y Z 1 2 3 4 .
. 5 6 7 8 9 0 .
. . . . . . . .
‹#›
Uploading Your Consciousness
29Transcendence -‐ 2014
‹#›
Human Connectome Project•Map of neural connection the brain
•The neurological version of the genome
•Unique to each person, changes over time
•Representation of memories, thoughts, etc.
•www.ted.com/talks/sebastian_seung
• humanconnectomeproject.org
• humanconnectome.org 30Image by Xavier Gigandet et. al.
‹#›
OpenWorm Project•Mapped the 300+ neurons and 7000+
connections of a worm (C. elegans)
•Created a software model of connectome
•Connected to LEGO robotic representation
•Behaved like worm (without training or programming)
• http://bit.ly/openwormlego & http://www.openworm.org/
31
‹#›
As of July 2014
Wearable User Interfaces
Brain-Computer Interface
Internet of Things
Autonomous Vehicles
expectations
Speech-to-Speech Translation
32http://www.gartner.com/newsroom/id/2819918
time
Plateau ofProductivitySlope of EnlightenmentTrough of
DisillusionmentPeak ofInflated
ExpectationsInnovation
Trigger
Plateau will be reached in:more than 10 years5 to 10 years2 to 5 yearsless than 2 years obsolete
before plateau
Gartner's 2014 Hype Cycle for Emerging Technologies August 11, 2014
BCI
Hype Cycle
Brain-Computer InterfacesRead-Only Consumer
‹#›
Consumer BCI Devices• Emotiv EPOC, Insight & EPOC+ www.emotiv.com
•NeuroSky’s MindWave neurosky.com
•OpenBCI www.openbci.com
•Neuroelectrics Enobio www.neuroelectrics.com
• InteraXon’s Muse choosemuse.com
X
‹#›34
• EEG with 1 dry electrode • Measures “attention,” “meditation” and eye-
blinks • Bluetooth communication • SDK for iOS, Android, PC & Mac • $129.99 to $79.95 retail neurosky.com • Hacking: ◦ http://makezine.com/2011/05/10/behind-the-brain-blinker/ ◦ http://hackaday.com/2012/12/20/modifying-an-eeg-headset-for-
lucid-dreaming/
NeuroSky’s MindWave (also in XWave headsets)
‹#›
Emotiv EPOC• 16 wet electrodes ◦ 14 EEG electrodes – read brain waves
+ 2 reference electrodes • Gyroscope to read head movements • 4 mental states, 13 conscious thoughts & facial expressions –
4 Processing suites • Wireless (encrypted) connection to Windows, Linux or Mac • $399 desktop or $499 for Bluetooth Smart (coming soon) • www.emotiv.com
35
‹#›
InteraXon’s Muse• 7 dry EEG sensors via Bluetooth • Focuses on “brain fitness” • 4 hour battery • Basic Mac & Windows SDK now,
full SDK coming soon (incl. iOS & Android)
• Works with Windows, OS X, iOS, Android, etc. • $299 available now • choosemuse.com
36
‹#›
OpenBCI• Open source hardware & software • Bluetooth, Arduino, EEG • 8 wet Electrodes per board (can daisy
chain boards) • Full open EEG & BCI platform • $449 - $799 (now shipping) • www.openbci.com • No iOS or Android yet
37
‹#›
Emotiv Insight• Expected 2015 • Starting at $299 • 5+2 dry sensors • Bluetooth 4.0 LE (Smart) • Battery Life: 4+ hours • SDK: Android, iOS, Mac, Linux
and Windows Platforms
38www.emotiv.com -‐ Comparison http://bit.ly/1cfliTn
‹#›
No More Woof (“work in progress”)• EEG for Dogs
•Recognize brain patterns and match with mental states:
•Hungry, Playful, Curious, etc.
•Translates to human
•By Nordic Society for Invention and Discovery (NSID)
•Raised $22K on IndiGoGo39http://www.nordicinvention.com/nmw.html
Emotiv EPOCMore Details on the
‹#›
Further Specs• Channels: ◦ AF3, AF4, F3, F4, F7, F8, FC5, FC6, P3 (CMS), P4 (DRL), P7, P8, T7, T8, O1, O2
• Battery: ◦ 12 hours operation
• Sampling rate: ◦ ~128Hz (2048Hz internal)
• Resolution: ◦ 16 bits (14 bits effective) 1 LSB = 0.51μV
• Bandwidth: ◦ 0.2 - 45Hz, digital notch filters at 50Hz and 60Hz
• More information: ◦ http://emotiv.wikia.com/wiki/Emotiv_EPOC
X
Int. 10-‐20 locations
‹#›
Head Rotation Suite•Built in solid state gyroscope
•Tracks angular velocity of head movement ◦ Yaw & Pitch ◦No roll detection
• Separate from EEG
X
‹#›
Expressiv Suite• Detect facial expressions ◦ Eyelid & eyebrow positions – Wink, blink, furrow
◦ Horizontal eye movement ◦ Smile, laugh, clenching & smirking • EEG sensors picking up signals to
muscles (not brain waves) • Very fast (10ms)
41
‹#›
Uses for Facial Detection•Realistic avatar expressions
•Anything requiring fast input
• Facial gesture detection
•Basic eye tracking
• Fast input & responses
42
‹#›
Affectiv Suite• Detect emotions / mental states ◦ Excitement ◦ Engagement or Boredom ◦ Meditation ◦ Frustration • Passive detection • Variable levels of each • Short & long term tracking
43
‹#›
Uses for Emotion Tracking•Realtime media evaluation
• Stress analysis and management
•Mood monitoring
• For more accuracy combine with other data sources
• Eye tracking, heart rate, etc.
44
‹#›
Cognitiv Suite• Detect conscious thoughts • Requires training • Detect 13 thought patterns with
intensity level • Left, right, up, down, forward, pull • CW, CCW, left, right, sway-back,
sway-forward, Disappear • Not as fast as Expresiv
45
‹#›
Uses for Cognitive Thought Identification•Whole new input method
•Variable intensity for fine control
•Observe subconscious responses
•More training improves accuracy
• Improves the model
•Trains your brain to focus
46
‹#›
The Pieces
47
Emotiv EPOC Brain-Computer Interface
Google Glass Wearable Computer
Parrot.AR Drone 2 Simi-Autonomous
‹#›
About Appmethod• Integrated Development Environment • Multi-platform visual designer • Native compiler for Win32, Win64, OS X, iOS & Android • Framework for rapid prototyping and accelerated
development with C++ or Object Pascal • Architecture allowing shared code & UI across platforms • Advanced language features like ARC, Generics, etc.
48
www.appmethod.com
Starts at $25/mo
per platform or
C++ Android Phone for
Free
‹#›
Laptop
EDKMy App
How it Works
49
Brain
Electrical
Signals
Wireless Signal
AT*REF=1,2
90718208
Provides WiFi
Hotspot
Feedback
Emotiv EPOC SDK + Parrot AR.Drone +
Google Glass
Demonstration
Tweet @Appmethod and #IoTor G+, LinkedIn, Facebook, etc.
‹#›
Emotiv EPOC SDK•Platforms
◦Windows
◦Mac OS X
◦Ubuntu
◦ Fedora
◦ iOS & Android coming soon
51
• Languages
◦C / C++ / Objective-C
◦C# (.NET)
◦ Python
◦Object Pascal / Delphi
◦ Java coming soon
‹#›
MindTunes• Three disabled musicians • Collaborated with DJ Fresh and
Julien Castet (tech) • Created dance track • http://youtu.be/HyF4ZxGhPHw
• Full documentary • http://youtu.be/PgfxKZiSCDQ
52
‹#›
More Information• Email: [email protected] • Emotiv EPOC: www.emotiv.com
• Parrot AR.Drone: projects.ardrone.org • Appmethod: www.appmethod.com • My blog: delphi.org • Full Slide Deck: http://www.slideshare.net/jimmckeeth/jim-mc-
keeth-wearable-thought-input • Code: https://github.com/jimmckeeth/Delphi-Emotiv-EPOC
53