debs 2015 tutorial when artificial intelligence meets the internet of things
TRANSCRIPT
When Artificial Intelligence meets the Internet of Things. DEBS’15 tutorial Speaker: Opher Etzion
2
The autonomous car
Needs sensors for observing what happens now, needs intelligence to understand what it observes, needs intelligence to drive, needs
actuators to carry out the driving.…
3
Like the human body, we need to sense, to make sense of what we sense, to make constant decisions and to carry them out .
Sensing
Making sense from the sensing
Real-time decision making
Acting
OUTLINE
A quick intro to IoT and its relationship with AI
Some applications of Intelligent IoT
The AI perspective
The future perspective
4
Topic I
TOPIC II
Topic III
Topic IV
OUTLINE
A quick intro to IoT and its relationship with AI
Some applications of Intelligent IoT
The AI perspective
The future perspective
5
Topic I
TOPIC II
Topic III
Topic IV
None of the authorized drivers location is near the car’s location
theft is concluded
Use a built-in car stopper to slow the intruder and dispatch the security company
A person enters a car and the car starts moving;
the person does not look like one of the authorized drivers
Such applicationsbecome possible
since everything isconnected
6
7
The term “Internet of Things” was coined by Kevin Ashton in 1999 .
His observation was that all the data on the Internet has been created by a human .
His vision was: “we need to empower computers with their own means of gathering information, so they can see, hear, and smell the world by
themselves .”
8
The world of sensors
1 Acoustic, sound, vibration2 Automotive, transportation3 Chemical4 Electric current, electric potential, magnetic, radio5 Environment, weather, moisture, humidity6 Flow, fluid velocity7 Ionizing radiation, subatomic particles8 Navigation instruments9 Position, angle, displacement, distance, speed, acceleration10 Optical, light, imaging, photon11 Pressure12 Force, density, level13 Thermal, heat, temperature14 Proximity, presence
9
The value of sensors
Kevin Ashton: “track and count everything, and greatly reduce waste, loss, and cost. We could know when things needs replacing, repairing or recalling, and whether they were fresh or past their best”
The value is in the ability to know and react in a timely manner to situations that are detected by sensors
10
Differences between the traditional Internet to the Internet of Things
Topic Traditional Internet Internet of Things
Who creates content? Human Machine
How is the content consumed?
By request By pushing information and triggering actions
How content is combined?
Using explicitly defined links
Through explicitly defined operators
What is the value? Answer questions Action and timely knowledge
What was done so far? Both content creation (HTML…) and content consumption (search engines)
Mainly content creation
Two separate but connected goals: Awareness and Reaction
Awareness Reaction
Event
Detect Derive Decide Do
Did SomethingHappen?
What should we do about it?
It Happened
Detect
Some Noun of
Importance but different
The act of bringing into a system’s sphere of understanding knowledge about an event.
The detection is done by sensors, instrumentation and human reports.
Swim
Lane
Trigger Event
Activity
StateChange
13
Intelligent Detection
Determine what is actually been sensed: vision understanding, voice understanding, text understanding.
DeriveThe act of becoming aware of events that are not directly detectable by bringing together events with other events, data, patterns and publishing the observation as a derived event.
Raw eventsRaw
eventsRaw events
A Person or a computer recognizes the pattern and enters the derived event or just reacts to it directly.
15
Event processing: making sense from what we sense…
Combining data from multi-sensors to get observations, alerts, and actions in real-time gets us to the issue of detecting patterns in event streams
16
Intelligent derivation
Find the causality between events and situations. We discuss the notion of causality later.
Decide
Automated decision by decision management system
The act of determining the course of action to do in response to the situation. This includes the background information needed to be collected to make the decision.
No Decision
Pass through: Sometimes there is no decision. There is only one course of action.
Automated Goal Oriented: Algorithmic decision via a decision management system that seeks a optimizing quantitative goals.
18
Intelligent Decision
Finding the best decision some times under real-time constraints may require an intelligent process.
DoThe act of performing the course of action that was decided upon.
Notification: Sending a signal of sort to either a person or system. This would include calling a web-service or subscription to alerts.
20
Intelligent Actions
Intelligent actuators
21
Knowledge acquisition for IoT based systems
How do we know how to make sense of all these data?
OUTLINE
A quick intro to IoT and its relationship with AI
Some applications of Intelligent IoT
The AI perspective
The future perspective
22
Topic I
TOPIC II
Topic III
Topic IV
23
IoT and robotics
Robots serve as intelligent actuators
24
Healthcare robotics
Rehabilitation robots: enhancing patients with motoric and cognitive skills
Assistive robots: Robots for independent living of disabled persons
25
Some future healthcare robotics applications
Automated assistance of monitored patients
26
Some future healthcare robotics applications
Help in sit-to-stand and sit-down actions for people with motor disabilities
27
Some future healthcare robotics applications
Autonomous moving of drugs and medical equipment within the hospital
28
Some future healthcare robotics applications
Support of medical staff in various activities
29
Some future healthcare robotics applications
People movement and movement monitoring
30
Some future healthcare robotics applications
People assistance in panic and danger situations
31
I(
The classical use of robots are for industrial purposes: production, machinery control, product design…
Industrial Robots
32
I(
Autonomic management and coordination of production activities among multiple robots
Industrial Robots and IoT
33
I(
Autonomous management of equipment and instruments
Industrial Robots and IoT
34
I(
Immediate reaction to critical situations such as: high temperature, harmful chemicals in the air
Industrial Robots and IoT
35
I(
Autonomic control of electrical and energy plants
Industrial Robots and IoT
36
Robotics for defense
Robots are used for unmanned tools (ground and air) for transport and intelligence , threat detection and combat
37
Robotics for defense and IoT
Autonomous and smart detection of harmful chemicals and biological weapons
38
Robotics for defense and IoT
Autonomic control of land vehicles and aircrafts
39
Robotics for defense and IoT
Identification and access prevention of suspicious people intruding to sensitive places
40
Robotics for defense and IoT
Rescue trapped people
41
The Internet of things for the elderly
and healthcare in general
42
Safety sensors
Motion sensor
Door sensor
ChairSensor
Voice Sensor
Alert family member
Alerts example:Door was not locked within 2 minutes after entranceFalling event detectedVocal distress detectedNo motion for certain time period detected
43
Medical sensors for the elderly
44
E-Health sensors
Personalized alerts based on collection of monitors
45
Pre-mature babies monitoring
Personalized alerts based on collection of monitors: when nurse should be alerted, when physician should be alerted.
There are many false alerts that are ignored, Missing or ignored alert is sometimes fatal
46
Track everything in a hospital
47
Track the progress of a surgery relative to the plan
Detect significant deviation from plan that requires rescheduling and trigger real-time rescheduling of surgeries, assignments, and equipments.
48
Dynamic planning
Example: traffic control; patient treatment; serviceman scheduling
49
AI meets IoT – Apple’s Perspective
Siri was released as Apple’s “intelligent personal assistant”.
A sensor enabled Siri is targeted as a “smart home solution”
50
AI meets IoT – Google’s Perspective
Google acquired a collection of IoT related companies and then acquired AI company DEEPMIND that uses Neural Nets and Reinforcement learning. The aim is to develop a machine with intelligence of a toddler with IoT providing sensing capabilities
51
AI meets IoT – Facebook’s Perspective
Facebook acquires wit.ai – a speech recognition company. Making the Internet of Things voice controlled.
OUTLINE
A quick intro to IoT and its relationship with AI
Some applications of Intelligent IoT
The AI perspective
The future perspective
52
Topic I
TOPIC II
Topic III
Topic IV
53
Vision understanding
Robocup tournament: Robots playing football. Strong vision capabilities are required.
54
Vision understanding
Tracking objects over time from a collection of cameras
55
Vision understanding
Grace, the robot, can communicate with her surrounding, understand gestures, attended conferences, understands that she had to stand in a line, go in an elevator and ask people to press the floor number…
56
Speech recognition
Acoustic analysis, linguistic Interpretation
57
Causality
In order to derive situations from events there is a need to identify causalities.
Statistical methods can infer correlations.
Causality inference is more tricky….
58
Causalities in events
Type I: predetermined causality - Event E2 always (or conditionally) occurs as a result the occurrence of E1, thus we don't need to have any sensor to detect event E2 we may assume it happened if E1 happened (and the condition is satisfied), some time offset or interval may be attached to this causality. Note that in this case E1 and E2 are both raw events.
Necessity and relevance
59
Causalities in events
Type II: The event E1 is an input to a processing element PE and event E2 is an output of PE. In this case E2 is a derived (virtual) event. The specification of PE is part of the system, thus the context and conditions are known.
Necessity and relevance
60
Causalities in events
Type III: The event E1 is an event that is sent from a computerized system to a consumer C. C applies (conditionally) some action AC, where the specification of AC is not known to us, but we observe that it emits the event E2. This is another type of causality (the event E2 would not have been emitted, if E2 would not have triggered AC), however, E2 may or may not have functional dependency with respect to E1
Necessity? and relevance?
61
Causal inference
How the knowledge about causality is being acquired?
Expert knowledge
Statistical inference
Inference using semantic or association net
Necessity? and relevance?
62
Dangers of using correlation as causality indicator
Correlation between A and B:
1. A causes B
2. B causes A
3. There is C which causes both A and B
4. A combination of all three interpretations
The faster windmills are observed to rotate, the more wind is observed to be.
Therefore wind is caused by the rotation of windmills.
63
Dangers of using correlation as causality indicator
Correlation between A and B:
1. A causes B
2. B causes A
3. There is C which causes both A and B
4. A combination of all three interpretations
Sleeping with one's shoes on is strongly correlated with waking up with a headache.
Therefore, sleeping with one's shoes on causes headache.
(correct answer: going to bad drunk causes both)
64
Dangers of using correlation as causality indicator
Correlation between A and B:
1. A causes B
2. B causes A
3. There is C which causes both A and B
4. A combination of all three interpretations
As ice cream sales increase, the rate of drowning deaths increases sharply.
Therefore, ice cream consumption causes drowning. (real answer: they are both in the same context – summer).
Temporal indeterminacy
Inexact indicator Probability
Event did not occur 0.4
Event occurred before T1 0.1
Event occurred in [T1, T2] 0.45
Event occurred after T2 0.05
T1 T2
False positives and negatives
False positive:The pattern is matched;The real-world situation does not occur
False negative:The pattern is not matched;The real-world situation occurs
Learning from experience
67
Data is not good enough…
68
Real-time decision under uncertainty
Robust RTOptimization
Stochastic RTOptimization
Simulation-based RT optimization
Handling event uncertainties
Uncertain whether an reported event has occurred (e.g. accident)
Uncertain what really happened. What is the type and magnitude of the accident (vehicles involved, casualties)
Uncertain when an event occurred (will occur): timing of forecasted congestion
Uncertain where an event occurred (will occur): location of forecasted congestion
Uncertain about the level of causality between a car heading towards highway and a car getting into the highway
Uncertain about the accuracy of a sensor input: count of cars, velocity of cars…
The pattern: more than 100 cars approach an area within 5 minutes after an accident derives a congestion forecasting
Uncertain about the validity of a forecasting pattern
Uncertain about the quality of the decision about traffic lights setting
Predictive Event Processing (1)
VS.
Photo by Michael Gray, Flickr
Predictive Event Processing (2)
VS.
+
Predictive Event Patterns
Pattern Future event, probability, time interval “4 high value deposits from different geographic locations within 3 days”
“0.6 chance for a large transfer abroad, in 1 day”
“Output event will occur with distribution D over interval (t1,t2)”
Stock decrease of > 5% in 3 hours Good chance for 2% increase within 2 hours
Limitations of the use of rules in specifying predictive event patterns
Limitations:1. Partial patterns
2. Uncertain input events
3. Complex relationship between random variables
Rule = hard-coded probabilistic Relationship
Dynamic event predictionTime Series Prediction
Graphical models
Temporal Graphical models
Learning patterns and causalities
EventPatterns
Pattern and causality acquisition
This is a direction to reduce the complexity of application development
There are challenges in doing it – since “detected situations” are “inferred events” and may not be reflected in past events
76
Security challenges of IoT
Getting security feeling is a necessary condition for the success of IoT to become pervasive.
77
Dangers and challenges
Confusing a sensor
Changing the rules of the game
Abusing an actuator
78
Security considerations of IoT
Murder by the Internet
“With so many devices being Internet connected, it makes murdering people remotely relatively simple, at least from a technical perspective. That’s horrifying,” said IID president and CTO Rod Rasmussen. “Killings can be carried out with a significantly lower chance of getting caught, much less convicted, and if human history shows us anything, if you can find a new way to kill, it will be eventually be used.”
EXAMPLES: Turn off pacemakers, Shutdown car systems while driving, stop IV drip from functioning
79
Confusing a sensor
The same as confusing the human eyes. See things that don’t exist, don’t see things that exist, distort picture…
80
Confusing a sensor
Can be used to sabotage anti-crime systems, to commit fraud, or just damage something or someone…
81
Confusing a sensor
Example from another domain: the Twitter hoax
82
Changing the rules of the game
The logic is rule-based. The ease of modification can be abused to add/delete/modify rules, change thresholds…
83
Changing the rules of the game
Changing data relevant for the system: maps, pictures, person’s data…
84
Abusing actuators
Deviating from course, shutting down, activating in wrong mode…
OUTLINE
A quick intro to IoT and its relationship with AI
Some applications of Intelligent IoT
The AI perspective
The future perspective
85
Topic I
TOPIC II
Topic III
Topic IV
TOPIC 4
A futuristic view of the Internet of Things following Ray Kurzweil’s predictions:
86
87Driverless car
Sensors that replace the human driver’s sensing, and actuators that drive the car.
2017
88
Automated personal assistant
Sensors that determine the context serves as active advisors. They understand your context and even listen to your conversations and give you suggestions of what to say (e.g. through google glass).
2018
89
Computing implants inside the human body
Sensors and actuators that fight any disease, operate in the level of cell, and reprogram the body to stop the aging process.
2020
2040
Short term: switch off our fat cells
Longer term: stay young forever
90
May 14, 2014
91
Summary: The Internet of Everything participates in many of the predictions about the future, including Kurzweil’s singularity.
The responsibility is upon us to create this future…
92
My main motivation is to use the experience and knowledge I have accumulated over the years to make a better world