1 the endeavour expedition: 21st century computing to the extreme randy h. katz, principal...
Post on 20-Dec-2015
220 views
TRANSCRIPT
1
The Endeavour Expedition:21st Century Computing to the eXtreme
Randy H. Katz, Principal InvestigatorEECS Department
University of California, BerkeleyBerkeley, CA 94720-1776
2
The Endeavour Expedition:21st Century Computing to the eXtreme
R. H. Katz, Principal Investigator, University of California, Berkeley
New Ideas• Systems Architecture for Vastly Diverse Computing Devices (MEMS, cameras, displays)• Wide-area “Oceanic” Data Information Utility• Sensor-Centric Data Management for Capture and Reuse (MEMS + networked storage)• Negotiation Architecture for Cooperating Components (Composable system architecture)• Tacit Knowledge Infrastructure to support High-Speed Decision-Making• Information Management for Intelligent Classroom Environments• Scalable Safe Component-based Design and UI Design Tools
Impact
• Enhancing human understanding by making it dramatically more convenient for people to interact with information, devices, and other people
• Supported by a “planetary-scale” Information Utility, stress tested by applications in decision making and learning, achieved thru new methodologies for design, construction, and administration of systems of unprecedented scale and complexity
Schedule
Jun 99Start
Jun 00 Jun 01 May 02End
Initial ArchitecturalDesign & Testbeds
Initial ApplicationImplementation &Evaluation
Information Utility
Information Applications
Design Methodologies
Initial Evaluation& 2nd Gen Redesign
Final Deployment& Evaluation
RefinedImplementation &Final Evaluation
Usability Studies &Early Tool Design
Implementation ofUI &Sys Design Tools
Tools Release &Final Evaluations
Initial ArchitecturalDesign Document
Initial Experiments &Revised Design Doc
Final Experiments &Architecture Docs
3
Expedition Goals
• Enhancing understanding– Dramatically more convenient for people to interact with
information, devices, and other people– Supported by a “planetary-scale” Information Utility
» Stress tested by challenging applications in decision making and learning
» New methodologies for design, construction, and administration of systems of unprecedented scale and complexity
– Figure of merit: how effectively we amplify and leverage human intellect
• A pervasive Information Utility, based on “fluid systems” to enable new approaches for problem solving & learning
4
Why “Endeavour”?
• Endeavour: to strive or reach; a serious determined effort (Webster’s 7th New Collegiate Dictionary); British spelling
• Captain Cook’s ship from his first voyage of exploration of the great unknown of his day: the southern Pacific Ocean (1768-1771)
– Brought more land and wealth to the British Empire than any military campaign
– Cook’s lasting contribution: comprehensive knowledge of the people, customs, and ideas that lay across the sea
– “He left nothing to his successors other than to marvel at the completeness of his work.”
5
Expedition Assumptions
• Human time and attention, not processing or storage, are the limiting factors
• Givens:– Vast diversity of computing devices (PDAs, cameras,
displays, sensors, actuators, mobile robots, vehicles); No such thing as an “average” device
– Unlimited storage: everything that can be captured, digitized, and stored, will be
– Every computing device is connected in proportion to its capacity
– Devices are predominately compatible rather than incompatible (plug-and-play enabled by on-the-fly translation/adaptation)
6
Expedition Challenges
• Personal Information Mgmt is the Killer App– Not corporate processing but management, analysis,
aggregation, dissemination, filtering for the individual
• People Create Knowledge, not Data– Not management/retrieval of explicitly entered
information, but automated extraction and organization of daily activities
• Information Technology as a Utility– Continuous service delivery, on a planetary-scale, on
top of a highly dynamic information base
• Beyond the Desktop– Community computing: infer relationships among
information, delegate control, establish authority
7
Driving Factors
• Technology Push– Accelerating developments at the eXtremes:
» Cluster-based compute/storage servers» MEMS sensor/actuators, CCD cameras, LCD displays,
…
• User Pull– More effective community leverage: the next power tool– Desire:
» Enhanced interaction, ease of use» Easier configuration, “plug and play”» Less fragile tools, “always there” utility functionality
8
Computing Revolution: Devices in the eXtreme
Evolution
Information Appliances:Scaled down desktops,e.g., CarPC, PdaPC, etc.
Evolved Desktops
Servers:Scaled-up Desktops,
Millennium
Revolution
Information Appliances:Many computers per person,
MEMs, CCDs, LCDs, connectivity
Servers: Integrated withcomms infrastructure;Lots of computing in
small footprint
Display
Keyboard Disk
Mem
Proc
PC Evolution
Display Display
Camera
Sm
art
Senso
rs
Camera
Smart Spaces
ComputingRevolution
WAN
Server, Mem, Disk
InformationUtility
BANG!
Display
Mem
Disk
Proc
9
Expedition Approach• Information Devices
– Beyond desktop computers to MEMS-sensors/actuators with capture/display to yield enhanced activity spaces
• InformationUtility
• InformationApplications
– High Speed/Collaborative Decision Making and Learning
– Augmented “Smart” Spaces: Rooms and Vehicles
• Design Methodology– User-centric Design with
HW/SW Co-design;– Formal methods for safe and trustworthy
decomposable and reusable components
“Fluid”, Network-Centric System Software
– Partitioning and management of state between soft and persistent state
– Data processing placement and movement
– Component discovery and negotiation
– Flexible capture, self-organization, and re-use of information
10
Interdisciplinary, Technology-Centered
Expedition Team• Alex Aiken, PL• Eric Brewer, OS• John Canny, AI• David Culler, OS/Arch• Joseph Hellerstein, DB• Michael Jordan, Learning• Anthony Joseph, OS• Randy Katz, Nets• John Kubiatowicz, Arch• James Landay, UI
• Jitendra Malik, Vision• George Necula, PL• Christos Papadimitriou, Theory• David Patterson, Arch• Kris Pister, Mems• Larry Rowe, MM• Alberto Sangiovanni-
Vincentelli, CAD• Doug Tygar, Security• Robert Wilensky, DL/AI
11
Organization: The
Expedition Cube
Information Devices
Information Utility
ApplicationsDesIgn
Methodology
MEMS Sensors/Actuators, Smart Dust, Radio Tags, Cameras, Displays, Communicators, PDAs
Fluid Software, Cooperating Components,Diverse Device Support, Sensor-CentricData Mgmt, Always Available, TacitInformation Exploitation (event modeling)
Rapid Decision Making, Learning,Smart Spaces: Collaboration Rooms,Classrooms, Vehicles
Base ProgramOption 1: Sys Arch for Diverse DevicesOption 2: Oceanic Data Utility
Option 4: Negotiation Arch for CooperationOption 5: Tacit Knowledge InfrastructureOption 6: Classroom TestbedOption 7: Scalable Safe Component-Based Design
Option 3: Capture and Re-Use
12
Base Program: Leader Katz
• Broad but necessarily shallow investigation into all technologies/applications of interest
– Primary focus on Information Utility» No new HW design: commercially available information
devices» Only small-scale testbed in Soda Hall
– Fundamental enabling technologies for Fluid Software» Partitioning and management of state between soft and
persistent state» Data and processing placement and movement» Component discovery and negotiation» Flexible capture, self-organization, info re-use
– Limited Applications– Methodology: Formal Methods & User-Centered Design
13
System Architecture for Vastly Diverse Devices
Leader Culler
• Design Issues for “Small Device OS”– Current: managing address spaces,thread scheduling,
IP stack, windowing system, device drivers, file system, APIs, power management
– How can OSs for tiny devices be made radically simpler, manageable, and automatically composable?
• Devices of Interest: Dust Motes
14
Communication-Centric Architecture
• Active Proxies– connected to the
infrastructure– soft-state, bootstrap
protocol– transcoding,
• Ubiquitous Devices– billions– sensors / actuators– PDAs / smartphones / PCs– heterogeneous
Service Path
• Base Scalable Infrastructure– highly available– persistent state (safe)– databases, agents– service programming environment
• Service Paths– aggregate flows (rivers)– transcoding operators
15
Servers
ClientsClients
ClientsClientsClients
ClientsServers
Servers
Infrastructure Services
Open
“The Large”: Service-Centric Platform Arch
• Enable distributed creation/deployment of scalable, available services
– Service registry, aggregate execution env., transparency – Persistent distributed data structures– Massive fluid storage (“Oceanic” Storage)– Adaptive high-bandwidth flows (rivers)
• Build infrastructure via composition of services
16
“The Small”: Radically Simple OS for Management &
Composition• Basic Assumptions:
– Communication is fundamental– Direct “user interface” is the exception not the norm– Critical resource is scheduling data movements, not arbitrary
threads of computation
• Tiny OS: Little more than an FSM– Commands: event stream merged with sensor/actuator events– General thread compiled to sequence of bounded atomic
xacts– Constant self-checking and telemetry– Rely on the infrastructure for complex processing
• Correctness-by-construction techniques for cooperating FSMs (tie in to HW/SW co-design)
17
Implementation & Deployment of Oceanic Data Info Utility
Leader Kubiatowicz • Ubiquitous devices require ubiquitous storage
– Consumers of data move, change access devices, work in many different physical places, etc.
• Needed properties:– Strong Security: data must be encrypted whenever it is in
the infrastructure– Coherence: too much data for naïve users to keep
coherent “by hand”– Automatic replica management and optimization: huge
quantities of data cannot be managed manually – Simple and automatic recovery from disasters: probability
of failure increases with size of system– Utility model: world-scale system requires cooperation
across administrative boundaries
18
Pac Bell
Sprint
IBMAT&T
CanadianOceanStore
IBM
Utility-Based Infrastructure
• Confederations of (Mutually Suspicious) Utilities– Settlement system among service providers– Buy and sell capacity as needed
19
OceanStore Architecture/Technology
• Name and Data Location– Issue: Find nearby data without global communication– Approach: Data location is aform of gradient-search of local pools of data (use
of attenuated Bloom-filters)
• High Availability and Disaster Recovery– Issue: Eliminate backup as independent/fallible technology– Approach: Erasure-codes/mobile replicas provide stable storage for archival
copies and snapshots of live data
• Introspective Monitoring and Optimization– Issue: Optimize performance on a global scale– Approach: Monitoring and analysis of access/usage relationships
• Rapid Update in Untrusted Infrastructure– Issue: Updates should not reveal info to untrusted servers– Approach: Incremental cryptographic techniques/oblivious function
techniques to perform update
20
Applications that Enhance Human Activity
• Tacit Information Mining: exploit info flows & relationships to improve collaborative work
– 3D “activity spaces” for representing decision-making activities, people, & information sources
– Visual cues to denote strength of ties between agents, awareness levels, activity tracking, & attention span
• Smart Spaces– Electronic collaborative problem-based learning– Physical and Virtual Learning Spaces– Enabled by information appliances– UI design/exploitation of tacit information
21
Experimental Testbeds
NetworkInfrastructure
GSMBTS
Millennium Cluster
Millennium Cluster
WLAN /Bluetooth
Pager
IBMWorkPad
CF788
MC-16
MotorolaPagewriter 2000
Velo
TCI @HomeAdaptive Broadband LMDS
H.323GW
Nino
Smart ClassroomsAudio/Video Capture Rooms
Pervasive Computing LabCoLab
Soda Hall
CalRen/Internet2/NGI
Smart
DustLCD Displays
WearableDisplays
22
Summary: Putting It All Together
1. eXtreme Devices
2. Data Utility
3. Capture/Reuse
4. Negotiation
5. Tacit Knowledge
6. Classroom
7. Design Methods
8. Scale-up
Devices
Utility
Applications
Fluid Software
Info Extract/Re-use
Decision MakingGroup Learning
Component Discovery& Negotiation
Self-Organization
23
Base Program ScheduleYear 1 Year 2 Year 3
Eval. & Initial Design ToolsSmart Space Testbed
1st Gen Fluid R/T Environ.
1st Gen Comp Neg. Protocols
1st Gen Persistent Fluid Store
1st Gen Sensor-Centric Info Mgmt
Design Document+ Early Evaluation
Cooperative Learning App
Rapid Decision Making App
Refined Doc+ Experiments
Refine & Use
Refine & Use
Perf Eval
Perf Eval
2nd Gen Persistent Fluid Store
2nd Gen Sensor-Centric IM
2nd Gen Fluid R/T Environ.
2nd Gen Negotiation
Final Doc+ Experiments
Refined Tools & Flow
DesignMethodology
InformationUtility
InformationApplications
24
Year 1 Milestones Design/initial deployment “smart
space” testbed; Initial usability evaluation/refinement;
Initial design, prototype, and early evaluation of fluid software run-time environ;
Initial design component advertisement protocols & i/f negotiation spec language;
Initial prototype/refinement of component advertisement protocols & interface negotiation specification language;
Initiate prototype & refinement of distributed, persistent storage system;
Initial design of sensor-centric/stream-capture oriented data mgmt system;
Initiate prototype & refinement of sensor-centric data mgmt system;
Design of distributed, persistent storage system;
Initial design of tool flow for infrastructure-embedded software functionality;
Initiate implementation of system design tools for early testing;
Completion of initial system architecture design document and early system evaluation;
25
Coherently managing billions of devices where none are “average”
Information on demand, available wherever needed, on a global scale, in an untrusted infrastructure
Pervasive management of massive stream-oriented information collection/inference in the wide-area
Data movement & transformation; Paths, not threads; Persistent state/soft state partitioning; Non-blocking RMI for remote functionality; Support for MEMS devices, cameras, displays, etc.
Serverless/homeless/freely flowing data; Opportunistic distribution, promiscuous caching, without administrative boundaries; High availability/disaster recovery, application-specific data consistency, security;Overlapping, partially consistent indices; Data freedom of movement; Expanding “search parties” to find data, using application-specific hints
Extract, manage, analyze streams of sensor data; Path-based processing integrated with storage; Data reduction via filtering/aggregation; Distributed collection & processing; “Evidence accumulation” from inherently noisy sensors
Problem Technical Approaches
26
Problem Technical ApproachesOverwhelming config-uration complexity of large & heterogeneous systems
Ineffectiveness of technology-mediated collaborative work;Better support for rapid decision making;
Enabling Problem-based Learning in Enhanced Physical & Virtual Spaces;
Correctness by Construction: Safe Component Design;
Dynamic self-configuration: advertise provided services, discover components providing required services, negotiate interface contracts, monitor compliance, eliminate non-performing confederates;
Infer communications flow, indirect relationships, availability, participation to enhance awareness & support opportunistic decision making; New collaborative applications: 3D “activity spaces” for representing decision-making activities, people, & info sources; Visual cues “weighting” relationships among agents, awareness levels, activity tracking & attention span
Device/net-independent people-to-people comms via pervasive translation/adaptation; Information dissemination technologies; Wide-area information mgmt/access;
Formal specifications and methods; Safety enforcement, design/development methods; Proof carrying code/secure protocol verification;
27
Summary and Conclusions
• 21st Century Computing– Making people’s exploitation of information more effective– Encompassing eXtreme diversity, distribution, and scale– Computing you can depend on
• Key Support Technologies– “Fluid software” computational paradigms– System and UI support for eXtreme devices– Pervasive, planetary-scale system utility functionality– Active, adaptive, safe and trusted components – New “power tool” applications that leverage community
activity
28
Industrial Collaborators
SRI