pdcs '07 1 towards an architecture for extreme p2p applications nadia shalaby john zinky 19 th...
TRANSCRIPT
PDCS '071
Towards an Architecture forExtreme P2P Applications
Nadia Shalaby John Zinky
19th Conference on Parallel and Distributed Computing Systems Cambridge, MANovember 21, 2007
PDCS '072
What is the Problem?• Scope of P2P paradigm expanded
beyond research arena
• Ubiquitous in commercial, industrial and military applications
• Optimized P2P solutions available for single resource dominant applications: e.g. file distribution, grid computing, Pub/Sub message passing
• Large scale, reliable, complex, real-time applications need custom development
• High-risk, time consuming, expensive
• How do we fill this gap? 1. Formulate properties of these apps
2. Propose general architecture
3. Propose implementation platform
4. Ensure backward compatibility
P2Papplications
P2P overlaysubstrates
serviceabstractions
resourceabstractions
cyberresources
App1App1 App3
App3App2App2
P2P overlayAPI
client/serverAPI
resource abstractionAPI
resourceAPI
…
wire/fiber/radio
wire/fiber/radio
networkstack
Message services
P2PO1
substrateP2PO1
substrate
socket
RPC
AppnAppn
Disk
filesystem
Database services
P2POk
substrateP2POk
substrate
files
DB language:
SQL
…
CPU
operatingsystem
Grid services
P2PO2
substrateP2PO2
substrate
process
job queues HTTP
PDCS '073
Outline
1. What is the Problem?
2. Taxonomy
3. Examples
4. Architecture
5. Cougaar’s Two-Tier Architecture
6. Architectural Mapping
7. Integration with Legacy Systems
8. Case Studies of Extreme Applications
9. Conclusion
PDCS '074
Taxonomy
Application Functional Requirements
(addressed at programming/development phase) 1. Communication:
client/server vs. P2P2. Development cycle:
waterfall vs. adaptive3. Process during operational lifespan:
fixed vs. evolving• Human participation level:
none vs. sensor vs. model vs. cognitive1. Cross-cyber resource usage
CPU vs. network vs. storage vs. all
System Resource Constraints
(exhibited during runtime) 1. Distributedness of cyber resources:
centralized vs. LAN vs. WAN2. Data plane speed:
batch vs. online vs. superhuman realtime• Survivability (reliability & performance):
non-crucial vs. exigent1. Security adversary level:
trust all vs. compartmentalized trust vs. malicious vs. insider threat
2. Scalability (nodes): 10s vs. 100s vs. 1000s vs. >10,000
Business Environment(real-world constraints)
1. Market share: large vs. medium vs. small
2. Integration environment:standalone vs. stovepipe vs.
new functionality w/ legacy system integration
Business System
Application
PDCS '075
Examples of Extreme Applications
Application Functional Requirements
(addressed at programming/development phase) 1. Communication:
client/server vs. P2P2. Development cycle:
waterfall vs. adaptive3. Process during operational lifespan:
fixed vs. evolving• Human participation level:
none vs. sensor vs. model vs. cognitive1. Cross-cyber resource usage
CPU vs. network vs. storage vs. all
System Resource Constraints
(exhibited during runtime) 1. Distributedness of cyber resources:
centralized vs. LAN vs. WAN2. Data plane speed:
batch vs. online vs. superhuman realtime• Survivability (reliability & performance):
non-crucial vs. exigent1. Security adversary level:
trust all vs. compartmentalized trust vs. malicious vs. insider threat
2. Scalability (nodes): 10s vs. 100s vs. 1000s vs. >10,000
Business Environment(real-world constraints)
1. Market share: large vs. medium vs. small
2. Integration environment:standalone vs. stovepipe vs.
new functionality w/ legacy system integration
Examples of Extreme Applications• Information Assurance • UAV surveillance on mobile sensor platforms• Field modifiable systems preventing evolving
malicious attacks• Proactive content distribution• Management of evolving data proc. Centers• ISP network management and optimization• Voice, video over Disruption Tolerant Networks
PDCS '076
Degrees of Extremeness
Extreme
S3Message
basedMessage
based
P2P
GridbasedGrid
based
DBbased
DBbased
Distributed
PDCS '077
Outline
1. What is the Problem?
2. Taxonomy
3. Examples
4. Architecture
5. Cougaar’s Two-Tier Architecture
6. Architectural Mapping
7. Integration with Legacy Systems
8. Case Studies of Extreme Applications
9. Conclusion
PDCS '078
Stack
Protocol
Stack
ProtocolData
Plane
Executor
Architecture of Extreme Applications
Sensor-BasedControl Loop
Model-BasedControl Loop
CognitiveControl Loop
state machine policysituation
inference rules
days to hours secs to msecsCPU
hours to secswire/fiber/
radiowire/fiber/
radioDiskhours to minutes
control plane data plane
SensorAgentsSensorAgents
Real-timeOptimizer
Agents
Real-timeOptimizer
Agents
app. statuscoordination
resource statuscoordination
resource trendscoordination
CognitiveLearnerAgents
CognitiveLearnerAgents
app. trendscoordination
SituationPredictorAgents
SituationPredictorAgents
app. patterncoordination
resource patterncoordination
SensorAgentsSensorAgents
AppAppAppAppAppApp
PDCS '079
Cougaar’s Two-Tier Architecture
• Local• Data driven• Distributed BB• Behavior (plugin)• State (BB)• Pub/Sub BB API
Environment • Distributed• Event driven• Components• Services• Service oriented API• Imported libraries
Agent/Env. API• sensor• activator• coordinator• active API• at every node
Agents
componentcomponent
service
BBBB
BehaviorBehavior
activatoractivator coordinator
coordinatorsensorsensor
componentcomponent
service service
componentcomponent
librarylibrary
Agent
BBBB
BehaviorBehavior
activatoractivator
coordinator
coordinator sensorsensor
componentcomponent componentcomponent
serviceservice
librarylibrary
Agent
Environment
domainspecific
systemspecific
PDCS '0710
Outline
1. What is the Problem?
2. Taxonomy
3. Examples
4. Architecture
5. Cougaar’s Two-Tier Architecture
6. Architectural Mapping
7. Integration with Legacy Systems
8. Case Studies of Extreme Applications
9. Conclusion
PDCS '0711
Architectural Mapping
Cougaar1. Import from native infrastructure2. Computational model:
• Agents • Behaviors• Agent coordinations
3. Cougaar Environment (systemic QoS)4. Agent societies5. Transitioning of control
loops human to automation
architectural mapping
Extreme App. Architecture 1. Data plane2. Computational model:
• Functional modules (ovals)• Functional heterogeneity• Inter-module coordination
3. Control plane of the application4. Hierarchy of functionality5. Sensor-, model-, cognitive-based
Mapping Dimensions1. Separation of data and control planes2. Computational model3. Decoupling of system and application4. Hierarchical control plan5. Evolving degree of operational human
involvement
PDCS '0712
Integration with Legacy Systems
… App
licat
ion n
embe
dded
dev
ices
sche
dulin
g
scie
ntifi
c co
mp.
web
Ser
vice
s
…App
licat
ion k
App
licat
ion n
ftp, t
elne
t, ss
h
embe
dded
con
trol
… …
CPU disk
wire/fiber/radio
wire/fiber/radioCPU disk
wire/fiber/radio
wire/fiber/radioCPU disk
wire/fiber/radio
wire/fiber/radio
C o u g a a rruntime
MPI Library
JESSJESS
processes/threadsprocesses/threads
Corba/RMI web services
enterpriseservice busenterprise
service bus
TCP/UDP network stack
TCP/UDP network stack
C o
u g
a a
r
run
tim
e
SQL DB services
OWL knowledge
base
OWL knowledge
base
filesfiles
C o u g a a rruntime
sem
antic
tagg
ing
bank
ing/
airli
nes
wor
d pr
oces
sing
App
licat
ion n
grid-based systems message-based systems DB-based systems
Main Cougaar architectural feature:imported libraries and component wrappers
PDCS '0713
Conclusion
1. Taxonomy for P2P Application space– Identify classes with progressively stringent
constraints
2. Novel architecture for Extreme Applications– Leverages the extreme development process,
gradual phasing out of human involvement during system operation
3. Two-Tier Cougaar architecture– Serves as a middleware platform, natural fit for
Extreme applications, backward compatible for legacy components