network computing laboratory experiment tutorial network computing lab. 2005. 5. 19
TRANSCRIPT
Network Computing Laboratory | 2
Korea Advanced Institute of Science and Technology
Table of contents
What is experiment? What is experiment design? Case study - CIGMA Three keys for experiment Step by step for experiment
Network Computing Laboratory | 3
Korea Advanced Institute of Science and Technology
What is experiment?
Proof of your proposal Convince people that your goal is well achieved Make your contribution to be clear and true Support that your technical problems are well solved
Network Computing Laboratory | 4
Korea Advanced Institute of Science and Technology
What is experiment design?
Experiment design 실험 계획표 작성 (Pre-report)
실험 목적1. Decide key aspects you want to show2. Choose the proper verification method
Demo, implementation and measurement, simulation, and analysis3. Identify metrics (dependent variables)4. Classify parameters (independent variables)
실험 환경 설정1. Decide key system components to be implemented
Plan to test your implemented component2. Prepare workload (benchmark, trace, synthetic workload)3. Plan to prepare test-bed (machine, OS, software, network link, simulation tools)
실험 예상1. Draw the expected graphs which you want to show (imaginary experiment w/ paper and
pencil)
Assignment Presentation slides of experiment design for your own experiment
Network Computing Laboratory | 5
Korea Advanced Institute of Science and Technology
Verification methods
1. Analysis 예 ) Quick sort = O(N log N)) Deterministic and Statistical Analysis
2. Simulation 예 ) Network protocol simulation (topology, bandwidth, link delay) 실험환경과 시스템을 모두 모델링 해서 검증하는 방법 Simulation 은 언제 하는가 ??
변수가 많고 시나리오가 복잡해서 분석이 불가능 할 때 구현 및 측정하기가 어렵거나 불가능 할 때 구현 및 측정하기 앞서서 미리 메커니즘 등을 검증하고 싶을 때
장점 : Easy to control, 단점 : Unrealistic
3. Implementation and measurement 예 ) 웹 서버 구현 후 , 초당 처리 수 측정
Network Computing Laboratory | 6
Korea Advanced Institute of Science and Technology
Case study – CIGMA: Motivation
CIGMA: aCtive Inventory service in Global e-MArket based on efficient catalog management
trave
l infor
mation
Internet
Travel ConsolidatorService
client A
cruise sites c1,c2,...
car-rental sitesr1,r2,...
travel information
hotel sitesh1,h2,...
web sitesd1,d2,..
vacation sitesv1,v2,...
airline sitesa1,a2,...
H
C
D
V
R
client
Highly dynamic e-commerce environment Constantly changing sales
item information Flooding information: a lot of
shopping sites “I want to know how I can
purchase goods with the best condition!!”
Network Computing Laboratory | 7
Korea Advanced Institute of Science and Technology
Case study – CIGMA: Goal
CIGMA: aCtive Inventory service in Global e-MArket based on efficient catalog management
trave
l infor
mation
Internet
Travel ConsolidatorService
client A
cruise sites c1,c2,...
car-rental sitesr1,r2,...
travel information
hotel sitesh1,h2,...
web sitesd1,d2,..
vacation sitesv1,v2,...
airline sitesa1,a2,...
H
C
D
V
R
client
CIGMA service and systemProvide aggregated catalog information from a lot of sitesAssure up-to-date informationServe large number of customers with fast response
Network Computing Laboratory | 8
Korea Advanced Institute of Science and Technology
Case study – CIGMA: Design and Implementation
Communication Module
update event+ modified data
ContentConverter
EventListener
invalidation msg+ modified data
Java Virtual Machine
ContentConversion
Table
updaterequest
(1)
(3) (4) (5)
(6)
ContentProvider'sDatabase
Triggers
EventReporter
Catalogtable (2) modified
data
CPMContent ProviderHandler
Content Provider Interface
invalidation msg+ modified data
(8)
(9)
Logger&
Monitor
Logging messageto SDBM
CacheUpdaterupdate
request
(7)
MCM
CPW
WSSLogic
Web ServerWeb Application Server
customer requests
contentretrieval
swappingmanager
Disk Storage
Memory Cache(Main Memory)
Network Computing Laboratory | 9
Korea Advanced Institute of Science and Technology
Case study – CIGMA: Experiment
Questions!!
What kinds of experiments do you want to do?
Why do you want to do those experiments?
Network Computing Laboratory | 10
Korea Advanced Institute of Science and Technology
Case study – CIGMA: Experiment
What and Why - “Remind your goal and contribution” CIGMA should deal with a lot of shopping sites. So it should process
a large number of update requests from multiple sites Catalog update throughput
CIGMA should assure up-to-date information. So update delay should be small Catalog update delay from shopping site to CIGMA
CIGMA should handle a lot of customers requests. So it should process a number of customer requests at the same time Catalog browse throughput
CIGMA receives update requests and customer requests concurrently. So it should process both of requests Mixed throughput
Network Computing Laboratory | 11
Korea Advanced Institute of Science and Technology
Case study – CIGMA: Experiment
One more Question!!
How do you do those experiments?
Network Computing Laboratory | 12
Korea Advanced Institute of Science and Technology
Case study – CIGMA: Experiment
What to do after you decide metrics such as Catalog update throughput Catalog update delay Catalog browse throughput Mixed throughput
What do you need in order to measure those metrics?? Do you have machines and software needed for implementation? Do you have your implementation which is working well? Do you have something to generate the workload?
What kinds of parameters affect the experimental results?? What is your expectation about your experimental results??
Network Computing Laboratory | 13
Korea Advanced Institute of Science and Technology
Case study – CIGMA: Experiment
What do you need in order to measure those metrics??
Do you have machines and software needed for implementation?
Prepare your machines and software.
Pentium III 1GHz, 2GB memory machine, 100M LAN Link, Linux OS, Tomcat server, FastDB Main Memory DB …
Do you have your implementation which is working well?
Implement your system’s core functionality to be verified and test your system.
Network Computing Laboratory | 14
Korea Advanced Institute of Science and Technology
Case study – CIGMA: Experiment
What do you need in order to measure those metrics??
Do you have something to generate the workload?
Use existing benchmark tools. If not available, implement your own workload generator.
Browse throughput: Httperf benchmark used (c.f. TPC-W, SPEC-Web)
Update throughput: Update request generator implemented Update delay: WAN emulation tool used
Network Computing Laboratory | 15
Korea Advanced Institute of Science and Technology
Case study – CIGMA: Experiment
What kinds of parameters affect the experimental results?? Find possible parameters Plan to choose and vary just one parameter at a time
What is your expectation about your experimental results?? Draw your expected graphs which show not only a
pattern of result (linearly increasing, logarithmically increasing, decreasing, or keeping constant)but also specific numbers
Help you to analyze your measured results Calculate your expected results mathematically
Network Computing Laboratory | 16
Korea Advanced Institute of Science and Technology
Case study – CIGMA: Experiment
And then??
Experiment!! Experiment!! Experiment!! Analyze your results Verify your expectations Modify your parameters, implementations, even design
Network Computing Laboratory | 17
Korea Advanced Institute of Science and Technology
Case study – CIGMA: Experiment
Example result graphs
Network Computing Laboratory | 18
Korea Advanced Institute of Science and Technology
Three keys for experiment
What Why How
Network Computing Laboratory | 19
Korea Advanced Institute of Science and Technology
Three keys for experiment
What What do you want to show to people?
Strongly connected to what you propose as your system goals and your contributions
Quantitative
Q. What kinds of metrics can you use? E.g. Throughput, latency, execution time, storage usage, # of Disk I/O,
# of lookups, % of failed lookups, BW consumption, etc. vs. Qualitative
Q. What are core functionalities to show? E.g. Demo, screenshots
Network Computing Laboratory | 20
Korea Advanced Institute of Science and Technology
Three keys for experiment
Why Why do you want to show it to people?
Also strongly connected to your system goals and your contributions
Q. Does it really prove your contribution?
Q. Does it show your achievement more effectively than others?
Network Computing Laboratory | 21
Korea Advanced Institute of Science and Technology
Three keys for experiment
How How can you show it to people?
Experiments using real implementation and benchmark tool Simulations Demos
Network Computing Laboratory | 22
Korea Advanced Institute of Science and Technology
Step by step for experiment
Determining What and Why Setting experimental environment
Implementation Benchmark tools
While (hypothesis verified to be true || your approach turn to be wrong || give-up) {
Making hypothesis Expect results and draw your expected graphs
Verifying hypothesis Do experiment and get results Analyze results
Giving feedback from your analysis To experimental parameters, system tuning To your implementation To your system design
} Explaining your results and lessons if you succeed