1 content-aware device benchmarking methodology (draft-hamilton-bmwg-ca-bench-meth-04) bmwg meeting...
TRANSCRIPT
1
Content-Aware Device Benchmarking Methodology
(draft-hamilton-bmwg-ca-bench-meth-04)
BMWG MeetingMaastrichtJuly 2010
Mike [email protected]
BreakingPoint Systems
2
Agenda
Why draft-hamilton? Charter objections/responses Goals reset Explicit goals of this draft Explicit non-goals of this draft
3
Why draft-hamilton?
RFC 2544 doesn’t specifically apply to some modern devices
Test vendors are already doing this in a one-off fashion BreakingPoint, Spirent, Ixia, Agilent, etc.
4
Charter Objections
• “the scope of the BMWG is limited to technology characterization using simulated stimuli in a laboratory environment.”
• “Said differently, the BMWG does not attempt to produce benchmarks for live, operational networks
• This does not restrict BMWG from creating benchmark tests that are representative of VERY SPECIFIC live, operational networks
5
Goals Reset
• Create a series of benchmark tests to MOST accurately predict device performance under realistic conditions FOR A SPECIFIC SIMULATED NETWORK
• RFC 2544 Quotes Page 11, Section 18, “Multiple Frame Sizes”• “The distribution MAY approximate the conditions on
the network in which the DUT would be used.”
• “The authors do not have any idea how the results of such a test would be interpreted other than to directly compare multiple DUTs in some very specific simulated network”
6
Explicit Goals
• Repeatable Results
• Compare Multiple DUTs
7
Explicit Non-Goals
• Not a replacement of RFC 2544
• Total Input Repeatability (discussion to follow)
8
Test Run Setup
• Methodologies Run• RFC 2544 Throughput (64B + 1518B)• RFC 3511 Throughput (1 kB + 512 kB)• IMIX Throughput• CAIDA• Spirent• Wikipedia• Agilent-simple• draft-hamilton-03 (random)• draft-hamilton-04 (shell)
9
Test Results
10
Fuzzing Results
11
Draft-04 Highlights and Reasons
• “Shell” Methodology• More reproducible
• Backoff on ‘realistic’• Compromise
• Dropped ‘security’• Difficult to scope and maintain currency
• Maintain ‘fuzzing’ aspect• Random but repeatable