1

eduPERTA participants viewpoint, including some history and the current/future status

Networkshop 44 #NWS44

Dr Alan Buxey

Loughborough University

Introduction…. a common aim.

• We will identify existing and emerging communities that run or are looking to run more demanding network applications

• Our service will include specialist assistance to communities, identifying requirements, helping to troubleshoot problems, and recommending technical solutions. We will also seek to deploy appropriate tools to assist users in visualising their use of the network.

• With the involvement of all stakeholders – user communities, computer service departments, network researchers and ourselves – we aim to raise the ‘high water mark’ for network applications

• We also aim to facilitate well-attended, productive workshops and vibrant mail list discussions, and foster strategic community oversight of the project through a dedicated, expert special interest group (SIG).

2

Janet end-to-end performance initiative

eduPERT• PERT –The Performance Enhancement and Response Team • Virtual organization • Made of networking experts (often in NOCs/central IT) • Help end-users diagnose and solve network performance issues • Enabler of the exchange of knowledge between experts • Training the experts.

3

eduPERT

4

2010 – eduPERT event at TNC2010 (Lithuania) GN3 (S.O.T.N./updates)(https://tnc2010.terena.org/schedule/meetings/5/)2012 – eduPERT event at TNC2012 (Iceland) GN3 (PerfSONAR)(https://tnc2012.terena.org/core/event/10)2012 – eduPERT stream at GEANT symposium, Vienna

eduPERT

5

Get the required (if not maximum) performance out of the infrastructure

eduPERT

6

• Multiple layers of systems with complex interactions • Applications, OS, disks/memory/network adapters, networks including

middleboxes such as firewalls/rate shapers etc.

• Multiple network domainsCampus LAN, regional/national/international backbones

• Defensive support organisations “finger-pointing loops” (my part of the network works fine, so it must be X!)

Why is e2e performance/diagnosis ‘hard’?

eduPERT

7

e2e Performance approaches• System auto-tuning• Performance monitoring (e.g. PerfSONAR, DTrace) • User self-diagnosis (e.g. NDT)• User education (e.g. PERT Knowledge Base)• Performance troubleshooting assistance (PERT)

Supported by central services/elements: •Common policy •Registry and accreditation (edupert.geant.net)•~Yearly PERT Training workshops•Meetings for information exchange•PERT Knowledge Base—kb.pert.geant.net

Must also mention https://fasterdata.es.net/ !Page 8, GEANT Connect Jan 2013

eduPERT

8

PERT teams

Arnes (.si)NIIF Institute (.hu) CARNet (.hr)CSC-Scientific Computing Ltd (.fi)SWITCH (.ch)Fondation RESTENA (.lu)Poznan Supercomputing and Networking Center (.pl)FCCN (.pt)DFN-Verein e.V. (.de),e-Arena (.ru)SANET (.sk)AMRES (.rs)RENA (.ge)BASNET (.by)GARR (.it)HEAnet (.ie)ACOnet (.at)RENATER (.fr)

NB SWITCH PERT - accredited PERT

eduPERT

9

Performance U!

eduPERT Advanced Training@TNC 15 18th of June, 2015, Porto, Portugal

• hands-on training for performance and analysis tools using perfSONAR and UDPmon

• overview of bwctl (iperf, iperf3, and nuttcp), owamp, bwtraceroute and bwping, then focus how to use these results for advanced troubleshooting and analysis

eduPERT

10

Outline• eduPERT work flow today • eduPERT facts today, the concept• SA3T3, eduPERT in context of a PM&V• eduPERT Discussion • Q&A

11

eduPERT/PERT – Workflow toady

12

eduPERT today

13

eduPERT is a world wide Open Community:Virtual organization coordinating performance issues over GEANT. eduPERT Service: http://swit.ch/eduPERTserviceDiscussion platform: pert-discuss@geant.org

eduPERT is moving an extended field of action:Towards the end-users, researchers and academic ICTs Umbrella for NREN’s PERT teamsLocal NRENs: http://swit.ch/localPERTs

eduPERT is the Competence Center on Performance Measurement/Management over GEANTPerformance U: http://swit.ch/eduPERTtrainingReports: http://swit.ch/eduPERTreportsKnowledge DB: http://swit.ch/eduPERTkb

GN4-1-SA3 T3, eduPERT in Context of a PM&V Framework

14

eduPERT – Coordinator Performance Issues / Knowledge pool

Wireless Crowd Sourced Performance Measurement and Verification

•End-User feedback•eduroam locations•SW Development (JavaScript deployment) collecting monitoring data on the campus from relevant WebSources.•Implementation /Demonstrations at DCU campus / TNC2015 / APAN40 / NORDUnet tech workshop etc. •Service Deployment

WiredMon in Context of a PM&V FundamentalsWhat we have / Capability / Future Steps

•New Use Cases•Using existing tools; perfSONAR (pS), CMon, SQM•Elaboration PM&V solution on chosen Use Cases•Evaluation of solutions •Collaboration Solutions of existing tools •Recommendation based on Fundamentals. •Service Deployment

Impact New Technologies/Expertise/Standards End-User feedback vs. Infrastructure information in a PM&V

eduPERT trainings/monthly calls/kb entries and performance Issues on virtualized network environments Progress on measurement tool deployments and/or new ideas

GN4-1-SA3T3, eduPERT – Discussion

15

eduPERT IST eduPERT future ?

Community eduPERT into GEANT -Coordinator PERT issues -Open Community-Circle of Experts -Work flow

eduPERT as a TF, part of GEANT -(N)RENs + -eduPERT Charter-Work items/packages-Volunteer base

Innovation eduPERT – Coordinator/knowledge/skills centre (project driven)-Mostly NRENs NOCs (expertise)-eduPERT trainings / monthly calls -KB-DB, Reports and knowledge Sources Best practice-Discussion Forum

eduPERT(-TF), subject driven-RENs, Universities +-R&D projects in PM&V-New tools -Forum sharing expertise-Service oriented (if possible but not a must)

Organization eduPERT into the GN4-1 as a subtask-Strongly Advisory character-Primarily Problem solving

eduPERT an Initiative associated with(out) GEANT, spin-off character-Independent Organization-R&D

16

Case studies - cisco ASA ring buffer issue (ASA-5585-X-SSP40)

http://services.geant.net/edupert/Resources/Documents/Firewall_Performance_TIP2013.pdf

NOC/PERT case SWITCH had with a customer (large university) March & October 2012

Firewall caused dropped BGP sessions because of relatively small DOS-attack on host behind

1.3M pps at 1Gbit/s total – enough to cause issue (tested by generating 100byte UDP flows)

eduPERT

17

Case study - SWITCH DNS Server

The most important performance indexes for a DNS Server are: a) Number of Queries per second that the server is able to answer without “dropping” (qps) b) The time needed to answer a request

Virtual results were 1/3 of that of a physical box. Bad.

Use of Virtio-net paravirtual drivers and the linux vhost-net driver – excellent results

eduPERT

18

A small amount of packet loss makes a HUGE difference in TCP performance

eduPERT

19

eduPERT

20

Better/Best practice models - Science DMZ

A light-touch Science DMZ Deployment

Add-on to existing network infrastructure• All that is required is a port on the border router • Small footprint, pre-production commitment

Easy to experiment with components and technologies • DTN prototyping• perfSONAR testing

Limited scope makes security policy exceptions easy • Only allow traffic from partners• Add-on to production infrastructure – lower risk

21

22

Why you should have a PERT

(managers)

•Potential to create positive “Moments of Truth” for your constituency •Differentiates your service from commodity offerings•Interesting work helps retain senior network engineers•Synergies across your organisation (network/systems/applications)

(techies/engineers)

•You'll help your network's users do exciting things•You'll learn new skills that are widely applicable•It's rewarding (if you don't mind occasional direct contact with users ;-) )•It's technically challenging and generally fun!

eduPERTThanks for listening!

With special thanks to (in no particular order!)

Jason Zurawski (ES.NET)

Bartek Gajda (POZNAN, PIONIER (.pl))

Kurt Baumann (SWITCH)

Simon Leinen (SWITCH)Alessandra Scicchitano (SWITCH)

Chris Welti, Alexander Gall (SWITCH)

23

Questions?

24