customer satisfaction 201 howard c. berkowitz [email protected] [email protected]...

Customer Satisfaction 201Customer Satisfaction 201

Howard C. BerkowitzHoward C. [email protected] [email protected]

[email protected]@clark.net

(703)998-5819 ESN 451-5819(703)998-5819 ESN 451-5819

NANOG 21 Customer Satisfaction Tutorial 2/17/20012

What is the Problem to be Solved?What is the Problem to be Solved?

• What are you?

• What characteristic applications do you support?

• To the customer, what are the perceived needs?

• How do you manage Service Expectations?

• How do you care for customers?


What are you?What are you?

• ISP

• Bandwidth provider

• IPSP

• BIPSP

Do you want to be seen as:

lowest cost

best service

something inbetween?


A QuestionA Question

• Why do vendors always offer solutions?

• Always remember you offer solvents.

• Solutions exist only when the customer problem dissolves in the solvent you provide.


Characteristic ApplicationsCharacteristic Applications

• Internet access by internal staff

• B2C: customer access to public servers

• Intranet— Usual entry point for voice/video convergence

• B2C/Extranet

• Bandwidth provider— Traditional telco space

— But VPNs, especially with SLAs, get close


Topology Questions for ApplicationsTopology Questions for Applications

• What part, if any, needs to be in global routing system?— Address space

— AS

• Trust topology

• Who does operational support?— In-house or outsource

— Separate application and network? Single number?


Customer Perceived NeedsCustomer Perceived Needs

• Uptime

• Affordable

• Adequate performance — (but may be perceived as "the best")

• Scalability

• Reasonable business relationship


Defining Service ExpectationsDefining Service Expectations

• Availability

• "Classic SLA"— SLA for interactive applications

— SLA for mission-critical data (computer-to-computer)

— SLA for voice

— SLA for imaging


What breaks?What breaks?

• Application

• Server

• Server farm/host site

• Connectivity


Availability Expectations: Your Site Availability Expectations: Your Site (1)(1)


Your Site (2)Your Site (2)


Your Site (3)Your Site (3)


Don't Forget Backup Power...Don't Forget Backup Power...


Oh, and I want Oh, and I want to pay to pay

$39 per month?$39 per month?


Specifying AvailabilitySpecifying Availability

• Period of coverage

• Restrictions on offered load?

• Maintenance windows?

• When does an outage begin? end?— see quality discussion later in this presentation

• Opportunities for less-than-ideal backup?

• Pricing incentives?


Traffic EngineeringTraffic Engineering

• Throughput

• Need for consistent latency (minimize jitter)

• Availability

• Enough bandwidth

• Bandwidth in the right place

• Transient congestion avoidance

• Alternative ways to supply resources


Higher Layer Threats & ResponsesHigher Layer Threats & Responses

• Single server failure or maintenance downtime

• Individual overloaded servers at single site

• Overloaded site or servers, but sufficient overall capacity

• Server crash

• Clustered servers at site; cold, warm, hot standby

• Local load distribution inside cluster

• Global load distribution among multiple clusters and sites

• Backups, checkpoints, mirroring


Lower Layer Threats & ResponsesLower Layer Threats & Responses

• Routing system failure

• Failure of direct provider or upstream links

• Failure of customer router on LAN

• Single medium failure between customer and ISP

• Multiple ISPs

• Multiple connection to single provider. Diversity contracts.

• VRRP/HSRP. BGP peering to loopbacks.

• Inverse multiplexing. SONET. Dial/ISDN backup. Local loop diversity


Some Routing ScenariosSome Routing Scenarios

Registered address spaceProvider 1Provider 2

Registered or private address space

Private address space


Single point of failure: Single point of failure: single-homed routingsingle-homed routing

ISP

EnterpriseRegistered address space directly allocated or ISP suballocation

ISP-assigned private address space

optional NAT StaticRouting

orKeepalive

DefaultRoute

Router


Multilink single-homed routingMultilink single-homed routing

ISP



StaticRouting

orKeepalive

DefaultRoute

optional NAT

Router


Simple Multihoming to a Single Simple Multihoming to a Single ProviderProvider



ISPPOP 1

optional NAT

Router Router

ISPPOP 1

optional NAT


Have I been solving the right Have I been solving the right problem?problem?



ISPPOP 1

optional NAT

Router Router

ISPPOP 1

optional NAT

HostedServer 1


Local DistributionLocal Distribution



ISPPOP 1

optional NAT

Router Router

ISPPOP 1

optional NAT

TCP Load Distributor


Global Distribution, Single ISPGlobal Distribution, Single ISP

Router Router

SmartDNS/NAT

SmartDNS/NAT


Simple Multihoming to Two ProvidersSimple Multihoming to Two Providers


Router

Primary ISPPOP

MorePreferredDefaultRoute Router

Backup ISPPOP

LessPreferredDefaultRoute


RFC 1998 MultihomingRFC 1998 Multihoming

POP1 POP2adver-tises

Internetroutes

adver-tises

Internetroutes

ISPwith 192.0.0.0/16

assigns 192.0.2.0/22 to customer

eBGP eBGP

customer with private AS

numbers west side hosts in 192.0.2.0/23 and east side in 192.0.3.0/23

iBGP

Advertises192.0.2.0/22192.0.2.0/23

Advertises192.0.2.0/22192.0.3.0/23

advertisessubset of

provider spacemarked

NO-EXPORTRouter 1 Router 2


RFC 2270 MultihomingRFC 2270 Multihoming

customer with private ASnumbers west side hosts in 96.0.2.0/23 and east side in 96.0.3.0/23

ISP 1 ISP 2

advertisessubset of

provider spacemarked

NO-EXPORT

adver-tises

Internetroutes

adver-tises

Internetroutes

Router 2iBGP

Router 1

eBGP eBGP

Advertises96.0.2.0/2296.0.2.0/23

Advertises96.0.2.0/2296.0.3.0/23

Remove privateAS

Remove privateAS


Multihoming & DFZ Table BloatMultihoming & DFZ Table Bloat

Enterprise1.0.3.0/24

ISP 11.0.0.0/20

ISP 12.0.0.0/20

Rest of Internet

1.0.3.0/24 1.0.3.0/24

1.0.0.0/201.0.3.0/24

2.0.0.0/201.0.3.0/24


ScalabilityScalability

• Abilities to:— Add more sites

– Add more users at large sites– Support telecommuters and road warriors

— Add more total users in enterprise

— Add new application types

— Improve availability when needed/perceived

• But it all has to be affordable

Address Space IssuesAddress Space Issues

Rules are always subject to interpretationRules are always subject to interpretation

Ferengi Rule of Acquisition #284Ferengi Rule of Acquisition #284


Address Registries LikeAddress Registries LikeEfficient Usage TechniquesEfficient Usage Techniques

• Dynamic addressing— LAN

– DHCP– BOOTP

— WAN– Local address pools– PPP IPCP– DHCP proxy services

• Aggregated routing announcements

But...


Question:Question:

What is the most important machine What is the most important machine in the hospital?in the hospital?


Operational Aspects:Operational Aspects:Dynamic addressing & Dynamic addressing & violations of the end-to-end assumptionviolations of the end-to-end assumption

• How do you ping/traceroute?— DHCP/DNS linkage

— IPCP linkage

— Layer 2 information

• What about tunnels?

• What about NAT?


Midboxes:Midboxes:who troubleshoots?who troubleshoots?

Classic NAT

PAT/NAPT

Packet Filter

Frame Filter

StatefulPacket Filter

CircuitProxy

ApplicationProxy

Traffic-AwareProxy

Content-AwareProxy

Load SharingNAT

Load AwareDNS

ApplicationCaches

IPsecTunnels


Efficient Addressing can beEfficient Addressing can beHarder to ManageHarder to Manage

• May complicate management— Registry policy (RFC2050) response:

life is hard and then you die. So?— Link DNS and dynamic assignment

• If something is boring and repetitive

Use a computer


Question:Question:

How many of your customers fill out How many of your customers fill out addressing templates?addressing templates?


Question:Question:

How many of your competitors call How many of your competitors call you mean and nasty you mean and nasty

for making your customers do for making your customers do things?things?


Information Gathering:Information Gathering: Think Requirements, not Subnetting Think Requirements, not Subnetting

• Number of sites

—Schedule for growth

• Requirements for flow among sites

—Degree of application level meshing

• Backup and recovery

Questions meaningful for the customer

What is your name?What is your quest?How fast is your sparrow?


Think Requirements, not SubnettingThink Requirements, not Subnetting

Applic-ation Server subnet

Man-age-ment Server Subnet

WAN links to other data centers

14-user Dial Access Server Subnet

100 local user Sub-net

60 local user subnet

Small clinic (11 users, server, router, switch)

Medium clinic (27 users, server, router, switch)

/28 /28 /30 /28 /25 /26 /28 /27 Oct 1999 Jan 2000 Apr 2000 Jul 2000


Schmoozing customersSchmoozing customers

• Price for amount of address space

• Possibly lower overall charges

• Possibly tie to bandwidth


Categorize Space in UseCategorize Space in UseBoth yours and customerBoth yours and customer

ClassfullyOrganized

ClasslesslyOrganized

Existing Public Space

Internal InternetVisible

RFC1918 Private Space

Bogus Space Unnumbered M edia

Address Space


What Addresses do you Need to What Addresses do you Need to Manage?Manage?

• Customer assigned blocks

• POP/Dialup— xDSL, cable, etc. NOT a solved problem

• Infrastructure— Inter-router links— Server farms

– Virtual domains– DNS, DHCP, SNMP, etc.

— Inter-AS links


Laws of Customer Address Laws of Customer Address AdministrationAdministration

• Avoid entering an address more than once

• Automate configuration updating— TFTP or telnet/expect— Replace vs. merge— Scheduled reboot

• Remember "the most important machine in the hospital" (M. Python)

• Document automatically— For troubleshooting— For justifying address allocation

Customer Customer CareCare

What impression do you want to What impression do you want to give?give?


What are you offering?What are you offering?

• Minimal cost residential/SOHO service

• Business SOHO

• Large site service

• Hosting services

Serious question: In your business model, how important is customer perception?


Components of Customer CareComponents of Customer Care

• Sales

• Pro-active quality

• Problem reporting

• Support


What's the difference between

used car salesmen and

network service sales?


Proactive QualityProactive Quality

• Hard to define -- "I know it when I see it"

• Track— Bandwidth utilization

— Downtime

— Number of support requests and response time

— Per-user growth (when known)

• Inform sales or customer BEFORE critical limit reached— Consider very low-key notifications

— Avoid perception of sales pressure

— Consider customer-accessible information


Problem Management for SOHOProblem Management for SOHO

• Provide non-labor-intensive status information— when a failure affecting dedicated user access exceeds

more than (4?) hours, email to subscribers and/or post something on an internal web

— "Technicians are aware of the problem" is minimally helpful

— Give customers an idea if they need to go to serious backup


Quality SuggestionsQuality Suggestions

• Establish a "quality suggestions" email— Use email, not just web, for minimal screening

— And possibly get more informative messages

• Top technical management, and possibly marketing, should routinely read. JW Marriott does.


Trouble Reporting for SOHO & Trouble Reporting for SOHO & BusinessBusiness

• Customers are busy— NEVER use music on hold -- people may have you on hold, in

conference, etc.

— A periodic, professional "you're still in queue," preferably with a waiting time estimate can be useful.

— DON'T tell me "it will be just a moment" if it won't.

— NEVER NEVER blather about how important my call is. If it were really that important, you would have answered it already.

— NEVER NEVER NEVER tell me about new products on the problem reporting line

• First line support doesn't have all the answers— Don't get defensive or pretend knowledge that isn't there

— Provide meaningful ways to get more detailed information

customer satisfaction 201 howard c. berkowitz [email protected] [email protected]...

Documents