Financial Institution Use Case

R O I S T U D Y

R O I S T U D Y

2

Forward Networks has developed the most scalable, vendor-neutral network verification platform in the industry. The platform can compare the intent of the network architects, including all connectivity and compliance requirements, to the actual behavior and capabilities of the network. Network IT teams can now troubleshoot faster and proactively eliminate errors prior to a breach or outage. Because Forward Enterprise automates the intelligent analysis of network designs, configurations and state, we provide an immediate and verifiable return on investment (ROI) in terms of accelerating key IT processes and reducing person-hours of highly skilled engineers in troubleshooting and testing the network. In this white paper, we quantify the ROI of a large financial services firm and document the process improvements that led to the cost savings and a more agile network.

Forward Networks was part of a corporate-wide initiative to increase network bandwidth, drive down business-impacting outages by 99%, and improving automation by coding their business policies into their network automation software. Forward Networks provides the network assurance improvements by verifying their network changes, and comparing their complex business requirements to network behavior across a very large, complex, rapidly changing environment.

Gartner Group sums up the benefits from these kinds of corporate initiatives through the addition of intent-based networking technology in their March 2018 report, “Market Guide to Network Automation”:

The network automation market is evolving from operationally focused network configuration and change management tools, to policy-based tools and orchestration tools, and – in rare, forward-leaning use cases – more strategic and business-aligned, intent-based networking systems.

R O I S T U D Y

3

ROI Methodology: Key Process Improvements and MetricsThe primary function of Forward Enterprise is to collect device configuration data from every network device, including state information that governs the active behavior of the device. Forward Enterprise can then model and anticipate the behavior of the entire network end-to-end, and report on potential vulnerabilities, policy violations or risk exposure. The common identified process improvements we have seen at several large customers are:

• Reduced time to trouble ticket resolution• Reduced time spent on audit related fixes and

updates• Fewer, shorter and more reliable change windows or

network updates• Some of the quantifiable savings include:• 50% faster trouble ticket resolution• 90% faster audit related fixes• 33% reduction in aborted or erroneous change

windows

For each of these processes, we have identified improvements in several steps to complete the process, as well as documenting the actual improvements and related costs from this specific financial services customer.

Process 1: Reduced Time to Trouble Ticket Resolution

Forward Enterprise improves the time to resolve the average high to medium-high complexity ticket. These events are defined as trouble tickets which require the operator to troubleshoot a path through a network of more than four network elements (switches, routers, firewalls, or load balancers). 

The typical work flow for a ticket resolution is defined in Figure 1a.

Figure 1a – Ticket resolution process flow.

R O I S T U D Y

4

The approximate time spent on each step is illustrated in Figure 1b. 

Using Forward Enterprise, steps 1 – 3 are completely automated and reduced to effectively zero time spent. These steps are done instantaneously as the network engineer begins troubleshooting. Forward Enterprise is designed to highlight network paths, including traffic origin and destination that could potentially result in an error or policy violation. All paths that either support or violate a defined policy or requirement are immediately highlighted for further analysis.

Similarly, in step 4, “investigation on a hop-by-hop basis”, effort is reduced significantly, usually to no more than a few minutes without the need to login into each network device. Forward Enterprise models the behavior of complete paths and quickly identifies through its emulation of each network device exactly which hop of the network could be causing a particular issue.

Step 5, “isolating the failure”, still requires significant time investment by a network engineer, but is reduced through the analytical tools and query capabilities of the Forward platform. As network paths are explored to isolate the root cause, it’s easy to explore each hop in the path to determine the behavior of each configured service, as well as to view configurations pertaining to those services.

In Figure 2, we show how a typical network path is presented. Each hop can be quickly analyzed to see how traffic is routed and modified for each service. In addition, once the desired device configurations are updated, Forward Enterprise can automate the testing of the new configuration in step 8. IT admins can

Figure 1b - Percentage of time spent in each step of the ticket resolution for a high to medium-high complexity trouble ticket.

Figure 2 – Forward Enterprise shows allowed paths through the network that match the policy or search query. We can explore paths device by device to quickly identify root-cause issues.

R O I S T U D Y

5

immediately see if the proposed changes remediate the problem as well as not introducing any potential new policy violations. We will explore the net benefits from this step as we look closer at changes windows.When considering the process improvements from steps 1-4 of remediating trouble tickets, a conservative measurement of time spent on ticket resolution ranges 25-50%. Our financial services customer conservatively estimated saving 25% of man-hours on trouble ticket remediation, or over 23,000 hours per year. As we document in the summary table, assuming $100/hour for the overhead costs of a network engineer, this translates to a savings of $2,300,000 per year.

Process 2: Reduced Time Spent on Audit Related Fixes

Forward Enterprise reduces the time needed to audit the network with regards to several fundamental configurations. Audit related problems are defined as any item within the Forward Platform that is found by using the “predefined check” library or customized Network Query Engine (NQE) checks. Significant examples are:

• Link speed mismatches• Maximum Transmission Unit (MTU) size

mismatches• Forwarding loops• VLAN definition inconsistencies• Port channel inconsistencies

Typically, audit related items are bulk remediated by a small team of network engineers. Audit processes are very detailed and time consuming: 95% of the time is spent inspecting network elements that do not exhibit the problem, simply verifying that they are configured correctly. 5% of the time is spent remediating failed network elements.

Forward Enterprise provides a report that instantaneously identifies configurations that are not compliant, so that network engineers do not have to look at network devices without problems. Additionally,

Forward Enterprise verifies that any changes have been successful, reducing time to resolution in case of more complicated audit items (e.g., forwarding loops).

R O I S T U D Y

6

In our ROI study, Forward Enterprise provides coverage for 60% of the customer’s self-identified audit items, and reduces time to resolve them by 95%. There is no appreciable change in the time to resolve the 40% of uncovered audit items. Across all self-identified audit items, this means time to audit is reduced 57%.

Process 3: Reduced Number of Change Windows

Forward Enterprise reduces the number of change windows needed to accomplish complex changes within the network. Roughly one third of changes fail because of change procedures, unexpected network conditions, or user error. Forward Enterprise creates a snapshot at the beginning of the change window and another snapshot at the end of the change window. After pushing all changes to the network, the operator can automatically verify that all services are functional.

A typical change window flow process is illustrated in Figure 4.

In Fig 4, 65% of changes result in a success (green) result. However, 35% of changes do not. 20% of changes resulting in “Revert” (gray) result in one additional change window. Because of improved troubleshooting, Forward Enterprise can help identify the failure quicker and troubleshoot 50% faster. (See Reduced time to ticket resolution)15% of changes result in exiting the change window

Figure 4 – Common workflow for a network change window, with aborted changes highlighted in black or gray.

Figure 3 – Forward Enterprise continually checks path and device-to-device configurations to identify errors and inconsistencies that cannot be detected by looking at devices individually.

Cantroubleshoot?

R O I S T U D Y

7

in a failed state. These failures result in two additional changes windows. One to solve the problem created by the change window, and a subsequent change window to solve the original problem. When using Forward Enterprise, the following work flow is observed as in Figure 5.

The result is only 10% of changes will require an additional change window due to a failure instead of 20% requiring one additional, and 15% requiring two additional. Given 100 theoretical changes, in the first example, 150 change windows are required. In the second example, with Forward, 110 change windows are required. The reduction in change windows necessary is 27%, with the complete elimination of self-inflicted emergency change windows.

ROI AnalysisBased on the process improvements discussed above, we modeled time and financial variables at our large financial customer according the following questionnaire:

Figure 5 – Immediately recognizing failures in proposed changes can greatly reduce the number of aborted change windows and verify the correctness of deployed changes.

Questionnaire Value

How many network engineers do you have working full time? This will help round out the overall cost to your organization.

60

On average, how much do you pay your network engineers per hour? $100

How many hours each week does each network engineer spend on ticket resolution?

30

On average, how many hours does each engineer work on audit related fixes? 10

On average, how many change windowsdo you do per month? 4

On average, how many engineers do you assign per change window? 10

On average, how long (hours) is your change window, include prep work and post work?

10

R O I S T U D Y

8

Based on the above inputs, we calculated the costs of managing their network trouble tickets, audit resolution and change window processes. We summarize the results in the following table:

Supporting Data from Other CustomersIn this section, we present benefits from three other customers to show a range in process improvements among different types of organizations and networks. This should help validate some of the conservative estimates of our large financial customer, as well as form a possible range for your own estimates.

Customer A

Customer A is another large financial institution. They are a U.S.-based Fortune 50 company with a network deployment of tens of thousands of devices. Their IT department exceeds ten thousand employees, mostly in the U.S. Their network forms a critical component of their IT infrastructure, with zero downtime expected. The data compiled from this customer consists of a subset of network devices, and a subset of their Forward Enterprise deployment. This network is a complicated network with more than five different switch, router and firewall vendors. 

• Size of network: 2000 devices • Reduced time to ticket resolution: 55% • Reduced time spent on audit related fixes: 95% with

60% coverage = 57% reduction • Fewer change windows: 27%• 60% of P1 tickets could be resolved 75% faster with

Forward Enterprise  

Operational Costs Today With Forward

Hours Spent on ticket resolutionHours Spent on audit related fixesHours Spent on change windows

Overall reduction in costsrelated to ticket resolution, audit related fixes andchange windows

PER WEEK

1,80060092

PER YEAR

93,60031,2004,800

PER WEEK

1,35038478

PER YEAR

70,20019,9684,080

$249,231 $12,960,004 $181,246 $9,424,803

PER WEEK

$67,985

PER YEAR

$3,535,201

R O I S T U D Y

9

Customer B 

Customer B is a U.S. government agency with a network deployment of over a hundred thousand devices. Their network forms a critical component of U.S. federal infrastructure. 

The data compiled from this customer consists of a subset of network devices, and a subset of their Forward Enterprise deployment. This network is a single-vendor network, mostly running Layer 2 switches and some firewalls. 

• Size of network: 200 devices • Reduced time to ticket resolution: 25% • Reduced time spent on audit related fixes: 60%, with

60% coverage = 36% reduction 

Customer C

Customer C is a large Internet services provider with thousands of network devices. The customer uses Forward Enterprise to avoid critical outages by frequent analysis of current configurations. Additionally, they have opened a self-service portal for application teams to assess the behavior policies of the network, which has reduced the number of network tickets opened by the application team.

• Reduced time to resolve Priority 1 incidents: 2 hours 14 mins, to 1 hour 4 mins. >50% reduction

• # of Priority 1 incidents in the year prior to Forward Enterprise and with Forward Enterprise: 12 incidents down to 4

• Approximate reduction in financial impact from reduced number of incidents: $380,000

• Monthly network tickets from app team before and with Forward Enterprise: 100 tickets down to 30. Saving 1 FTE networking headcount, at approximately $100,000/year.

• Total two-year cost-savings from reduced P1 incidents, time to resolve incidents and reduced network tickets from the application team: >$600,000

R O I S T U D Y

1 0

SummaryForward Networks has delivered the first behaviorally-accurate software model of large enterprise networks. Our platform provides a revolutionary approach to analyzing and troubleshooting complex network issues, as well as allowing functional validation of network designs. Cost savings from leveraging Forward Enterprise can be a significant percentage of network IT spending when factored over the three areas of trouble ticket resolution, audit-related fixes and reduced number of change windows. Based on these process improvements and savings, we expect intent-based verification to be an important part of IT networking strategy and future network automation projects.

For more information: http://forwardnetworks.com.