make better funding decisions with accurate broadband ... - make better...case study: inaccurate...

Make Better Funding Decisions withAccurate Broadband Network Data A guide for federal, state and local governments

© 2006-2020 Ookla, All Rights Reserved Ookla White Paper Series - Government Broadband Funding | 2

1

2

3

How Bad Data Can Impact Government Broadband Spending All Politics Is Local — and So Is Broadband Understanding — and Closing — the Digital Divide

Case Study: Inaccurate Network Performance Data Creates an Incomplete Picture of Broadband Access in Upstate New York

Considerations for Evaluating a Network Data Provider 1. Accurately testing a network's full throughput capacity 2. Testing to a local server to eliminate latency and other performance bottlenecks 3. A proven testing methodology to ensure accurate network performance data 4. Accurate test geolocation and precise location mapping 5. Meaningful analytics showing locations where providers are actually delivering broadband services

Conclusion: Bad Data Can Negatively Impact Government Funding

Our Mission

References

Appendix

3

7

13

27

29

30

31


How Bad Data Can Impact Government Broadband Spending


State and federal officials are charged with spending billions of dollars in funding to improve broadband availability, particularly in rural areas. While many yearly budgets already earmarked money for broadband development projects before COVID-19, the pandemic has highlighted deep digital divides at a time when the public is more reliant than ever on the internet for work, education and other essential services. In the coming years, Congress will likely budget even more funding toward ensuring every citizen has access to high-speed internet access.

For federal, state and local governments to correctly allocate this funding to serve the most constituents, they need accurate data on broadband availability and network performance. Data drives budget and spending decisions, and historically, a significant portion of these funds have been misdirected by relying on bad data.

To improve this decision-making process, it is vital that governments have comprehensive, accurate and up-to-date data on the state of broadband networks within their jurisdictions.

Furthermore, government officials and employees must be able to correctly interpret this data to discover where broadband investments will have the most impact. Understanding the performance and availability of existing broadband networks means that governments can use limited resources wisely, instead of wasting them.

This white paper explains:

• How crowdsourced network performance data can help drive better policy • How to define what data is valuable for broadband prioritization • What purposes different datasets serve in the decision-making process • Selection criteria for choosing a network data provider • What types of insights can be derived and how to use them to make better spending decisions

Whatever level of funding is made available in the next round of legislation, it will not likely be enough to build out broadband access to every corner of the country.Still, it is critical that these funds be spent as wisely as possible. Overbuilding networks that already meet target thresholds for speed and availability of services means denying other communities their opportunity to have broadband access and high-speed connectivity.

1


All Politics Is Local — and So Is Broadband

As if broadband access wasn’t already a hot topic, the shift to working and learning from home has underscored the need for high-speed connectivity across the entire country. In the U.S., our elected leaders are getting contacted at an unprecedented rate by constituents who find themselves excluded from schooling and job opportunities due to a lack of connectivity. Legislators whose districts include rural areas have long been aware of how the “digital divide,” created by lack of broadband access, negatively impacts communities.

Many households are trying to do much more with their internet connections than they ever have before. What had been a reasonable download speed for streaming a movie on Netflix or Hulu is not sufficient to support several people operating multiple devices on one network. Video conferencing requires streaming on the upload side of the connection as well as the download side. In many households, more than one employee is working from home and reliant on video conferencing. With many students participating in distance learning and also using home internet for entertainment, most households are learning what the upper limits of their network capabilities are the hard way.

Considering that 30 years ago almost no private residences and few businesses had internet service, we should be proud of the progress we have made. Even so, we should also recognize that internet access has changed our society in dramatic and unexpected ways. The internet provides access to work and educational opportunities as well as vital services like telemedicine and entertainment. People who do not have access to broadband service that meets the Federal Communications Commission (FCC) minimum guidelines for network speeds (25 Mbps download and 3 Mbps upload) are quickly falling even farther behind.

Access to broadband has three basic requirements:

1. Connectivity at accepted minimum speeds must be available in the desired location. 2. The user must have the equipment needed to connect. 3. The user must have the necessary knowledge to use the equipment and navigate online.




Download speed measures how quickly you can pull data from a server on the internet to your device. Download speed is measured in megabits per second (Mbps).

Megabits per second. Used to measure download and upload speeds.

Gigabits (billions of bits) per second. Used to refer to speeds in excess of 1,000 Mbps (1 Gbps).

Upload speed measures how quickly you send data from your device to the internet. Upload speed is measured in megabits per second (Mbps).

Jitter is a measure of the variability in ping over time. This metric shows the average di�erence between consecutive latency measurements. Jitter is also called Packed Delay Variation (PDV).

Latency or ping is the reaction time of your connection. Ping is measured in milliseconds (ms).

Mbps

Gbps

Network Performance Metrics and Definitions

What Internet Speed do you need?

The above image shows typical speeds required for a single user to perform various activities on their network connection. The required speeds increase proportionally when more than one user is sharing the same network.

2 Mbps 10 Mbps

4k and beyondStreaming

HDStreaming

#DownloadSpeed

SD VideoStreaming

25 Mbps


Understanding — and Closing — the Digital Divide

Urban constituents typically have high-speed internet service available at or near their address — but for families experiencing economic stress, the cost of food, rent and other necessities consumes too much of their monthly budget to justify adding a service previously perceived as a luxury. Poverty further impacts access to computers and other home internet equipment, and therefore these constituents don’t get the opportunity to build experience using the internet or the skills required to participate in the digital economy.

Rural constituents often face similar challenges getting access to knowledge and economic resources, but are faced with the additional challenge of a lack of connectivity in their area. To commercial internet service providers (ISPs), there is a tipping point where population density is too low to make investments in high-speed internet infrastructure profitable.

Ookla tracks over 65,000 ISPs globally and nearly 10,000 ISPs across the U.S. and its territories. The networks these ISPs operate vary in scale dramatically with the number and size of markets they serve. Some are nationally recognizable, such as Comcast (aka Xfinity), Verizon, AT&T and Spectrum. Most are regional and local providers with which you might only be familiar if you live within their service area.

These smaller players have proven critical in helping rural America achieve its current level of connectivity. As larger operators have consolidated over the past several decades through acquisitions of other ISPs, less populated areas have not attracted the same level of investment. This is due, in part, to the amount of infrastructure that must be built per household being served. The more rural the landscape, the more expensive it becomes to provide service to each residence. Challenging topography like mountains and wetlands can also drive up deployment costs and disincentivize even the largest ISPs from development into some areas.



Incomplete Picture of Broadband Access

Case Study: Inaccurate Network Performance Data Creates an Incomplete Picture of Broadband Access in Upstate New York

In August 2020, the office of Congressman Anthony Brindisi, New York, District 22, U.S. House of Representatives, released a report highlighting the lack of broadband service across the district.1

New York’s District 22 (NY-22) is large, touching approximately 20 miles of shoreline on Lake Ontario and stretching south to the Pennsylvania border. The landscape is mostly rural but includes the relatively small metropolitan areas of Binghamton and Utica-Rome and skirts the easternmost suburbs of Syracuse. Running east to west, Interstate 90 bisects the district into uneven northern and southern sections, and I-81 runs north to south, detouring outside the district for a few miles to run through Syracuse. District 22 comprises four complete counties — Chenango, Cortland, Madison and Oneida — and portions of four more — Broome, Herkimer, Oswego and Tioga. The 2018 Census Bureau estimate pegs the population at just under 700,000. The people of the district are more evenly distributed between city and country life than the average district in the U.S., with 57.5% living in urban areas and 42.5% (roughly four out of ten people) living in rural areas.

Like so many rural regions of the U.S., broadband has not yet reached all of the constituents in NY-22. Households without high-speed connectivity have been particularly impacted by the effects of the ongoing COVID-19 crisis and the ensuing lockdown. As for many urban dwellers, working and learning from home are the only options for many employees and students. Video calls and streaming classrooms are the new normal, and broadband access has quickly shifted from a luxury to an absolute necessity for most people.

Details of the Report

The NY-22 report was issued August 3, 2020, and we agree with the congressman’s call to action as he makes four recommendations for his district:

1. Smarter decisions with better data 2. Bigger broadband buildout 3. Stronger oversight of taxpayer dollars 4. Greater market competition

2


The report included a survey of 1,187 respondents from the district, and Ookla applauds such efforts to get direct feedback from the public. Data collected in this manner will include contextual information that may not be captured in any other way. Individuals who care enough about internet connectivity to seek out their representative’s website and report their experience are generally personally invested in the issue.

Some people may not be able to perform a network performance test from their home because they have no connectivity at all or because the network is so poor the test cannot be completed. Informing lawmakers by lodging a complaint once they get to a connected location may be a citizen’s only way of taking part in the data collection that will inform policy decisions.

Our concern with the rest of the report is that the network performance test results the report was derived from painted an inaccurate picture of what constituents were actually experiencing in the district. The results presented greatly underestimated the speeds being delivered by the service providers throughout most of the study area while overestimating some others. The speeds included in the report used network performance information exclusively from tests taken with M-Lab.2 The speeds measured by Speedtest® for the same areas and the same time period are dramatically higher in most areas — indicating that additional infrastructure investments are unnecessary where constituents can already achieve network speeds that meet FCC minimums.

There are certainly communities within this district that have inadequate service and need greater investment. However, by relying on numbers that inaccurately indicate lower speeds than reality, the congressman’s office runs the danger of targeting certain areas for funding that already have adequate broadband service. Resources are limited and these funds should be allocated to areas that lack the connectivity needed to meet the FCC’s minimum of 25 Mbps download speed and 3 Mbps upload speed.

The report on NY-22 refers to the speeds shown as averages. There is more than one way to calculate an average, and the distinction is an important one. The “mean” average is the sum of all measurements divided by the number of records used. This number is valuable, but it can be influenced by a small portion of records that may be extremely high or low (outliers). As fiber is installed within an area, a significant number of tests from ultra-high-speed connections can skew mean averages up.




ZIP Code

13844

13862

13324

13054

13416

13843

13431

13491

13812

13409

13801

13830

13460

13425

13438

13485

13733

13077

13815

13803

13477

Town

South Plymouth

Whitney Point

Cold Brook

Durhamville

Newport

South New Berlin

Poland

West Winfield

Nichols

Munnsville

McDonough

Oxford

Sherburne

Oriskany Falls

Remsen

West Edmeston

Bainbridge

Homer

Norwich

Marathon

Vernon Center

M-Lab

0.6

0.9

1.2

1.3

2.4

2.5

2.5

2.4

2.3

3.0

3.4

3.9

4.1

4.5

4.8

4.6

4.8

5.1

5.3

5.5

5.3

Ookla

8.3

71.6

63.7

4.3

90.3

11.0

70.5

58.6

69.3

50.6

9.2

13.7

16.3

23.3

47.4

34.3

62.4

65.8

43.5

14.5

23.9

(+1,283%)

(+7,851%)

(+5,212%)

(+229%)

(+3,664%)

(+339%)

(+2,720%)

(+2,340%)

(+2,914%)

(+1,586%)

(+171%)

(+251%)

(+299%)

(+418%)

(+887%)

(+645%)

(+1,200%)

(+1,190%)

(+721%)

(+163%)

(+350%)

M-Lab

0.3

0.3

0.5

0.6

0.4

0.5

0.6

0.8

1.1

0.6

0.6

0.7

0.7

0.9

0.6

0.9

0.9

0.7

0.9

1.0

1.4

Ookla

0.9

11.2

11.2

0.7

11.5

1.1

11.5

11.1

11.4

6.0

1.0

1.5

2.3

5.8

11.0

12.2

11.6

11.6

10.7

1.6

4.4

(+190%)

(+3,617%)

(+2,138%)

(+18%)

(+2,785%)

(+110%)

(+1,823%)

(+1,293%)

(+935%)

(+898%)

(+60%)

(+111%)

(+221%)

(+544%)

(+1,738%)

(+1,258%)

(+1,189%)

(+1,554%)

(+1089%)

(+61%)

(+217%)

Upload Speed (Mbps)Download Speed (Mbps)

In this use case, it’s more effective to consider the “median,” the middle value within a set of measurements. Mathematically, half of users experience download speeds above the median value, and half of users’ speeds fall below the median. This method ensures that outlying incidents of very high or very low performance do not skew the final number used to represent the entire data set.

The table below shows comparisons of the median download and upload throughputs for the twenty ZIP codes specified in the report as having the “worst” speeds within NY-22. Looking at Ookla and M-Lab data side by side, you can see that M-Lab vastly under-reported the network throughput in every single ZIP code represented in the congressional report.

Numbers shown in red do not meet FCC minimum speeds of 25 mbps download / 3 Mbps upload.

Comparing Median Download and Upload Speeds in M-Lab’s Reported “Worst” ZIP Codes for NY-22 Ookla® Speedtest® and M-Lab Data | December 30, 2019 through June 21, 2020


The ZIP code showing the least amount of difference by percentage between Ookla and M-Lab data was 13803 (Marathon) where M-Lab’s recorded median was 5.5 Mbps and the median from Ookla data was 14.5 Mbps. So the typical speed in Marathon measured by Ookla’s Speedtest was over two and a half times as fast as the average measurement captured by M-Lab. On the other end of the scale, in Whitney Point, M-Lab’s recorded median was 0.9 Mbps while Ookla measured a median of 71 Mbps, almost eighty times faster.

Contrary to M-Lab’s data, Ookla data determined that 12 of the listed ZIP codes met the FCC minimum threshold of 25 Mbps download / 3 Mbps upload, with two additional ZIP codes falling just below the thresholds.

A policymaker looking at M-Lab’s data alone might incorrectly assume that every single ZIP code in the district is wildly underserved and thus needs funding.




When bad data leads to underserved communities — a spotlight on New Berlin, NY

In a few outlying ZIP codes, the speeds measured by Ookla were actually much slower than those measured by M-Lab. We zoomed in on one example where M-Lab’s data looked questionable in the very rural town of New Berlin (13411). M-Lab results showed a median download speed of 103.5 Mbps, but the median upload speed of 102.6 Mbps looked too good to be true. If this measurement was accurate, it would be outstanding service for such an isolated community. M-Lab’s report names New Berlin’s ZIP code the fastest in the entire district, which may have come as a shock to the residents there.

Ookla’s results for the New Berlin ZIP code show a strikingly different picture: a median download speed of 18.5 Mbps and median upload speed of 3.3 Mbps. While the upload number meets FCC minimums, the download certainly does not.

If ZIP codes are used to determine eligibility for broadband funding, the M-Lab results would indicate that the area around New Berlin is not in need of broadband infrastructure assistance.

The primary internet provider in this part of rural New York is Frontier, serving their subscribers with DSL (Digital Subscriber Line) service. The median speeds reflected for Frontier in the table below — 10.62 Mbps download and 1.05 Mbps upload — are typical for this older technology. Frontier subscribers represent over half the tests taken within the ZIP code, so the median speeds measured by Speedtest are reflective of the typical experience in this area. M-Lab’s results are nearly ten times faster on the downlink and nearly one hundred times faster on the uplink — likely skewed by a high volume of tests on the new OEConnect fiber network, which only serves part of the ZIP code. While M-Lab’s measurements may reflect what some residents experience, those speeds are not representative of typical service in this area, even for users who do have broadband access.

Ookla Speedtest results by ISP for New Berlin, NY 13411

The results of our comparison of the 20 “best” ZIP codes in NY-22 can be found in the appendix of this document.

ISP

Frontier

Spectrum

OEConnect

Test Count

305

136

117

User Count

93

26

20

1.05

11.36

97.72

Upload Speed (Mbps)

10.62

66.60

78.19

Download Speed (Mbps)


One key issue with relying on ZIP codes as the boundary to determine funding eligibility is that some communities may still experience average speeds below 25 Mbps in small pockets. We examine this subject further on page 22 of this white paper. That section also provides policymakers more information on how to understand differing broadband performance across arbitrary geographic boundaries such as ZIP codes.

It is clear from these results that M-Lab’s performance test does not measure the full capacity of a network connection — and thus does not accurately reflect the real-world internet speeds consumers are experiencing. So what accounts for such large disparities in the results?

These disparities in measured speed generally arise because some network data providers have low user adoption among consumers, limitations in their testing infrastructure, questionable testing methodologies, or inadequate geolocation resources to precisely locate where a given test was taken.

Read on to discover the key capabilities and outputs government officials and employees should look for when considering a network data vendor — and especially when choosing where to allocate budget to best serve constituents.



3

1

2

3

4

5

Considerations for Evaluating a Network Data Provider


Not all network testing providers are created equal. To get a clear picture of broadband speeds and understand where communities aren’t being adequately served, it is crucial to look at the most robust, comprehensive data available on the networks.

There are five key capabilities that any network testing provider should offer — but that many do not:

The capability to test the full throughput capacity (i.e., the maximum achievable upload and download speeds) of a network connection

The ability to test to a local server, eliminating potential bottlenecks due to distance or local latency

A proven testing methodology that provides accurate insights on the internet service being provided to subscribers

The capability to correctly geolocate a given test and map its results to precise locations

Meaningful analytics showing locations where providers are actually delivering services that meet or exceed the current definition of broadband

The following sections provide more information about how these capabilities support federal, state and local policymakers in deciding where to spend broadband funding.



Accurately testing a network’s full throughput capacity

To accurately measure the download speed of an internet connection, a testing application such as Ookla’s Speedtest or M-Lab’s Network Diagnostic Tool, running on the end users’ machine, pings dedicated testing servers to send as much data as possible. The testing application then measures how much data it receives back from the servers during a period of time (usually 10 or 15 seconds).

Each test requires a large enough data transfer to ensure that it fully saturates the network connection and measures the full throughput capability. With ISPs offering high-speed connectivity such as optical fiber to the home, this problem is only getting worse. These connections are able to handle speeds between 1 and 2 Gbps, roughly 40 to 80 times more than the minimum broadband speed of 25 Mbps.

Some network testing providers, however, do not have adequate testing infrastructure to account for normal demand on the network, and thus are incapable of accurately measuring peak network speed.

Since M-Lab is a Google partner, search engine results drive traffic their way for performance testing. This is not because they are the best test, but because of the relationship between the two organizations. In fact, M-Lab’s testing infrastructure is extremely limited in a way that produces inferior testing outcomes. Currently, M-Lab has fewer than 60 servers across the entire United States listed on their infrastructure map (see first image on page 16) with no servers shown in Alaska, Hawaii or the U.S. territories.

1



When a user takes a test through M-Lab, the test measures the speed between the user’s device and a single — and often distant — server. When data travels between the user’s device and a distant server it may have to traverse many network “hops” (when a packet of data is passed from one network segment to the next) to get there. The additional lag time this introduces to the test results can negatively impact the user’s perception of the local network’s performance. If the server being used for that specific test is also trying to run many other tests at the same time, it may not have sufficient capacity to provide an accurate result. If there are multiple users simultaneously testing their high-speed connection, the tests might consume all the available throughput from a single test server, thus denying other users the capacity required to measure their own connection. Simply put, M-Lab’s infrastructure is insufficient for internet performance testing in the modern era.

The Speedtest Server Network™ was purpose-built to manage a global scale of volume, with testing servers strategically located in every country and most major population centers. We have over 12,000 servers in the network with more than 1,600 in the United States and its territories. Ookla has 68 servers in New York State alone.

The Speedtest platform begins each test by polling several nearby servers to ensure that the closest servers with the lowest latency are used for the test. Ookla’s Speedtest goes one step further: understanding the critical role that a single test server can play, it automatically asks several servers nearby to all generate traffic simultaneously, thus eliminating the potential bottleneck caused by a single, over-taxed server. If one server is at capacity, the others can make up for it.



Ookla® Speedtest Server Network™ Servers in the United States and New York State, October 2020

Although we don’t have visibility into the internal metrics of other testing providers to assess the full impact of the stress of multiple tests at once, our internal metrics indicate that during peak times, a high level of traffic has the potential to overload systems and return degraded results when there are not enough available servers to support high-throughput testing. It is possible that high volumes of traffic are overwhelming M-Lab capacity during peak times, which results in significantly lower download and upload speeds than those reported by Speedtest and other network testing solutions.

Accurate test measurements are critical. When a testing provider consistently under-reports network speeds across geographies, it can look like every community is in crisis — and funding might not end up in the communities experiencing the most need.

M-Lab Servers in the United States and New York State, October 2020 Source: measurementlab.net/status/



Testing to a local server to eliminate latency and other performance bottlenecks

When evaluating how a given ISP is performing for a given location, it is important that a testing provider takes proximity and latency into account. With our aforementioned 12,000+ servers, a Speedtest user can almost always test to a nearby server, often a server on their internet service provider’s own network — with the remainder of tests pinging nearby servers on other providers’ networks. Connections to a distant server or one too far outside of an ISP’s network can introduce variables in performance that are beyond the local ISP’s ability to control.

We often imagine the internet as a single entity, but in reality, it is actually a network of networks. Thousands of service providers around the globe connect with one another forming a “web” so that, regardless of where the desired content may be, there is a pathway to connect the user to it. The content retrieved to fulfill a given request may be provided from a server just a few miles away or from one on the other side of the world.

The speed at which the requested data is returned can be influenced by every hop between networks and by the responsiveness of the site from which the requested data must be retrieved. Slow page loading and sluggish responses can be caused by much more than a poor local internet connection.

2

Nearby servers have optimal and full performance

Distant servers introduce latency and bottlenecks which reduce performance



You could be many hops away, through multiple data centers, from the content you are trying to reach — and each hop can slow your total connection speed. It also can negatively impact latency, which measures how quickly your device and a server can talk to one another.

Ookla’s Speedtest Server Network is highly distributed. Referring to the information we previously shared about the NY-22 district, there are over 20 Ookla Speedtest servers in Upstate New York. This includes locations in Utica and Binghamton, which are located in the district, as well as Syracuse, whose eastern suburbs are also in the district. There are no M-Lab servers anywhere in the district. The closest M-Lab servers to upstate residents are in New York City, where they currently operate only seven servers to handle testing traffic for all or part of nine nearby states, representing over 50 million people. In contrast, Ookla has over four dozen servers in the NYC metro area alone and over 150 servers in the northeastern United States. In areas like Kansas, Minnesota, Alaska and the more than 40 other states and territories where M-Lab has no servers at all, rural users are unlikely to get an accurate picture of their broadband network performance.

ISPs can only improve and be held responsible for the portion of the internet they control. Their key responsibility is to maintain adequate bandwidth from their local network to the nearest “hub” that connects their clients to the larger internet. Penalizing a network provider for issues related to external forces would be an unfair practice and could discourage companies from participating in the investment needed to improve broadband connectivity to some communities.

Testing to local servers provides the most accurate measure of local network performance — and gives policymakers an accurate understanding of which ISPs are meeting the needs of their constituents.



A proven testing methodology to ensure accurate network performance data

Every analyst has a method of interpreting data and reaching a conclusion. As mentioned above, the results from both Ookla and M-Lab shown in the table of ZIP codes are medians, representing the value at the middle point in the list of performance test results in a given area for a specified time range. This choice in averaging represents only one aspect of the overall methodology.

Consumer-initiated network performance testing like Speedtest creates millions of individual results, at whatever intervals the public decides to check their internet speeds. This crowdsourced data is then aggregated into samples to make sure that no single user’s results skew overall network metrics when that user takes a high volume of tests.

Some people simply run more tests than others. Users may be logging their results from multiple tests for a variety of reasons ranging from curiosity to showing their provider they are not receiving consistent service at the level they are paying for. People who are experiencing poor connections may run several tests in a short period of time. If these account for an outsized percentage of the tests taken on a given platform, the results will report speeds significantly below the true user experience, regardless of which averaging method is used.

Establishing a methodology that accounts for these scenarios is an important part of determining the true level of service that a given ISP is providing their customers within a defined geographic area.

3

Why does someone use Speedtest?

While sometimes it is because a user is experiencing poor network conditions, many times it is simple curiosity: Am I getting the speed I’m paying for with my existing subscription package? Will I have difficulty with my upcoming video call? Other events also spur users to run a test, such as traveling to a new location or buying a new laptop, phone or Wi-Fi router.

Consumers can also use Speedtest to diagnose a problem with their networks. If a web service such as a social media or gaming site is responding slowly, conducting a network performance test can help determine if the problem lies with the internet connection or with the service itself.



Ookla’s Speedtest methodology combines all of a single user’s test results into a single sample that represents their overall experience during the time period in question. As part of the sample-building process, we apply several data quality filters to ensure that each sample accurately represents both the user and the network. Our team of data scientists ensures our measurements are accurate and representative of real-world network performance. With over 14 years of leading this industry, Ookla is the gold standard for network testing and a trusted data source for global regulators, governments, industry groups, journalists and communications service providers. Ookla is also a trusted data source for global regulators, governments, industry groups, journalists and communications service providers.

Other network datasets, such as M-Lab’s, do not always factor this level of quality assurance into their methodology.3 Some providers use results from failed or incomplete tests when calculating their average results. Some do not have the technical capability to attribute sets of results to a single user over time, and allow heavy daily users of their tests to disproportionately influence overall network performance measurements. As an open-source project, M-Lab also does not have the methodological rigor required in the private sector. Ookla has a global clientele consisting of the largest ISPs, mobile operators, enterprises and regulators, who demand the highest quality data of its kind.

Policymakers can confidently rely on Ookla data to correctly interpret vast amounts of — often conflicting — information about a given network.



Accurate test geolocation and precise location mapping

Precise location information is required for each measurement in order to accurately describe typical internet performance in a given region. Without precise location, data can easily be mis-categorized into the wrong ZIP code, census block or other boundary.

However, determining the exact location where a test was performed is not always simple. Mobile phones and tablets all contain a GPS chip which can provide highly accurate location information for the majority of tests taken on those devices, assuming the user has opted to allow the location to be recorded. Without a GPS reading, the location fix is much less precise. Broadband network performance tests taken from a desktop and most laptops will provide a location based upon IP address geolocation, a method of determining a computer’s location using the device’s IP address. Although internet service providers know which IP addresses belong to each address served, third parties (such as network testing providers) do not report location to this level due to privacy concerns. This type of location will only define a general area, and that area is usually identified as the centroid (central point) of a ZIP code boundary. Further, larger ISPs may cycle their IP addresses throughout large regions, which can lead to stale and highly inaccurate locations being used.

Looking again at the NY-22 congressional district, we have layered all the results from the Speedtest platform that could be mapped to an accuracy of 500 meters or less in the image on the following page. These tests took place during the first six months of 2020. Layers were placed in a hierarchical method where the tests with the fastest download speed are represented on the top layer and those with the slowest download speeds are on the bottom layer. Points on the map shown in a shade of red indicate tests with download speeds below 25 Mbps. Similar visuals can be replicated for upload speeds to determine if other areas must be considered, as a location may meet one set of criteria and not the other.

4



Speedtest Download Speed Results in New York’s 22nd Congressional District January 2020 through June 2020

This map makes it immediately clear which areas have zero indication of speeds that meet the FCC minimum guidelines. The data shown represents real-world measurements directly from users of the local networks. No single test should ever be considered indicative of a network’s average performance, but clusters of samples that show consistently poor results provide a rapid way to identify the areas of greatest need.

Booneville

Forestport

Middleville

OneidaSullivan

Marathon

Sherburne

Hamilton

Morrisville

Deruyter

Herkimer

Oxford

GreeneWhitney

Point

Windsor

Afton

Apalachin

Cortland

Rome

Utica

Binghamton

Syracuse

T I O G A

C H E N A N G O

C O R T L A N D

M A D I S O N

O N E I D A

O S W E G O

H E R K I M E R

B R O O M E

Oriskany Falls

90

81

81

90

88

West Winfield

Pulaski

Camden

Altmar

Mexico

Amboy

Norwich

Oneida Lake

Download SpeedsBelow 25 MbpsBetween 25-300 Mbps300 Mbps and above

NY-22 District (US House)County Boundaries



Note that the test results represented in blue in the image on the previous page indicate speeds above 300 Mbps. These speeds suggest the probable availability of direct fiber connectivity. Although there are some tests above this speed within rural areas, the vast majority of high-speed results are within urban zones — evidence that the urban-rural digital divide is wide in NY-22. Ookla sees similar results in every state.

Keep in mind that we excluded tens of thousands of additional tests from this image because they did not meet Ookla’s stringent geolocation criteria. Without GPS accuracy, those results can only be used at an aggregate level, providing additional data for averages of performance levels within a given ZIP code, municipality, county or state.

When location accuracy determines which communities receive broadband investments, getting access to data at this granular level is crucial to the decision-making process.



Meaningful analytics showing locations where providers are actually delivering broadband services

ISPs need aggregated numbers to quickly assess their position relative to their competition, track performance trends across their network and understand how they may be viewed by their subscriber base. Fixed broadband networks are, by definition, built to serve a specific building or address. They will sometimes serve only a portion of a given ZIP code or other area defined by a geographic boundary.

Reporting data aggregated by ZIP code is common among network testing providers, but ZIP codes were created for a single purpose — to efficiently deliver the mail via linear routes. While an urban ZIP code may contain several neighborhoods in the same city, rural ones can encompass several small communities many miles apart from one another. ZIP code names do not reflect every community served, but are usually named for the community that hosts the postal facility.

5



The map above shows the boundary of the 13425 ZIP code, which lies within NY-22. It is southwest of Utica and the ZIP code’s largest community, Oriskany Falls, is a village in the southeastern portion of the delivery area, home to a population of roughly 700 people. Approximately 2,000 people live within the ZIP code — so over half live outside the primary population center.

The above map shows network performance tests captured on Ookla’s Speedtest platform that were conducted on Android and iOS mobile devices running our Speedtest mobile application. Instead of the location being defined only as “somewhere in the ZIP code,” we can identify the precise location of the test, within a few meters.

Download speeeds measured using the Speedtest mobile application for iOS or Android from January 1 to June 30, 2020. Location accuracy estimate filtered to 500 meters or less.

Plotting Speedtest® measurements for download speeds shows geographic pockets within the sample zip code that do not meet the FCC minimum standard.

Deansboro

CongerCorners

Knoxboro

Solsville

Augusta

Oriskany Falls

Madison

13425

12A

315

12B26

37

3

26

26

20

20

ONEIDA CO

UNTY

MADISO

N COUNTY

© 2020, Ookla, LLC

Download SpeedsBelow 25 MbpsBetween 25-300 Mbps300 Mbps and above

Zip Code Boundary

Fig. 4

Speedtest Download Speed Results in New York’s 13425 ZIP Code, January 2020 through June 2020



Clearly there are tests being conducted that exceed the FCC minimum. In fact, some of the tests conducted in Oriskany Falls exceed 300 Mbps, indicating that there is likely fiber available to some addresses. Northwest of there, in the smaller community of Augusta, Speedtest results fell well above 25 Mbps. Some tests measuring over 100 Mbps indicate that high-speed broadband is already available in the area.

Simply averaging all the tests taken in this ZIP code, Speedtest shows a median download speed of 23.30 Mbps. However, this number does not fully represent access to broadband throughout the area. There are clearly areas on the map where zero tests meet the minimum guideline of 25 Mbps download speed. Averaging speeds in rural areas with those that have the densest population obscures the communities that need help in the form of funding for better broadband.

If a policymaker were to use only data from M-Lab, they would only see average download speed of 4.5 Mbps, without the detail of how those speeds fall within the broader context of tests taken throughout the various communities in the ZIP code. This could lead to a decision to invest in areas that actually already have high-speed internet access.

Regardless of the disparate volume of tests and differences in methodologies between testing providers, the fact that Ookla is even capturing measurements at these speeds indicates that some portions of this ZIP code should not be prioritized for government subsidy.

By pinpointing where providers are actually delivering services that meet or exceed the current definition of broadband, Speedtest data can help policymakers eliminate costly data collection and analysis for areas that already have adequate service. Resources can instead focus on areas that show a clear need for broadband infrastructure investment. Customers experiencing slower speeds relative to their neighbors’ are likely experiencing isolated technical problems or are subscribing to a lower tier of service. If the level of service promised in the agreement is not being delivered, Speedtest is also a way for customers to record results and demand improved service from their ISP.

Having detailed insights into how — and precisely where — network performance differs within large geographic areas can help policymakers identify vulnerable communities and avoid unnecessary spending.


Conclusion: Bad Data Can Negatively Impact Government Funding

There are billions of dollars of federal, state and local government funding at stake, not to mention the sufficient internet connectivity of millions of constituents. Some states have set aside money to subsidize ISPs’ and utility companies’ efforts to extend their reach or add new services to underserved communities. Scores of municipalities have undertaken the construction of their own fiber networks to both connect city services and provide broadband service to homes and businesses.

There are multiple funding mechanisms that are controlled by multiple agencies at the federal level in the United States. The Connect America Fund (CAF) is administered by the FCC. The Agriculture Department administers the ReConnect program through an agency known as the Rural Utility Service, an agency that has been around since the push to bring electricity to rural America during President Franklin Delano Roosevelt’s days in the White House. The CAF and ReConnect funds are primarily directed toward improving fixed broadband access to rural communities. The FCC has also announced an upcoming program for mobile communications called the 5G Fund for Rural America to bring high-speed, low-latency wireless internet access to rural areas, with a focus on precision agriculture.

All of these programs depend upon network performance data to help drive the decision-making process. In the past, the various federal agencies tasked with collecting this data have received criticism from the public, the communications industry and from within Congress regarding how this data is used to determine eligibility for federal subsidies. Historically, when agencies have trusted bad data providers, they have made bad decisions — and current policy makers should be careful not to make the same mistakes.

While testing providers that do not leverage a vast network of local testing servers can provide limited insight into an ISP’s perceived performance in isolated instances, they paint an overall incomplete picture of whether an ISP can meet minimum national standards. To do so, a testing provider must be able to assess a network’s full throughput capacity and measure real-world speeds — which is only possible when testing to multiple local servers.


Individual providers can only be held responsible for the portion of the internet they control and for maintaining adequate bandwidth from their local network to the nearest “hub” that connects their customers to the larger internet. Policymakers should be evaluating local broadband network performance on these criteria.

Committing to spend billions of dollars helping ISPs build new infrastructure and bring broadband to underserved communities across the country is a momentous decision for legislators and regulators. While no one expects the current round of funding to get every citizen access to high-speed broadband, policymakers owe it to their constituents to make the most informed decisions possible, using the most accurate and comprehensive data to drive their decision-making processes. It would be irresponsible to rely solely on inaccurate or incomplete network performance data. Using reliable data from trusted sources can help connect the maximum number of homes and businesses. It’s critical that funding goes directly to communities in the greatest need, to bring their constituents into the information age.


Our Mission

Ookla’s mission is to make the internet better, faster and more accessible for everyone. Hundreds of millions of people worldwide use Speedtest to measure their internet connection. With over 10 million consumer-initiated tests taken daily and billions of data points gathered, Ookla® data paints a clear picture of the performance, quality and availability of virtually every network in the world.

During 2019, 250 million consumer-initiated tests were performed on fixed and mobile networks in the United States using Speedtest. During just the first nine months of 2020, over 225 million tests were conducted in the U.S. In each year, these tests were taken on over 40 million unique user devices.

About Ookla

Ookla® is the global leader in mobile and broadband network intelligence, testing applications and technology. Speedtest®, Ookla’s flagship network testing platform, collects hundreds of millions of measurements about the performance and quality of networks around the world each day. Operators, businesses and government agencies alike rely on Ookla for unparalleled and immediate information on the state of networks and online services.


Contributors

Bryan Darr, Vice President, Smart Communities, Ookla Bryan Darr is the Vice-President of Smart Communities at Ookla. His focus is on bringing Ookla’s vast network performance and coverage intelligence to the public sector and to the telecom infrastructure industry. Bryan has spent most of his professional life in the wireless industry, beginning his career in cellular as a sales representative for Cellular One of Memphis in 1985. After discovering his customers were having difficulty traveling with their phones, he founded Mosaik in 1988 (originally branded as American Roamer) and began developing consumer roaming guides. Soon, Mosaik was researching and designing mobile operator coverage maps enabling them to market the reach of their mobile networks. In June 2018, Mosaik was acquired by Ookla. Bryan has served on many industry committees during his career. He currently serves on CTIA’s Smart Cities Business & Technology Working Group.

Marc von Holzen, Vice President of Software Engineering, Ookla Marc von Holzen is the VP of Software Engineering at Ookla, where his team builds and maintains the Speedtest ecosystem, including the web, mobile and desktop apps, embedded versions for routers and customer-premises equipment, as well as the backend APIs, data pipelines and analytics applications that provide network performance and benchmarking insights to service providers across the world. He has been managing software engineering teams for the last 20 years, specializing in telecommunications and internet service provisioning, monitoring and billing. Marc holds a Masters in Computer Science from the Swiss Federal Institute of Technology, in Lausanne Switzerland.

Dr. Brian Connelly, Data Science and Analytics Lead, Ookla Brian Connelly is the Data Science and Analytics Lead at Ookla, where his team develops new metrics for mobile, broadband, and Wi-Fi networks across the globe and helps users uncover new insights from large, multi-faceted datasets. Prior to joining Ookla in 2017, he was a National Science Foundation Postdoctoral Research Fellow at the University of Washington, where he led research targeting the evolution of communication and cooperative behaviors using microbial systems, computer simulations, and mathematical models. Brian has PhDs in Computer Science and Evolutionary Biology from Michigan State University and a BS in Computer Science from Purdue University.


References

1. Report on the State of Broadband Access in New York’s 22nd Congressional District, August 3, 2020. https://brindisi.house.gov/sites/brindisi.house.gov/files/Brindisi%20Report%20on%20NY22%20Broadband.pdf

2. M-Lab, United States Dashboard (data for counties addressed in this document, Dec 30, 2019 through June 21, 2020 (Weeks 1 to 25, 2020). https://datastudio.google.com/u/0/reporting/1djtGEuqV4Qwrj26GQTN_xzp3rsMYYcmv/page/YW8NB?s=rzD5rHYkLT4

3. M-Lab Methodology for the dataset used in the report (see Fig. 1). https://datastudio.google.com/u/0/reporting/1djtGEuqV4Qwrj26GQTN_xzp3rsMYYcmv/page/s0CPB?s=rzD5rHYkLT4

Fig. 1

How are the Median Speeds Calculated?

1. Initial Test Set Selection We first selected all tests in a given date range which were conducted by people in the United States. For data presented by state, zipcode, and congressional district, this first step also limits the initial set of tests by including only tests where the tester’s IP address was geolocated within the specific geography.

-State: the field “region_code” is used to identify the state -County, Congressional District, & Zipcode: The geolocated latitude and longitude of each tests’ IP address is used to find the corresponding geography, using BigQuery’s GIS functions and public BigQuery tables for each geography provided by the Google Public Datasets Program.

The sample size counts are based on the initial test set.

2. Calculate Per-IP, Per-Day Median Values Next, we aggregate the median values “Per-Day” to limit the impact of “skew” or bias from the followin situations:

-Some testers may run many tests per day, while others may only run tests periodically -Many tests per day from very fast or very slow connections can pull the overall median up or down if not accounted for in some way

3. Calculate Median Values Over Date Range Finally, we calculate teh final median values from the “Per-IP” medians in step 2, to arrive at the final reported value over the whole selected range of days in the selected geography.


Appendix

Comparing Median Download and Upload Speeds in M-Lab’s Reported “Best” ZIP Codes for NY-22 Ookla® Speedtest® and M-Lab Data | December 30, 2019 through June 21, 2020

*Ookla uses Google services for some geolocation functions. Google’s boundary definition of ZIP code 13123 (and therefore the boundaries of those contiguous to it) differs from federal sources and we have taken care to calculate our estimates based upon our best understanding of the area considered for the median download and upload speeds reflected.

ZIP Code

13411

13101

13408

13304

13131

13354

13052

13123

13104

13904

13037

13114

13413

13493

13440

13461

13350

13903

13357

13501

Town

New Berlin

McGraw

Morrisville

Barneveld

Parish

Holland Patent

DeRuyter

North Bay*

Manilus

Binghamton

Chittenango

Mexico

New Hartford

Williamstown

Rome

Sherrill

Herkimer

Binghamton

Ilion

Utica

M-Lab

103.5

91.9

111.3

80.0

63.3

63.5

66.3

38.4

44.9

21.0

42.1

41.6

40.0

40.9

38.6

41.4

37.9

36.2

35.8

35.4

Ookla

18.5

70.9

46.0

76.0

83.0

70.0

10.6

83.9

74.8

67.9

73.5

87.1

70.9

68.8

84.8

76.4

79.4

67.6

76.4

70.4

(-82%)

(-23%)

(-59%)

(-5%)

(+31%)

(+10%)

(-84%)

(+118%)

(+66%)

(+223%)

(+75%)

(+109%)

(+77%)

(+68%)

(+120%)

(+85%)

(+110%)

(+87%)

(+113%)

(+9)

M-Lab

102.6

83.6

18.9

46.8

44.3

23.4

9.5

25.6

10.3

33.4

10.8

9.6

11.3

7.9

9.3

5.6

9.0

9.6

9.0

9.4

Ookla

3.3

11.7

11.5

11.4

11.6

17.0

1.0

10.4

11.7

11.5

11.6

11.6

11.6

9.8

11.7

11.8

11.5

11.5

11.5

11.5

(-97%)

(-86%)

(-39%)

(-76%)

(-74%)

(-27%)

(-90%)

(-59%)

(+13%)

(-66%)

(+7%)

(+20%)

(+3%)

(+24%)

(+26%)

(+110%)

(+28%)

(+20%)

(+28%)

(+22%)

Upload Speed (Mbps)Download Speed (Mbps)

make better funding decisions with accurate broadband ... - make better...case study: inaccurate...

Documents