Microsoft IT Showcase

Data insights help enhance the user experience at Microsoft Microsoft employees and vendors worldwide use Microsoft products and services to find information, attend meetings, communicate with others, and perform many other tasks. How can we ensure the best end-to-end user experience for them? The User Experience Services team in Microsoft IT addresses this question from a data-driven, holistic perspective. To enhance the user experience for our own employees, we use products and services instrumentation, Cortana   Intelligence   Suite , data science, and the insights that these approaches give us. Instead of looking through the lens of a single product, our view is the broader landscape of five high-level—often overlapping—experiences related to overall user experience across Microsoft products and services: Finding Meeting Supporting Communicating CollaboratingWe examine how to improve these five user experiences, as defined by our leadership. For example, how do we help people quickly find what they’re searching for, or give a good meeting experience? This scope requires us to be scalable, strategic, and agile. We need a sustainable model, a robust data platform, and a systematic way of gaining insights, rather than just reacting to data requests and questions from teams, executives, and business analysts companywide. For technical details of this solution—such as how we gather the data, set up the data platform, what technology and architecture we use, and how we apply data science, machine learning, and algorithms—see Microsoft IT uses analytics and data science to enhance the user experience.

Our approachWe started by defining our strategy. Looking at products, services, devices, and infrastructure, we map them to one or more of the five experiences, and identify areas of potential improvement. To do this, we get product and service telemetry like crash data and client device information. We also gauge sentiment with user feedback and satisfaction metrics from employee surveys and from Microsoft Helpdesk support data. We store data in Microsoft Azure Data Lake and analyze it with machine learning. From there, stakeholders can visualize the data in Microsoft Power BI self-service reports. Our strategy ties into the larger One Microsoft culture of focusing on shared experiences rather than only on individual products. We closely collaborate with teams and decision makers across Microsoft to capture, correlate, and share holistic data and feedback with them. To connect the dots even more, we’re always adding data sources and instrumentation. And down the road, we’ll add predictive analytics to see how we can prevent issues.The platform that we’ve built unifies data from products and services, and supports digital transformation with data-driven systems. Think of it as a single place to see the big picture—for example, Microsoft Windows, apps, devices, and network data. Although product teams have their own metrics, the data tends to be related to the product or service that they work on. We’re joining technology, data, and people for reinvention, fast insights, and data-driven decisions to make others succeed.

Enhancing products, empowering people, and optimizing operationsProducts and services instrumentation, Cortana Intelligence Suite, and the insights we get propel us—and help others—to reinvent and enhance user experiences.

Page 2 | Data insights help enhance the user experience at Microsoft

Here are some ways we’re bringing about change: Validating the user experience before Microsoft products and services are released. We

work closely with teams—for example, Microsoft Windows and Devices—to give and get feedback. Our shared goal is to prevent issues before products/services go out to Microsoft employees and vendors.

Enabling teams to be self-sufficient. We help teams get insights on their own by enabling them to quickly generate self-service reports, or adapt existing ones, with Power BI—whenever they want.

Empowering teams throughout the company to be successful by building on each other’s ideas. Rather than working in a silo, or starting from scratch, we collaborate closely with product groups. Teams build on each other’s efforts and expertise to achieve common and overlapping goals.

Providing information to product teams. To identify opportunities for enhancements, product teams want to know about issues, user sentiment, and how products and services have been adopted.

Giving senior executives visibility into what’s happening. Executives also look at this information—including adoption—and give feedback to the product teams. We help them dive deep.

Helping the support team resolve tickets and reduce costs. Because support is one of the five experiences we look at, improvements to underlying products and services can translate to fewer tickets, thereby lowering costs.

Giving insights to the support team to incorporate in their documentation and overall process. Suppose we see that people have specific issues with the meeting experience. Sharing data with the support team helps them identify what to include in Helpdesk documentation. If people search for troubleshooting information and find what’s needed to resolve the issue, it can reduce support tickets. The support team can also highlight this information on a website or in support calls.

Examples of questions and insightsTo help decision makers strategize and plan, we share insights by looking at questions like: How many devices comply with the latest security and software updates? How many users have adopted Windows 10 Anniversary Update? What issues have affected someone’s device, and what’s the cause? What steps can people take to prevent an issue from happening again? What trends in app crashes do we see, and can we correlate these with any updates? What’s the overall user sentiment we see from satisfaction surveys and sentiment analysis?These are examples of the answers we provide to help revamp the user experience. But how do we go about improving an experience? Let’s look at a couple of sample scenarios.

Scenario 1: Improving the finding experience The goal of the finding experience is to help people find what they need on channels like Office 365, internal company portals, and Cortana. We study data from these channels because these are where people search. We also distribute user satisfaction surveys. From the survey responses, we calculate net satisfaction (NSAT) by doing sentiment analysis. Plus, we study search-success metrics of internal searches and work with product teams to understand the data that they get. We can use their learnings and even complement them by adding our analytics.

Scenario 2: Improving the meeting experienceAnalytics on the meeting experience should reveal insights on what’s needed to make an employee productive during a meeting, either in person or remotely. Actions that are taken on these insights

microsoft.com/itshowcase June 2017

Page 3 | Data insights help enhance the user experience at Microsoft

directly affect the person’s satisfaction level with the app that’s used for meetings and with the meeting experience itself. If people use Microsoft Skype for Business to attend a meeting and they have an issue, they might give feedback about the issue in a survey. The feedback could be about the meeting experience, and it could also be about the support experience. So, getting insights can affect and improve several areas. Here’s an example of what we do in this scenario: Like scenario 1, in our user satisfaction survey, we look at user sentiment and NSAT rates via

sentiment analysis. We see whether Skype for Business users created Helpdesk tickets during the time when we

received survey input or detected issues. And, if they did, can we correlate the survey input and Helpdesk tickets with a known spike in app crashes, for example? Are there other correlations that we can make?

We look at the resolution rate of the crashes and identify which executable program crashed. We see if a subset of users experienced an issue in a specific period or event, like during a

Windows deployment. A variety of data points and perspectives give us rich insights into how the pieces fit together.

Business benefitsOur overall scope of user experience is for all devices, Microsoft employees, and vendors. Here are some of the benefits that we get from this solution: It enables us to measure key performance indicators like reliability, productivity, adoption rate,

and compliance rate, and gives us insights to answer questions like: What’s the average, minimum, and maximum companywide installation time, and how many

devices are in each category? How much did installation time change compared to earlier releases? What is the productivity hit? We then drill down into the devices that took more time, if there’s

data available. How long does it take for updates to be deployed? How many devices are running a specific update? Are we hitting our update deployment targets? What’s the active device count? How do inactive devices affect adoption?

We’ve moved from responding to impromptu data requests or questions from teams, senior executives, business analysts, and other stakeholders to a systematic, well-defined strategy of gaining insights.

We’ve enabled stakeholders to do self-service reporting at any time.

microsoft.com/itshowcase June 2017

Page 4 | Data insights help enhance the user experience at Microsoft

Technologies for creating this solutionWe use Cortana Intelligence Suite and other Microsoft technologies, and then apply machine learning and algorithms to the data. Here’s a summary: Azure Data Factory. We use Azure Data Factory pipelines to ingest, prepare, process, and

transform terabytes of structured and unstructured data and to move data into Azure Data Lake. Azure Data Lake. We use Azure Data Lake to store the data. Data Lake makes it easy for

developers, data scientists, and analysts to store data, and to process and analyze across platforms and languages.

SQL Server Analysis Services. We use SQL Server Analysis Services to create multidimensional cubes, so that business users who create their own reports can slice and dice the data in a variety of ways.

Machine learning. We use machine learning to get insights from our verbatim survey analytics. To answer business-stakeholder questions, data scientists take raw data from Data Lake and do multiple levels of analysis. For verbatim survey response analytics, we use key phrase extraction, deep semantic similarity, and sentiment analysis.

Microsoft Power BI. Stakeholders use Power BI to visualize the data in self-service reports.

Overcoming implementation challengesWe’re seeing lots of success in improving user experiences across the company, reducing support costs, and tracking adoption. But, like with any effort to transform and evolve processes, there can be hurdles. Some of the business-related and technical challenges that we’re overcoming include: Ingesting raw data from multiple sources and correlating it to provide insights. It’s

challenging to take the data points that Windows, Skype for Business, and other products and services give us and correlating them with a common identifier. But we’re using crash data and identifying algorithms that can provide more correlations.

Eliminating silos. Although we’ve made very significant gains, silos still exist in places. This is partly because it takes time to get access to some data sets. But we’re always adding companywide data sources.

Best practicesSo far, using products and services instrumentation to improve user experiences has given us some valuable guidelines: Enable telemetry within products like Office. Microsoft technologies for big data are key to

our success. We recommend configuring telemetry and using Cortana Intelligence Suite. It’s vital to get data on crashes, events, and device vulnerabilities because these affect performance, support costs, device management, and security.

Have a methodical, clearly defined strategy. A central part of this strategy is—on an ongoing basis—understanding the business context and the questions that business stakeholders want to explore. And, to chart our course, we always align with the five high-level experiences.

Ensure variety. A good team needs diverse skill sets. Different perspectives and expertise yields deeper, richer insights. Typical skills needed include machine learning, Data Lake, R and Python, and U-SQL.

Fail fast/learn fast. We’re always iterating based on what has and hasn’t worked well. In some cases, we started with a certain process or workflow, and then changed course.

Reduce scope. Because this approach is new, we don’t want to overwhelm teams and executives by giving them thousands of insights at one time. Instead, we focus on delivering smaller, actionable, valuable pieces—like sentiment analysis—that improve an experience.

Use Power BI for self-service reporting. Power BI makes data visualization and reporting capabilities easy and widely available. Anyone can generate a report as needed without having to wait on other people.

microsoft.com/itshowcase June 2017

Page 5 | Data insights help enhance the user experience at Microsoft

Our roadmapThere are lots of opportunities to bring in a variety of data and make more correlations that help decision makers. Here’s where we are and where we want to go: Our overall platform is in production. We have machine learning, verbatim analytics, and

sentiment analytics components in early evaluation mode, and we’re targeting these for production later this year.

We keep adding data sources and providing details and insights that decision makers need. We want to keep making the insights richer. For example, to further enhance the finding experience, we want to do more analytics for Office 365, Cortana, our internal Software Center, and Intune Company Portal.

We have Windows and network instrumentation, and hope to get Office and Skype for Business instrumentation.

We’ll be adding predictive and prescriptive analytics. We’d like to add thresholds of issues and anomalies and have our system send notifications to

support, service engineering, and service managers. They would be able to track anomalies from our backlog on a dashboard.

We want to add or expand use of Cognitive Services, Bot Framework, and Cortana.

Adding predictive and prescriptive analyticsHere are some of the ways we want to use predictive and prescriptive analytics down the road: Help with capacity planning. Predict whether a certain issue will happen. Or for existing issues, predict if it will become more

critical. Forecast how many support tickets are expected in an upcoming month, accounting for factors like

holidays or major updates. If we see issues that will affect the number of tickets and issues that people contact the Helpdesk about, we can alert the support team so they can plan accordingly.

Look at mobile app usage to see how many people install/uninstall, share usage rates with the product teams, and offer suggestions. Building an engine for prescriptive analytics scenarios like this is in the pipeline.

Leading by exampleAligned with the One Microsoft mindset and strategy of enabling digital transformation, we’re leading by example. We’re bringing instrumentation from products and services to enhance the user experience across Microsoft products and services. Microsoft is a multifaceted, vast space. In spite of—and because of—that, we’re joining systems, data, and people for reinvention, fast insights, and data-driven decisions to help others succeed.

For more informationMicrosoft ITmicrosoft.com/ITShowcaseMicrosoft IT uses analytics and data science to enhance the user experienceCortana Intelligence SuiteAzure Data LakeIT Showcase data analytics content© 2017 Microsoft Corporation. This document is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

microsoft.com/itshowcase June 2017