real time data strategy and architecture

Real Time Data Strategy And Architecture

Alan McSweeney

http://ie.linkedin.com/in/alanmcsweeney



Real Time Data Collection Strategy And Architecture Approach

• These notes are concerned with describing a generalised approach to defining a strategy for collecting (near or actual) real time, high volume data

• This is data is generated by sensors that is transmitted to a central location for processing, reporting, analysis and ultimately action

• Sensors can be regarded as logical or physical sources of streams of measurement data

• (Near) real time data can be termed Telemetry

March 8, 2016 2

Real Time Data Collection Strategy And Architecture Approach

• Approach adopted from TMForum Resource Domain Frameworx eTOM (Enhanced Telecoms Operating Model) Business Process Framework

• Approach has been generalised for real-time data and telemetry

March 8, 2016 3

Real Time Data Collection Strategy And Architecture

• Collect data from range of data sources across the organisation’s (internal and external) operating landscape

• Approach can be applied to collection of measurement data from multiple sources and of multiple types through “sensors”: −Different entities interacting with the organisation and the

gathering of data on different actions and events

• Approach can be applied to areas such as Telemetry, Big Data, Smart Metering and Internet of Things implementations and operations

March 8, 2016 4

Real Time Data Collection Strategy And Architecture

March 8, 2016 5

Real Time Data Strategy

Telemetry Strategy Big Data

Strategy

Internet Of Things

Strategy SmartX Strategy

Digital Strategy

Why Have A Real Time Data Collection Strategy And Architecture?

• Real time situational data gives rise to situational awareness and understanding which in turn presents opportunities for effective and rapid situational decisions −What is happening – usage, performance

−What can be improved

−What are the optimisation and productivity opportunities

• Real time situational data enables greater situational visibility which means increased operational intelligence

March 8, 2016 6

Organisation Operating Landscape

March 8, 2016 7

Organisation Operating Landscape

• The operating landscape of the organisation defines the number and type of interactions outside the organisation

• This operating landscape affects the decision on what and how to measure

• Not all interactions with all entities are measured

• Need to be realistic about what can be collected and processed

• Need to understand the need for sensors to collect data

March 8, 2016 8

Data Sensors – Combinations Of Options

• Sensor Type – Physical sensors are actual units such as RTUs (Remote Telemetry Units) that measure and generate data while Logical sensors are representations of data sources

• Sensor Ownership – Direct sensors are those installed and maintained by the collecting organisation while Indirect sensors are installed by a third-party

March 8, 2016 9

Sensor Type

Logical Physical

Sensor Ownership

Direct • Web Site and App Activity

and Usage Data • Internet of Things Devices • Remote Real Time Unit • Smart Devices

Indirect • Third-Party Web Site and

App Activity and Usage Data

• Third-Party Devices

Measurement Data Sensors

March 8, 2016 10

Measurement Data Sensors

• These can gather data for different measures using multiple measurement techniques

• These can be regarded as collectors of any data – usage, activity, performance, throughput

• Each measurement type will have a unit or dimension

• Logical representation of data collectors

• Need to decide on what can be measured, what to measure and how to measure it

March 8, 2016 11

Measurement Data Sensors – Decide On What And How To Measure

March 8, 2016 12

Real Time Data Architecture Complexity

March 8, 2016 13

Real Time Data Architectures Tend To Focus On The Simplicity

Of A Possible Real Time Data Collection Architecture …

… And Ignore The Complexity And Difficulties Of Sensor Installation, Operation, Maintenance, Errors,

Rework, Logistics, Service Management, Data Volumes


• Structured approach is intended to ensure that complexity is understood and can be appropriately addressed at an early stage before problems become to embedded to be solved

March 8, 2016 14


March 8, 2016 15

Don’t Let The Ignored But Knowable And Addressable Complexity Sink Your Real

Time Data Programme/Initiative

Real Time Data Strategy And Architecture Issues

• What business benefits will a real time data strategy yield?

• How can the benefits be realised?

• What is the business case for investment in a real time data strategy?

• What infrastructure, communications/connectivity, data and application architectures are needed to support a real time data strategy?

• What integration is required with existing applications?

• What skills, capabilities and changes does the organisation need to adopt and exploit a real time data strategy?

• How is the real time data strategy managed and serviced?

• What solution and service providers and tools/platforms and sourcing strategy should be selected?

• What are the privacy and security issues and requirements?

March 8, 2016 16

Evolution Of Real Time Data Architecture

March 8, 2016 17

Create Awareness for Real Time Data

Investment

Undefined/unarticulated/uncertain real time data strategy. Real time data processes are ad hoc, focussed on individual solution and

outcomes vary widely.

Cra

wl

W

alk

Ru

n

F

ly

Building Real Time Data Investment

Foundation

Implement investment controls and develop key foundational capabilities

Developing Complete Real

Time Data Portfolio

Comprehensive selection and control processes with benefit and risk criteria

linked to strategy requirements

Improving Real Time Data Processes

Process evaluation techniques focus on improvement of performance and

management

Leveraging for Strategic Outcomes

Real time data management, reporting and analysis techniques are deployed for strategic mission/business outcomes

Admitting

There must be a better way

Communicating

Establishing and communicating business case

Governing

Making and implementing effective investment

decisions

Managing

Processes, mechanisms and metrics

Optimising

Sense and respond

Developing A Real Time Data Strategy – Generalised High Level Steps

March 8, 2016 18

Real Time Strategy And

Planning

Real Time Capability Delivery

Real Time Development

And Retirement

Real Time Management

and Operations

Support And Readiness

Real Time Provisioning

Real Time Data

Collection And

Distribution

Real Time Problem

Management

Real Time Performance Management

Workforce Management

Real Time Data

Aggregation and

Reporting

4 1 2 3 5

10 9 8 7 6


• Comprehensive set of steps from definition of what is required from real time data to commissioning of real time data collection facilities to effective use of collected data

• Not all steps are relevant to all real time data initiatives − For example, if installation, commissioning, operation and maintenance

of physical sensors is not applicable then related steps will not be required

• Represents an idealised organisation and process breakdown across entire spectrum of real time data from strategy to workforce management to device installation, data collection and data usage and actioning

• Provides a basis for developing a work breakdown and an implementation plan

• Represents a comprehensive structure that can be adapt to meet its long-term real time data needs

March 8, 2016 19

Developing A Real Time Data Strategy – Generalised High Level Steps 1 – 2

Step Scope 1. Real Time Strategy And Planning

• Develop real time data strategy, policies and plans for the organisation governed by long-term business, market, product and service needs directions

• Perform research and analysis to determine real time targets and strategies to reach the defined targets

• Understand the real time data capabilities of the existing infrastructure • Build real time data model • Define approaches to real time data quality and real time data governance • Define and agree the infrastructure needs based on market, product and service strategies of the

organisation • Manage the capabilities of the suppliers and partners to develop and deliver new real time data

capabilities and detail the approach to the deployment of new and enhanced infrastructure • Define the real time data implementation standards sought, key real time data capabilities

required, real time data support levels and approaches required, real time data design elements to be developed, and real time data cost parameters and targets.

• Define the policies relating to technical real time data sensors and their implementation

2. Real Time Capability Delivery

• Ensure that network, application and computing real time data facilities are deployed • Provide the physical real time data capabilities necessary for the ongoing operations and long-

term well-being of the organisation and ensure the basis on which all real time data capabilities and services will be constructed

• Plan real time data resource supply logistics • Plan real time data sensor installation • Verify the real time data sensor installation • Handover real time data capabilities to operations

March 8, 2016 20


Step Scope 3. Real Time Development And Retirement

• Develop new or enhance existing technologies and associated real time data types applying the capability definition or requirements defined by the Real Time Strategy And Planning step

• Decide on acquisition of real time data resources from third parties • Retire or remove technology and associated real time data resource types that are no longer

needed by the organisation

4. Real Time Management and Operations Support And Readiness

• Manage the types of real time data resources and ensure that necessary application, computing and network facilities are available and ready to implement and manage resource instances

5. Workforce Management

• Manage the direct and indirect personnel who perform work assignments or work orders relating to real time data resources installation, commissioning and maintenance as well as managing the actual activity being performed

• Report and monitor activities • Establish, manage and allocate work assignments to direct and indirect personnel • Establish and manage priority and urgent assignment capabilities to allow for modification of

work assignments as required to meet urgent and high priority conditions

6. Real Time Provisioning • Allocate, install, configure, activate and test of real time data resources to meet the defined requirements

• Resolve real time resource capacity issues, availability issues or failure conditions • Configure and activate physical and/or logical real time resources • Update of real time resource register database to reflect that the specific real time resource has

been allocated, modified or recovered

March 8, 2016 21


Step Scope 7. Real Time Data Collection And Distribution

• Collect and distribute management information and real time data between data sources and service instances and other organisation functions and processes

• Work with the real time data resource and service instances to collect usage, network and technology events and other management information for distribution to other processes within the organisation

• Handle and process command, query and other management information for distribution to resource and service instances

• Process the data and management information through actions such as filtering, aggregation, formatting, transformation and correlation of the information before presentation to other processes, real time data instances or service instances

• Perform usage reporting, fault and performance analysis, service quality management analysis, resource performance analysis of resources and services

8. Real Time Problem Management

• Manage real time data resource problems including security events • Detect, analyse, manage and report on resource alarm event notifications • Initiate and manage real time data resource problems reports • Perform real time data resource problem localization analysis and resolve problems • Reporting progress on resource trouble reports to other processes • Assign and track real time data resource problem testing and resolution activities • Managing real time data resource problem urgent conditions

March 8, 2016 22


Step Scope 9. Real Time Performance Management

• Manage, track, monitor, analyse and report on the performance of real time data resources • Identify real time data resource performance disruptions or a service performance disruptions

10. Real Time Data Aggregation and Reporting

• Manage real time data resource events by correlating and formatting them into a usable format • Report of real time data resource data

March 8, 2016 23

Sample Expansion – Step 1 – Real Time Strategy And Planning – Activities 1.1 – 1.7

March 8, 2016 24

Step Scope 1.1 Gather And Analyse Real Time Data Information

Research and analyse customer, technology, competitor and marketing information to identify new real time data requirements and industry real time data capabilities and availability.

1.2 Manage Real Time Data Research

Manage internally driven research investigations and activities which are used to provide detailed technical assessment or investigation of new and emerging real time data capabilities.

1.3 Establish Real Time Data Strategy And Architecture

Establish the real time data strategies based on market trends, future needs, technical capabilities and addressing shortcomings in existing real time data support.

1.4 Define Real Time Data Support Strategies

Define the principles, policies and performance standards for the operational organisation providing real time data support.

1.5 Produce Real Time Data Business Plans

• Develop and deliver annual and multi-year real time data plans in support of services, products and offers that include volume forecasts, negotiation for required levels of resources and budgets.

• Obtain real time data development and management as well as supply chain commitment and executive approval for the plans.

• Identify the impacts that new or modified real time data infrastructure will cause on the installed infrastructure and workforce and establish the functions and benefits that new or modified real time data will provide to users.

1.6 Develop Real Time Data Partnership Requirements

Identify the requirements for real time data capabilities to be sourced from partners or suppliers, and any real time data capabilities to be delivered internally to the organisation.

1.7 Gain Enterprise Commitment To Real Time Data Plans

Obtain organisation commitment to the resource strategy and business plans including all aspects of identification of stakeholders and negotiation to gain stakeholder approval.

Sample Expansion – Step 1 - Real Time Strategy And Planning – Activity 1.5 – Tasks 1.5.1 – 1.5.5

March 8, 2016 25

Step Scope 1.5.1 Develop And Deliver Annual/Multi Year Real Time Data Business Plans

Develop and deliver annual/multi year real time data business plans focus on developing and delivering annual and multi-year real time data in support of services, products and offers that include volume forecasts, negotiation for required levels of resources and budgets, gaining real time data development and management as well as supply chain commitment and executive approval for the plans.

1.5.2 Forecast High Level Real Time Data Demand And Capture New Opportunities

Forecast real time data demand and capture new opportunities processes ensures that budgets are assigned which allow the organisation to implement the real time data capabilities and capacity necessary for the future needs of their customers and potential customers.

1.5.3 Assess Impact Of Real Time Data Business Plans

Asses impact of real time data business plan processes assess the impacts that new or modified real time data infrastructure will cause on the installed infrastructure and workforce, and establish the functions and benefits that new or modified real time data will provide to users

1.5.4 Identify Timetables For New Real Time Data Capability Introduction

Identify timetables for new real time data capability introduction

1.5.5 Identify Logistics For New Real Time Data Capability Introduction

Identify logistics for new real time data capability introduction

Real Time Data Quality And Data Governance

• Real-time data is inherently: − High Volume – lots of sensors generating lots of data − Noisy – lots of variation, statistical noise, inaccuracies, sensor drift, errors,

incorrect calibration − Changing – data landscape subject to substantial change along the dimensions

of data sources, volumes, types − Inconsistent – different sensor types measuring different values with different

units of measure and at different intervals − Heterogeneous – very mixed data sources − Non-standardised – multiple, emerging, overlapping standards and

approaches

• Sophisticated approach to data quality and data governance must be embedded in any real-time architecture

• Traditional approach of data collection storage and analysis may need to change to handle data volumes and quality – data filtering, quality, summarisation and transformation component

March 8, 2016 26

Real Time Data Principles

March 8, 2016 27

To manage and utilise real time information as a strategic asset

To implement processes, policies, infrastructure and solutions to govern, protect, maintain and use real time information

To make relevant and correct real time information available in all business processes and IT systems for the right people in the right context at the right time

with the appropriate security and with the right quality

To exploit real time information in business decisions, processes and relations

Real Time Data And Data Governance And Data Quality

• Data Quality - measure, assess, improve, and ensure the fitness of data for use

• Data Governance - authority and control over the management of data assets

March 8, 2016 28

Real Time Data Governance

• Core function of real time data management

• Interacts with and influences each of the surrounding ten data management functions

• Data governance is the exercise of authority and control (planning, monitoring, and enforcement) over the management of data assets

• Data governance function guides how all other data management functions are performed

• High-level, executive data stewardship

• Data governance is not the same thing as IT governance

• Data governance is focused exclusively on the management of data assets

March 8, 2016 29

Real Time Data Governance – Definition and Goals

• Definition − The exercise of authority and control (planning, monitoring, and

enforcement) over the management of data assets

• Goals − To define, approve, and communicate data strategies, policies,

standards, architecture, procedures, and metrics

− To track and enforce regulatory compliance and conformance to data policies, standards, architecture, and procedures

− To sponsor, track, and oversee the delivery of data management projects and services

− To manage and resolve data related issues

− To understand and promote the value of data assets

March 8, 2016 30

Real Time Data Governance Structure

March 8, 2016 31

Real Time Data Governance Framework

Real Time Data Architecture to Implement

Data Governance

Real Time Data Infrastructure to Implement Data

Architecture

Real Time Data Operations to Manage

Data Infrastructure

Real Time Data Governance Activities

March 8, 2016 32


Real Time Data Management Planning

Understand Strategic Enterprise Real Time Data Needs

Develop and Maintain the Real Time Data Strategy

Establish Real Time Data Professional Roles and Organisations

Identify and Appoint Real Time Data Stewards

Establish Real Time Data Governance and Stewardship Organisations

Develop and Approve Real Time Data Policies, Standards, and Procedures

Review and Approve Real Time Data Architecture

Plan and Sponsor Real Time Data Management Projects and Services

Estimate Real Time Data Asset Value and Associated Costs

Real Time Data Management Control

Supervise Real Time Data Professional Organisations and Staff

Coordinate Real Time Data Governance Activities

Manage and Resolve Real Time Data Related Issues

Monitor and Ensure Regulatory Compliance

Monitor and Enforce Conformance withReal Time Data Policies, Standards and Architecture

Oversee Real Time Data Management Projects and Services

Communicate and Promote the Value of Real Time Data Assets

Real Time Data Governance Inputs And Outputs

March 8, 2016 33

•Business Goals •Business Strategies •IT Objectives •IT Strategies •Data Needs •Data Issues •Regulatory Requirements

Inputs

•Business Executives •IT Executives •Data Stewards •Regulatory Bodies

Suppliers

•Intranet Website •E-Mail •Metadata Tools •Metadata Repository •Issue Management Tools •Data Governance KPI •Dashboard

Tools

•Executive Data Stewards •Coordinating Data Stewards •Business Data Stewards •Data Professionals •DM Executive •CIO

Participants

•Data Policies •Data Standards •Resolved Issues •Data Management Projects and Services •Quality Data and Information •Recognised Data Value

Primary Deliverables

•Data Producers •Knowledge Workers •Managers and Executives •Data Professionals •Customers

Consumers

•Data Value •Data Management Cost •Achievement of Objectives •# of Decisions Made •Steward Representation / Coverage •Data Professional Headcount •Data Management Process Maturity

Metrics


Real Time Data Quality Management

• Critical support process in organisational change management • Data quality is synonymous with information quality since poor

data quality results in inaccurate information and poor business performance

• Data cleansing may result in short-term and costly improvements that do not address the root causes of data defects

• More rigorous data quality program is necessary to provide an economic solution to improved data quality and integrity

• Institutionalising and operationalising processes for data quality oversight, management, and improvement hinges on identifying the business needs for quality data and determining the best ways to measure, monitor, control, and report on the quality of data

• Continuous process for defining the parameters for specifying acceptable levels of data quality to meet business needs, and for ensuring that data quality meets these levels

March 8, 2016 34

Real Time Data Quality Management – Definition and Goals

• Definition − Planning, implementation, and control activities that apply quality

management techniques to measure, assess, improve, and ensure the fitness of data for use

• Goals − To measurably improve the quality of data in relation to defined

business expectations

− To define requirements and specifications for integrating data quality control into the system development lifecycle

− To provide defined processes for measuring, monitoring, and reporting conformance to acceptable levels of data quality

March 8, 2016 35

Real Time Data Quality Management Inputs And Outputs

March 8, 2016 36

•Business Requirements •Data Requirements •Data Quality Expectations •Data Policies and Standards •Business metadata •Technical metadata •Data Sources and Data Stores

Inputs

•External Sources •Regulatory Bodies •Business Subject Matter Experts •Information Consumers •Data Producers •Data Architects •Data Modelers

Suppliers

•Data Profiling Tools •Statistical Analysis Tools •Data Cleansing Tools •Data Integration Tools •Issue and Event Management Tools

Tools

•Data Quality Analysts •Data Analysts •Database Administrators •Data Stewards •Other Data Professionals •DRM Director •Data Stewardship Council

Participants

•Improved Quality Data •Data Management •Operational Analysis •Data Profiles •Data Quality Certification Reports •Data Quality Service Level Agreements

Primary Deliverables

•Data Value Statistics •Errors / Requirement Violations •Conformance to Expectations •Conformance to Service Levels

Metrics

Real Time Data Quality

Management •Data Stewards •Data Professionals •Other IT Professionals •Knowledge Workers •Managers and Executives Customers

Consumers

Real Time Data Quality Plan Definition Activities

March 8, 2016 37

Real Time Data Quality Plan Definition

1. Develop and Promote Data Quality Awareness

2. Define Data Quality Requirements

3. Profile, Analyse and Assess Data Quality

4. Define Data Quality Metrics

5. Define Data Quality Business Rules 6. Test and Validate Data Quality

Requirements

7. Set and Evaluate Data Quality Service Levels

8. Continuously Measure and Monitor Data Quality

9. Manage Data Quality Issues 10. Clean and Correct Data Quality

Defects

11. Design and Implement Operational Data Quality Management Procedures

12. Monitor Operational Data Quality Management Procedures and

Performance


• 10 high level steps with activities and tasks

• Over 180 detailed tasks for a complete view of work required

• Comprehensive set of steps from definition of what is required from real time data to commissioning of real time data collection facilities to effective use of collected data

March 8, 2016 38

Real Time Data Strategy and

Implementation

1 Real Time Data Strategy And Planning

Activities

Tasks

2 Real Time Data

Capability Delivery

3 Real Time Data

Development And

Retirement

4 Real Time Data

Management and

Operations Support And

Readiness

5 Workforce Management

6 Real Time Data

Provisioning

7 Real Time Data

Collection And

Distribution

8 Real Time Data Trouble Management

9 Real Time Data

Performance Management

10 Real Time Data

Aggregation and

Reporting

Real Time Data Strategy and Implementation – Organisation, Function And Process Structure – Steps 1-10

March 8, 2016 39

Real Time Data Strategy and

Implementation

1 Real Time Data Strategy And

Planning

1.1 Gather And Analyse Real Time Data Information


1.3 Establish Real Time Data

Strategy And Architecture

1.4 Define Real Time Data

Support Strategies

1.5 Produce Real Time Data

Business Plans

1.6 Develop Real Time Data

Partnership Requirements

1.7 Gain Enterprise

Commitment To Real Time Data

Plans

2 Real Time Data Capability Delivery

2.1 Map And Analyse Real Time

Data Requirements

2.2 Capture Real Time Data Capability Shortfalls

2.3 Gain Real Time Data Capability

Investment Approval

2.4 Design Real Time Data

Capabilities

2.5 Enable Real Time Data

Support And Operations

2.6 Manage Real Time Data

Capability Delivery

2.7 Manage Handover To Real

Time Data Operations

3 Real Time Data Development And

Retirement

3.1 Gather And Analyse New Real Time Data Ideas

3.2 Assess Performance Of

Existing Real Time Data

3.3 Develop New Real Time Data

Business Proposal

3.4 Develop Detailed Real

Time Data Specifications


Development


Deployment

3.7 Manage Real Time Data Exit

4 Real Time Data Management and

Operations Support And

Readiness


Provisioning


Performance Management

4.3 Support Real Time Data Trouble

Management


Collection And Distribution

4.5 Manage Real Time Data Inventory

4.6 Manage Logistics


5.1 Manage Schedules and Appointments

5.2 Plan and Forecast


5.3 Administer and Configure


5.4 Report Workforce

Management

5.5 Manage Work Order Lifecycle

6 Real Time Data Provisioning

6.1 Allocate And Install Real Time

Data

6.2 Configure And Activate Real Time

Data

6.3 Test Real Time Data

6.4 Track And Manage Real Time Data Provisioning

6.5 Report Real Time Data

Provisioning

6.6 Close Real Time Data Order

6.7 Issue Real Time Data Orders

6.8 Recover Real Time Data

7 Real Time Data Collection And

Distribution

7.1 Collect Management

Information And Data

7.2 Process Management


7.3 Distribute Management


7.4 Audit Management And

Security Data Collection And

Distribution

8 Real Time Data Trouble

Management

8.1 Survey And Analyse Real Time

Data Trouble

8.2 Localise Real Time Data Trouble

8.3 Correct And Resolve Real Time

Data Trouble

8.4 Track And Manage Real Time

Data Trouble

8.5 Report Real Time Data Trouble

8.6 Close Real Time Data Trouble

Report Flow

8.7 Create Real Time Data Trouble

Report

9 Real Time Data Performance Management

9.1 Monitor Real Time Data

Performance

9.2 Analyse Real Time Data

Performance

9.3 Control Real Time Data

Performance


Performance

9.5 Create Real Time Data

Performance Degradation

Report

9.6 Track And Manage Real Time Data Performance

Resolution

9.7 Close Real Time Data

Performance Degradation

Report

10 Real Time Data Aggregation and

Reporting

10.1 Aggregate Real Time Data

Records

10.2 Report Real Time Data Records

Step 1 – Real Time Data Strategy And Planning – Processes And Functions Details

March 8, 2016 40

1 Real Time Data Strategy And

Planning


1.1.1 Gather Real Time Data

Information

1.1.2 Analyse New Real Time Data Requirements

1.1.3 Analyse To Develop

New/Enhance Real Time Data

Requirements


1.2.1 Manage Real Time Data Research

Investigations

1.2.2 Manage Administration Of

Real Time Data Research

1.2.3 Define Real Time Data Research

Assessment Methodologies


1.3.1 Establish Real Time Data Strategy

1.3.2 Develop Real Time Data Strategy

1.3.3 Establish Real Time Data Delivery

Goals

1.3.4 Establish Real Time Data

Implementation Policies

1.4 Define Real Time Data Support

Strategies

1.4.1 Define Real Time Data Support

Principles


Policies


Performance Standards

1.5 Produce Real Time Data Business

Plans

1.5.1 Develop And Deliver

Annual/Multi Year Real Time Data Business Plans

1.5.2 Forecast High Level Real Time

Data Demand And Capture New Opportunities

1.5.3 Assess Impact Of Real Time Data

Business Plans

1.5.4 Identify Timetables For New

Real Time Data Capability

Introduction

1.5.5 Identify Logistics For New

Real Time Data Capability

Introduction

1.6 Develop Real Time Data

Partnership Requirements

1.6.1 Identify The Requirements For

Real Time Data Capabilities

1.6.2 Recommend Real Time Data

Partnership

1.6.3 Determine Extent Of Real Time

Data Capabilities Sourcing

1.7 Gain Enterprise Commitment To Real Time Data

Plans

1.7.1 Identify Stakeholders To Real Time Data

Strategy And Real Time Data Plans

1.7.2 Gain Real Time Data Strategy And

Real Time Data Plans Stakeholders

Approval

1.7.3 Gain Enterprise

Commitment To Real Time Data

Strategy And Real Time Data Plans

Step 2 - Real Time Data Capability Delivery – Processes And Functions Details

March 8, 2016 41

2 Real Time Data Capability Delivery

2.1 Map And Analyse Real Time

Data Requirements

2.1.1 Capture Real Time Data Demand And Performance

Requirements

2.1.2 Agree Real Time Data

Infrastructure Requirements

2.2 Capture Real Time Data Capability

Shortfalls

2.2.1 Capture Real Time Data Capacity

Shortfalls

2.2.2 Capture Real Time Data

Performance Shortfalls

2.2.3 Capture Real Time Data

Operational Support Shortfalls

2.3 Gain Real Time Data Capability

Investment Approval

2.3.1 Develop Real Time Data Capability

Investment Proposals

2.3.2 Approve Real Time Data Capability

Investment

2.4 Design Real Time Data Capabilities

2.4.1 Define Real Time Data Capability

Requirements

2.4.2 Specify Real Time Data Capability

Infrastructure

2.4.3 Select Real Time Data Capability

At Other Parties

2.5 Enable Real Time Data Support And

Operations

2.5.1 Design Real Time Data

Operational Support Process

Improvements

2.5.2 Identify Real Time Data Support Groups, Skills And

Training

2.5.3 Identify Real Time Data Support

Requirements

2.6 Manage Real Time Data Capability

Delivery

2.6.1 Co-Ordinate Real Time Data

Capability Delivery

2.6.2 Ensure Real Time Data Capability

Quality

2.6.3 Manage Commissioning Of

New Real Time Data Infrastructure

2.6.4 Establish Real Time Data Capability

Sourcing

2.7 Manage Handover To Real

Time Data Operations

2.7.1 Co-Ordinate Real Time Data

Operational Handover

2.7.2 Validate Real Time Data

Infrastructure Design

2.7.3 Ensure Real Time Data Handover

Support

Step 3 - Real Time Data Development And Retirement – Processes And Functions Details

March 8, 2016 42

3 Real Time Data Development And

Retirement

3.1 Gather And Analyse New Real Time Data Ideas

3.1.1 Gather Real Time Data

Information

3.1.2 Analyse Real Time Data Classes

3.1.3 Develop Real Time Data Classes

3.2 Assess Performance Of

Existing Real Time Data

3.3 Develop New Real Time Data

Business Proposal

3.3.1 Develop Real Time Data Business

Proposal

3.3.2 Gain Real Time Data Business

Proposal Approval

3.4 Develop Detailed Real Time Data Specifications

3.4.1 Develop Detailed Real Time

Data Technical Specifications


Data Support Specifications

3.4.3 Develop Detailed Real Time Data Operational

Specifications


Data Manuals


Development

3.5.1 Identify Required Processes And Procedures For

Real Time Data

3.5.2 Develop Required Processes And Procedures For

Real Time Data

3.5.3 Develop Service And Operational

Agreements For Real Time Data

3.5.4 Gain Service And Operational

Agreements Approval For Real

Time Data

3.5.5 Produce Supporting

Documentation And Training Packages

For Real Time Data


Deployment

3.6.1 Manage Real Time Data Process

And Procedure Implementation

3.6.2 Manage Real Time Data

Operational Staff Training

3.6.3 Develop Real Time Data

Supplier/Partner Operational Support


Acceptance Testing

3.7 Manage Real Time Data Exit

3.7.1 Identify Unviable Real Time

Data

3.7.2 Identify Impacted Real Time

Data Customers

3.7.3 Develop Real Time Data Transition

Strategies

3.7.4 Manage Real Time Data Exit

Process

Step 4 - Real Time Data Management and Operations Support And Readiness – Processes And Functions Details

March 8, 2016 43

4 Real Time Data Management and Operations Support And

Readiness

4.1 Enable Real Time Data Provisioning

4.1.1 Plan And Forecast Real Time Data Infrastructure

Requirements And Manage Capacity Planning

4.1.2 Establish, Manage, And Develop Organisetion, Tools And

Processes

4.1.3 Develop And Implement Capacity And Operational Rules

And Procedures

4.1.4 Perform Acceptance Test And Address And Monitor The

Change

4.1.5 Track And Supervise The Rollout Of New And/Or Modified Infrastructure

4.1.6 Monitor, Report And Release Mgmt. Of Real Time

Data Infrastructure And Capacity Utilisation

4.1.7 Optimise Existing Real Time Data Infrastructure

Utilisation

4.1.8 Track, Monitor And Report Real Time Data Provisioning

4.1.9 Update Inventory Record

4.2 Enable Real Time Data Performance Management

4.2.1 Monitor And Manage Regulatory Issues

4.2.2 Establish And Maintain Performance Threshold

Standards

4.2.3 Undertake Performance Trend Analysis

4.2.4 Monitor And Analyse Real Time Data Performance Reports,

And Identify Issues

4.2.5 Correlate The Performance Problem Reports And Manage

Inventory Repository

4.2.6 Manage Real Time Data Performance Data Collection

4.2.7 Establish, Maintain And Manage The Support Plans

4.2.8 Assess And Report Real Time Data Performance Management Processes

4.2.9 Provide Supporting Procedures And Quality Management Support

4.3 Support Real Time Data Trouble Management

4.3.1 Manage Real Time Data Trouble And Performance Data

Collection

4.3.2 Manage Real Time Data Infrastructure, Provisioning And

Preventive Maintenance Schedules

4.3.3 Create Report

4.3.4 Establish Warehouse And Manage Spares Including Other

Parties

4.3.5 Track, Monitor And Manage Real Time Data Trouble

Management Processes Including Other Parties

4.3.6 Provide Support For Real Time Data Trouble Management

And Support Service Problem Management Processes

4.4 Enable Real Time Data Collection And Distribution

4.4.1 Manage And Administer Real Time Data Collection And

Distribution

4.4.2 Manage Real Time Data Storage Facilities And Associated

Processes

4.4.3 Track, Monitor And Report Real Time Data Collection Processes And Capabilities

4.4.4 Identify Data Collection Issues And Report

4.5 Manage Real Time Data Inventory

4.5.1 Manage Real Time Data Inventory Database And

Processes

4.5.2 Track And Monitor Real Time Data Repository

Capabilities

4.5.3 Identify Repository Issues And Provide Reports And

Warnings

4.6 Manage Logistics

4.6.1 Manage Warehousing

4.6.2 Manage Orders

4.6.3 Track And Monitor Logistics And Manage Real Time

Data Inventory

4.6.4 Identify Logistic Issues And Provide Reports

Step 5 - Workforce Management – Processes And Functions Details

March 8, 2016 44


5.1 Manage Schedules And Appointments

5.1.1 Workforce Management Schedule

5.1.2 Determine Work Schedule

5.1.3 Manage Reservations

5.1.4 Manage Appointments

5.2 Plan And Forecast Workforce Management

5.2.1 Forecast Demand

5.2.2 Forecast Workforce Availability

5.2.3 Adjust Durations

5.3 Administer And Configure Workforce

Management

5.3.1 Configure Work Catalog

5.3.2 Administer Human Real Time Data Catalog

5.3.3 Administer Organisation's Catalog

5.3.4 Administer Tools And Materials Catalog

5.3.5 Configure Skill Catalog

5.3.6 Configure Schedules

5.3.7 Administer Registration And Access

5.3.8 Configure Logging And Audit

5.4 Report Workforce Management

5.5 Manage Work Order Lifecycle

5.5.1 Issue Work Order

5.5.2 Analyse And Decompose Work Order

5.5.3 Assign Task

5.5.4 Dispatch Task

5.5.5 Track And Manage Work Order

5.5.6 Close Work Order

5.5.7 Report On Work Order

Step 6 - Real Time Data Provisioning – Processes And Functions Details

March 8, 2016 45

6 Real Time Data Provisioning

6.1 Allocate And Install Real Time

Data

6.1.1 Determine Real Time Data

Availability

6.1.2 Reserve Real Time Data

6.1.3 Release Real Time Data

6.1.4 Allocate Real Time Data

6.1.5 Install And Commission

Real Time Data

6.2 Configure And Activate

Real Time Data

6.2.1 Configure Real Time Data

6.2.2 Implement Real Time Data

6.2.3 Activate Real Time Data

6.3 Test Real Time Data

6.3.1 Test Specific Real

Time Data

6.3.2 Develop Test Plans

6.3.3 Capture Test Results

6.4 Track And Manage Real

Time Data Provisioning

6.4.1 Coordinate Real Time Data

Provisioning Activity

6.4.2 Track Real Time Data




6.4.4 Update Real Time Data

Repository


Provisioning

6.5.1 Monitor Real Time Data

Order Status

6.5.2 Distribute Real Time Data

Order Notification

6.5.3 Distribute Real Time Data

Provisioning Reports

6.6 Close Real Time Data Order

6.7 Issue Real Time Data

Orders

6.7.1 Assess Real Time Data

Request

6.7.2 Create Real Time Data

Orders

6.7.3 Mark Real Time Data Order

For Special Handling

6.8 Recover Real Time Data

6.8.1 Develop Real Time Data Recovery Plan

6.8.2 Provide Real Time Data

Recovery Proposal

Notification

6.8.3 Request Real Time Data

Recovery Authorisation

6.8.4 Commence Real Time Data

Recovery

6.8.5 Complete Real Time Data

Recovery

6.8.6 Recover Specific Real

Time Data

Step 7 - Real Time Data Collection And Distribution – Processes And Functions Details

March 8, 2016 46

7 Real Time Data Collection And

Distribution

7.1 Collect Management


7.1.1 Intercept Events/Information

7.1.2 Deliver Management Information

7.2 Process Management


7.2.1 Determine Recipients For

Information/Data

7.2.2 Filter Information/Data

7.2.3 Aggregate Information/Data

7.2.4 Format Information/Data

7.3 Distribute Management


7.3.1 Distribute Information/Data

7.3.2 Manage Distribution

7.3.3 Confirm Distribution And Clean-

Up

7.4 Audit Management And Security Data

Collection And Distribution

Step 8 - Real Time Data Trouble Management – Processes And Functions Details

March 8, 2016 47

8 Real Time Data Trouble

Management

8.1 Survey And Analyse Real

Time Data Trouble

8.2 Localise Real Time Data

Trouble

8.3 Correct And Resolve Real Time

Data Trouble

8.4 Track And Manage Real

Time Data Trouble


Trouble

8.6 Close Real Time Data

Trouble Report Flow

8.7 Create Real Time Data

Trouble Report

Step 9 - Real Time Data Performance Management – Processes And Functions Details

March 8, 2016 48

9 Real Time Data Performance Management

9.1 Monitor Real Time Data Performance

9.1.1 Manage Real Time Data Performance Data

9.1.2 Record Real Time Data Performance Data

9.1.3 Correlate Real Time Data Performance Event

Notifications

9.2 Analyse Real Time Data Performance

9.2.1 Perform Specific Real Time Data

Performance Diagnostics

9.2.2 Manage Real Time Data Performance Data

Collection Schedules

9.3 Control Real Time Data Performance

9.3.1 Instantiate Real Time Data Performance

Controls

9.3.2 Instantiate Real Time Data Trouble

Controls

9.4 Report Real Time Data Performance

9.4.1 Monitor Real Time Data Performance

Degradation Report

9.4.2 Distribute Real Time Data Quality

Management Reports And Summaries

9.5 Create Real Time Data Performance

Degradation Report

9.5.1 Generate Real Time Data Performance

Degradation Problem

9.5.2 Convert Report To Real Time Data Performance

Degradation Report Format

9.6 Track And Manage Real Time Data

Performance Resolution

9.6.1 Coordinate Real Time Data Performance

9.6.2 Request Other Parties Performance Degradation Report

Creation And Update

9.6.3 Update First In Real Time Data Testing

Results

9.6.4 Cancel Real Time Data Performance

Degradation Report

9.6.5 Escalate/End Real Time Data Performance

Degradation Report

9.6.6 Clear Real Time Data Performance

Degradation Report Status

9.6.7 Engage External Party Real Time Data

9.7 Close Real Time Data Performance

Degradation Report

Step 10 - Real Time Data Aggregation And Reporting – Processes And Functions Details

March 8, 2016 49

10 Real Time Data Aggregation And

Reporting

10.1 Aggregation Real Time Data

Records

10.1.1 Validate Real Time Data

Records

10.1.2 Normalise Real Time Data

Records

10.1.3 Convert Real Time Data

Records

10.1.4 Correlate Real Time Data

Records

10.1.5 Remove Duplicate Real

Time Data Records

10.2 Report Real Time Data Records

Using Real Time Data Strategy and Implementation Approach

March 8, 2016 50

Activity Timeline Real Time Data Strategy and Implementation Plan

1 Real Time Data Strategy And Planning


1.1.1 Gather Real Time Data Information

1.1.2 Analyse New Real Time Data Requirements

1.1.3 Analyse To Develop New/Enhance Real Time Data Requirements


1.2.1 Manage Real Time Data Research Investigations

1.2.2 Manage Administration Of Real Time Data Research

1.2.3 Define Real Time Data Research Assessment Methodologies


1.3.1 Establish Real Time Data Strategy

1.3.2 Develop Real Time Data Strategy

1.3.3 Establish Real Time Data Delivery Goals

1.3.4 Establish Real Time Data Implementation Policies

1.4 Define Real Time Data Support Strategies

1.4.1 Define Real Time Data Support Principles

1.4.2 Define Real Time Data Support Policies

1.4.3 Define Real Time Data Support Performance Standards

…

Using Real Time Data Strategy and Implementation Approach

• Use the proposed work breakdown to produce a detailed plan

March 8, 2016 51

Real Time Architecture High-Level Components

March 8, 2016 52

Data Sources Data Collection

And Data Source Management

Communications And Security

Data Integration

Data Quality/ Summary/

Filter/ Transformation

Data Storage Data Storage Infrastructure

Data Reporting and Analysis

System Management, Administration

and Control

External Systems (Asset Management,

Workforce Management)

Real Time Architecture High-Level Components

March 8, 2016 53

Component Description Data Sources These are the data collection/generation sources/sensors installed in the data collection landscape and new

sensor devices and signal data sources. Over time the approach to data collection may be standardised and old equipment replaced.

Data Collection And Data Source Management

This logical component acts as a local front-end to existing signal data collection/generation sources/sensors. It eliminates the need to replace existing devices. It offers a standard interface. It manages the sensor infrastructure and landscape

Communications And Security

This is the communications infrastructure used to securely transmit remotely collected data from local data sources/sensors and data collection units to the central facility.

Data Integration This component manages the receipt of multiple data types in multiple formats from multiple data sources.

Data Quality/ Summary/ Filter/ Transformation

This component applies data quality algorithms to intrinsically noisy real time data to make it more usable. Data may be filtered, summarised and transformed prior to storage

Data Storage This is the software component for storing in a structured manner and providing access to real time data.

Data Storage Infrastructure

This is the underlying data storage infrastructure. The volumes of real time data are potentially very substantial – hundreds of millions of data points per day. The data storage requirements could amount to thousands of Terabytes over several years. This components includes facilities for backup, recovery, archiving and deletion.

System Management, Administration and Control

This component provides facilities to manage, administer and control the overall real time system.

Data Reporting and Analysis

This provides reporting and analysis facilities to meet a wide variety of business requirements.

External Systems Real time data can be merged with other data such as asset to provide a usage dimension to static asset data and to integrate with workforce management to manage sensor infrastructure.

Possible Approaches To Real Time Data Architecture

• You can expend a great deal of time, resources and money on defining the requirements of an idealised real time data architecture before embarking on any procurement and implementation

OR

• You can research the viable products and their functionality and what other water utilities have implemented and define requirements in terms of what is realistically achievable, reducing costs and time and delivering results and learning more quickly

March 8, 2016 54

Getting Real Time Data Right

• Avoid false starts. Balance implementation urgency and pace with organisational readiness and maturity. Success requires change management

• Cannot do it all at once

• Real time data is an enabler and not an end in itself

• Embed use of real time data data in the organisation

• Don't turn it into a "finding the right tool" decision

• Recognise the interaction of governance, processes and tools that enables the organisation optimise its real time data investment

• Focus on value, risks and prioritisation with active engagement of key stakeholders

• Avoid using it solely for one-time annual budget decisions. It is about continuous alignment, tracking and benefits realisation of existing real time data systems and new projects

• Clarify governance and put a process in place that uses portfolio management approach to consider, make and enforce decisions

March 8, 2016 55

Summary

• Real time data includes Telemetry, Big Data, Smart Metering and Internet of Things

• Represents an idealised organisation and process breakdown across entire spectrum of real time data from strategy to workforce management to device installation, data collection and data usage and actioning

• Provides a basis for developing a work breakdown and an implementation plan

• Real time data infrastructure and systems without organisation and processes will yield few benefits

• Represents a comprehensive structure that an organisation can adapt to meet its long-term real time data needs

• Enables application and use of real time data to be embedded in the organisation

March 8, 2016 56

real time data strategy and architecture

Technology