meng - cs - technology management
TRANSCRIPT
Technology Management 873 Individual Assignment 2015
Compton Saunders [email protected]
Name: Compton Saunders
Student Number: 13718436 Degree: MEng- Engineering Management
Lecturers: Prof Tinus Pretorius Due Date: February 20th, 2015
ii
Table of Contents
List of Abbreviations and Acronyms ......................................................................................... iii List of Figures ............................................................................................................................ iii List of Tables ............................................................................................................................. iv
List of Equations ........................................................................................................................ iv
Individual Assignment: The Application of Technology Readiness Levels and Integration Readiness Levels in order to assess a Maximum Demand Control system utilising ZigBee Wireless Mesh Technology ........................................................................................................ 1
1. Introduction ........................................................................................................................... 1
2. Technology Readiness Assessment (TRA) .............................................................................. 2
3. Technology Readiness Assessment (TRA) Submission Document ......................................... 4
3.1 Purpose of This Document ............................................................................................... 4
3.2 Programme Objective ...................................................................................................... 5
3.3 Programme Description ................................................................................................... 6
3.4 System Description ........................................................................................................... 7
3.5 Critical Technology Elements (CTEs) .............................................................................. 11
3.6 Review of TRL Findings ................................................................................................... 14
4. Review of Demand Control System Using Qualitative Maturity Multi Metric Technique .. 17
4.1 SRL Calculation for the Demand Control System ........................................................... 21
5. Conclusion ............................................................................................................................ 23
Appendix A ............................................................................................................................... 26
Appendix B ............................................................................................................................... 30
Bibliography ............................................................................................................................. 31
iii
List of Abbreviations and Acronyms
AD Advancement Degree of Difficulty ADMD After Diversity Maximum Demand CTEs Critical Technology Elements DOD Department of Defense EU European Union GAO General Accountability Office IRL Integration Readiness Level IRT Independent Review Team NASA National Aeronautics and Space Administration NATO North Atlantic Treaty Organization OECD Organisation for Economic Co-operation and Development PC Personal Computer PLC Programmable Logic Controller SCADA Supervisory Control and Data Acquisition SRL System Readiness Level TRA Technology Readiness Assessment TRL Technology Readiness Level TCP/IP Transmission Control Protocol and Internet Protocol VA Volt-Ampere
List of Figures
Figure 1: Technology Readiness Levels. Source: (DoD 2011) ................................................... 3
Figure 2: Requirements of a Technology Readiness Assessment Document. Source (DoD 2011).................................................................................................................................................... 4
Figure 3: Basis of Technology Maturity Assessments throughout Acquisition. Source (DoD 2009) .......................................................................................................................................... 5
Figure 4: Arial view of property where ZigBee Wireless Mesh Technology load switches are installed. Large 200m radius area. ............................................................................................. 8
Figure 5: Load management load shape objectives. Source (Malik and AL Mata’ni 2007) ...... 9
Figure 6: Demand of 175 KVA was reached prior to MDC install – monthly cost R15 118 ..... 10
Figure 7: Demand of 135 KVA was reached after MDC install – monthly cost R11 626 ......... 10
Figure 8: Online Web portal to via historical and quasi real-time data .................................. 10
Figure 9: SCADA interface which allowing users direct monitoring and control capability .... 11
Figure 10: Real World Ecosystem: Smart Metering with energy efficient heat pumps and Zigbee based maximum demand control ................................................................................ 11
Figure 11: Technology Assessment process proposed by (Bilbro 2007) ................................. 18
Figure 12: Relationship between TRL, IRL and SRL .................................................................. 18
Figure 13: System Readiness Level Calculation. Source (SIT 2010) ........................................ 22
Figure 14: Different Engineering Lifecycles and how the System Readiness Level (SRL) is mapped. Source (Sauser and Ramirez-Marquez 2007) ........................................................... 23
Figure 15: Descriptive Requirements for Technology Readiness Assessment Document. Source (DoD 2009) ............................................................................................................................... 30
iv
List of Tables
Table 1: Technology Readiness Levels for Demand Control System ........................................ 14
Table 2: Summary of CTE Technology Readiness Level ............................................................ 14
Table 3: Techniques for Assessing Qualitative Maturity. Source (Azizian et al. 2009) ............ 17
Table 4: Integration Readiness Levels. Source (Sauser and Ramirez-Marquez 2007)............. 19
Table 5: Hardware Technology Readiness Definitions, Descriptions, and Supporting Information. Source (DoD 2009) .............................................................................................. 26
Table 6: Hardware Technology Readiness Definitions, Descriptions, and Supporting Information. Source (DoD 2009) .............................................................................................. 27
Table 7: Additional Definitions of TRL Descriptive Terms. Source (DoD 2009) ....................... 29
List of Equations Equation 1 ................................................................................................................................ 19
Equation 2 ................................................................................................................................ 20
Equation 3 ................................................................................................................................ 20
Equation 4 ................................................................................................................................ 20
Equation 5 ................................................................................................................................ 21
Individual Assignment: The Application of Technology
Readiness Levels and Integration Readiness Levels in
order to assess a ZigBee Wireless Mesh Technology
based Demand Control system.
1. Introduction Globally, engineers are faced with the development of technology and the integration of
these technologies within larger systems. System integration is defined by Buede (2000) as
“the process of assembling the system from its components, which must be assembled from
their configuration items.” The impression one gets from this definition is that putting
together a system is a relative simplistic task, however Buede (2000) elaborates to say that
the process of integration is very complex and contains numerous tasks that overlap and are
iterative in order to create a system which meets the original requirements as well as
successfully operate in the intended environment. Many challenges exist within the
integration process. Some of these challenges are technology specific while others are related
to the integration and emergence observed due to integration; and dependant on how
mature the technology or the integration of technology is.
In order asses the maturity of technology and its integration, numerous metrics have been
developed in order to assist decision making. Some of these assessment metrics include the
Technology Readiness Level (TRL) (DoD 2009); Integration Readiness Level (IRL) and the
System Readiness Level (SRL) (Sauser, Gove, Forbes and Ramirez-Marquez 2010, Sauser and
Ramirez-Marquez 2007). These metrics attempt to provide a consistent manner in which
different technologies can be compared in terms of its maturity.
This paper will use some of these technology maturity assessment techniques in order to gain
an understanding of how they are applied and used in practise. A case study will then discuss
2
an energy demand control system which was developed and commercially deployed in South
Africa, but experienced issues due to technology and integration challenges.
2. Technology Readiness Assessment (TRA)
During the 1970s the National Aeronautics and Space Administration (NASA) developed the
Technology Readiness Assessment (TRA) as a tool to manage risk within its research and
technology development programmes (Mankins 2009). The purpose of the TRA is to gain an
understanding of the TRL of all the various technologies being used within a greater system.
The TRL is essentially a metric to evaluate the risk related to technology development. The
first comprehensive definitions of each of TRLs were released in 1995 and have since been
adopted by the U.S. Congress’ General Accountability Office (GAO); U.S. Department of
Defense (DOD); Organisation for Economic Co-operation and Development (OECD); European
Union (EU); North Atlantic Treaty Organization (NATO) and other countries such as Australia,
Canada and the United Kingdom (Mankins 2009, Bolat 2014).
According to the U.S. DOD TRA Deskbook (DoD 2009:6), the definition of a TRA is:
“A TRA is a formal, systematic, metrics-based process and accompanying report that assesses
the maturity of technologies called Critical Technology Elements (CTEs) to be used in systems.
CTEs can be hardware or software.”
The definition of a CTE is: “A technology element is “critical” if the system being acquired
depends on this technology element to meet operational requirements (within acceptable cost
and schedule limits) and if the technology element or its application is either new or novel or
in an area that poses major techno- logical risk during detailed design or demonstration.”
(DoD 2009:6)
A technology can be classified as a CTE when it poses a significant risk and in such a case the
TRA should include technical information that can be used to reduce risk. TRLs are used as a
metric by an Independent Review Team (IRT) consisting of subject matter experts (SMEs)(DoD
2009).
3
The TRL scale ranges from one through nine, as seen in Figure 1, where (Technology Readiness
Level) TRL1 would consider early stages of scientific investigation and TRL9 will indicate that
a technology was successfully used within a system.
Figure 1: Technology Readiness Levels. Source: (DoD 2011)
In order to determine the maturity of a particular technology, the programme ideas,
technology requirements as well as the proven technology capabilities are evaluated by a
TRA. Typically a CTE will be assigned a readiness level based on the TRA. TRLs are indicative
of a reached level of maturity at the time that the CTE was measured and does not indicate
how valid a design is and also does not provide an indication of the challenges involved with
progressing to the next level. When CTEs are identified they should be assessed from a
systems engineering perspective and the assumption should be made that the relevant CTE
is capable of performing its required function. CTEs needs to be evaluated while considering
how it will be integrated into a system as the CTE may appear as mature on its own but could
prove to be immature due to other system effects. CTEs can also be classified as hardware or
software and depending on the classification it will have different evaluation criteria which
can be seen in Appendix A (DoD 2011).
4
3. Technology Readiness Assessment (TRA) Submission
Document
The U.S. DOD TRA Deskbook (DoD 2009) provides an outline for a TRA Submission as seen in
Figure 2.
Figure 2: Requirements of a Technology Readiness Assessment Document. Source (DoD 2011)
This study will only consider key areas of the document requirements in order to introduce
and assess the case study from a TSA and TRL perspective. A more detailed requirement of
the TRA document can be found in Appendix B. The proceeding sections will attempt to
develop the TRA document based on the demand control system case study. The following
sections will follow the guidelines provided in Figure 2 and an attempt to compile a TRA
document.
3.1 Purpose of This Document
This document is representative of a TRA, performed independently, for the demand control
programme in support of the Milestone B decision. The TRA was performed at the request of
Company X Technology Director.
There are three major Milestones which indicate a stage within the acquisition cycle which
are Milestone A, B and C. The reason Milestone B was selected is due to the programme
already being in the Engineering and Manufacturing Development phase of the Acquisition
System and the need to identify which technologies are not mature and would result in
additional costs and delays within development schedules (DoD 2009).
5
Figure 3: Basis of Technology Maturity Assessments throughout Acquisition. Source (DoD 2009)
3.2 Programme Objective
The demand pattern at which electricity is utilised by a system can be managed by suing
demand control or load management systems. Load management is regarded as being
integrated with demand side management and involves the decrease and alteration of the
demand required by the system over time with the goal of improving the balance observed
between the energy requirements of the customer or consumer and the current generation
capacity of the supplier, future generation capacity, future generation capacity, transmission
and distribution resources (Malik and AL Mata’ni 2007).
There are a number of ways which user energy demand can be managed by altering
consumption patterns. However, long term sustainability of demand management system
largely depends on the behavioural response of users and the how the design of a load
management system influences users and, in addition, the success or failure of such a system
is often determined by the attitude of the user (ABU-Zeid and AL-Shakarchi 2002). ABU-Zeid
and AL-Shakarchi (2002) also state that the load management goal is to flatten the load curve
by influencing the behavioural consumption of energy.
The purpose of the programme is to develop a modular control system capable of dynamically
switching non-essential electrical loads such as geysers, boilers, air conditioners, heat pumps,
chillers, freezers and lights in order to manage the electrical load required by a system.
6
Commercial consumers are normally billed for their energy consumption based on tariffs
which have components such as levies and consumed energy in kilowatt-hours (kWh). Tariffs
can also include a charge for demand or notified demand1 which is dependent on the
maximum demand.
Once the load can be managed, it will become possible to manipulate consumption or
demand profiles which can reduce the costs of electricity. The cost saving can be realised by
reducing notified demand, shifting consumption to off peak periods, reducing overall energy
consumption and provide flexibility to adjust billing tariff to exploit adjustable demand
profile.
3.3 Programme Description
The demand control system is an incremental improvement on existing systems with similar
capability. The proposed programme will however not build on any existing system but
develop a new system with new technology in order to achieve demand control objectives.
Although the basic methods for demand control exist, the proposed programme will develop
its own demand control philosophy and methodology and algorithms on a processing
platform, the Remote Data Acquisition and Control (RDAc) platform, which has never been
used in such a deployment.
In addition to the processing platform, ZigBee Wireless Mesh Technology (ZigBee) will be used
in order to provide a two way communication highway to send and receive data across the
system. Historically, large scale demand control systems used radio frequency ripple control
systems which only had one way communication and the load switches used to control
devices could only receive commands but not provide any feedback regarding their status.
The utilisation of ZigBee Technology stems from requirements set out by Eskom which
dictates that demand-side management (EEDSM) projects use two way communications
systems within its system architecture in order to send and receive data. The ripple control
systems thus had to rely on statistical methods using after diversity maximum demand
1 Maximum demand notified in writing by the customer and accepted by the utility (Eskom).
7
(ADMD)2 in order to make decisions regarding the shedding of load clusters without having
real-time decision making data. The use of ZigBee will enable load switching devices to
communicate measured values and status in real-time, drastically improving demand
management capabilities.
3.4 System Description
The goal of a load management or demand control system is to manage the power
requirements of an energy consuming system. Power (watts) is measured instantaneously
and is the rate at which work is performed while energy (kwh) is the integral of power over
time. As an example, If a 100 watt light bulb requires 100 watts of power and is switched on
for one hour, that light bulb will use 100 watt-hours of energy. The maximum demand of the
light bulb will be 100 watt. Maximum demand can be seen as the maximum instantaneous
power required from the main power grid by a system over a specific timeframe. Demand is
measured in volt-ampere (VA).
The assessed system uses ZigBee load switching and measurement devices that have the
capability of real-time measurement of the instantaneous power (in watt), energy it
consumption (in watt-hours) and status (on/off) of an industrial electrical appliance. The
ZigBee device can then in real-time send the measured information back to a central
processing unit or platform via the ZigBee Wireless Mesh network. This capability enables the
demand control system to dynamically determine the loads that are consuming power at that
specific instance in time (switched on)3. The demand control system can calculate what the
reduction in demand will be when the load is switched off.
Another clear benefit of using the ZigBee Technology is that it can cover large areas. The
control and monitoring coverage of large distributed electrical loads is not possible and too
2 Simultaneous maximum demand of a group of consumers divided by the number of consumers, expressed in kilovolt amperes. 3 It is important to know whether a load is consuming energy as a geyser could be switched on at the control point but not consuming energy as the thermostat switch can be off due to the water in the geyser being at its desired temperature.
8
costly using standard cables or conventional Wi-Fi technology. ZigBee Technology forms a
self-healing mesh where each node can act as a relay hop to form the mesh network and so
enables the communication coverage of large areas in a cost effective manner. Figure 4 shows
a large school property that has distributed geyser loads which is controlled via ZigBee based
load switches.
Figure 4: Arial view of property where ZigBee Wireless Mesh Technology load switches are installed. Large 200m radius
area.
The demand of the entire system and all the loads in Figure 4 are then sent to the central
processing platform every 10 seconds via the ZigBee Wireless Mesh Network.
The processing platform will be the Remote Data Acquisition and control or RDAc technology
that provides hardware intelligence for control, data logging and storage while integrating
multiple inputs and outputs with various communication possibilities. The RDAc platform is
regarded as a hybrid between cell phone, programmable logic controller (PLC) and Personal
Computer (PC) technology.
As the RDAc receives the data in real-time, it uses control algorithms specifically developed
for the system, to predict what the system demand would be within a 30 minute integration
period. The algorithm then calculates how much load (devices/appliances) it needs to shed
(switch off) or it restore (switch on) while remaining within the constraints of the demand set
9
point provided to the system. The switching of devices does not just consider the on/off status
of a device but considers other parameters such as priority and cool down periods depending
on the device it is controlling. The demand control algorithm and system can very accurately
determine what to control due to its two-way communication capability compared to older
systems that use statistical techniques.
There are a range of load management techniques such as peak clipping, valley filling,
strategic load conservation and load shifting as seen in Figure 5.
Figure 5: Load management load shape objectives. Source (Malik and AL Mata’ni 2007)
The assessed system predominantly uses the load shifting techniques which still utilises the
same total energy but results in lower electricity bills by reducing maximum demand peaks as
well as moving energy consumption into cheaper off-peak periods.
Figure 6 shows a scenario prior to the deployment of the assessed demand control system.
The required demand is more than 175 kilo volt ampere (KVA). When considering Figure 7,
the demand is managed to remain below 140 KVA via the demand control system which
equals a monthly saving of about R3500.
10
Figure 6: Demand of 175 KVA was reached prior to MDC install – monthly cost R15 118
Figure 7: Demand of 135 KVA was reached after MDC install – monthly cost R11 626
The system also uses smart meters to measure the load at the grid connection point. This
information is also sent back to the RDAc via the ZigBee network. All system data is also logged
in the RDAc in 1 minute and 30 minute log intervals and read back to a central long term data
storage and analytics server using automated meter reading via GPRS/3G on the Vodacom
network.
The users have two main ways of interacting with the system. The first is via an online web
portal which graphically displays the logged system data and enables the analysis of demand
trends and tariff studies. This can be seen in Figure 8 below.
Figure 8: Online Web portal to via historical and quasi real-time data
Users are also capable of monitoring and controlling the system via
supervisory control and data acquisition (SCADA) software, as seen in Figure 9, developed
11
specifically for the demand control system. The SCADA system enables users to monitor,
interact and control the physical system, in real-time, via a computer user interface.
Figure 9: SCADA interface which allowing users direct monitoring and control capability
3.5 Critical Technology Elements (CTEs)
Figure 10 shows the entire demand control ecosystem or architecture of the system under
assessment.
Figure 10: Real World Ecosystem: Smart Metering with energy efficient heat pumps and Zigbee based maximum demand
control
12
Table 1 outlines the identified CTEs within the demand control system which will be assessed
according to the hardware and software readiness definitions, descriptions and supporting
information set out in Appendix A (DoD 2011).
Technology Hardware/
Software
Function in Relation to System TRL Reason for TRL level
CTE1 ZigBee
Technology
HW Suite of high-level communication protocols
that allow the creating of a local mesh
network. Mesh network is critical for
deployment in a large decentralised
environment and enables the relay of two
way communication and data transfer. Load
switches are ZigBee enabled and capable of
measuring and controlling the connected
load. Measurement data is then relayed back
to the central control platform via the ZigBee
mesh network. Load switching devices as well
as all smart meters have ZigBee capabilities.
TRL9 The Development of ZigBee based
systems first emerged around 2005
(Eady 2005) and has now already
successfully been implemented and
operated in a range of applications.
Examples of where ZigBee is used
within a similar data acquisition
scenario can be found in Calmeyer
(2012) and Shariff, Rahim and Hew
(2015).
CTE2 3G
Technology
HW 3G (third generation) is regarded as the third
generation of mobile telecommunications
technology. 3G was used to send logged data
as well as real-time data back to the central
database server over the mobile operator
network. The main processing platform, the
RDAc, has 3G capabilities as well smart
meters.
TRL9 3G technology was introduced to the
market in 1998. The technology has
already progressed to 4G and 5G
technologies. Within the deployment
of the system the 3G technology
worked within an operational
environment.
CTE3 Linux
Technology
SW The RDAc platform uses Linux technology, an
open source platform, as its operating
system. An operating system enables the
software running on the device to access
hardware functions available on the device.
The RDAc platform hosts a range of hardware
capabilities such as digital inputs, analogue
inputs, digital outputs, Ethernet Interface, 3G
communications, RS232 interface, RS485
interface, storage media, LC and C-
programming language. The Linux operating
system acts as an interface to all the
hardware and software functions.
Linux is, in simplest terms, an operating
system. It is the software on a computer that
enables applications and the computer
TRL9 Linux was first introduced in 1991 and
has since become a stable platform
used in many types of applications.
Linux is known for its stability and
flexibility (Proffitt 2009). The Linux
operating system proved itself to be
stable within the operating
environment.
13
Technology Hardware/
Software
Function in Relation to System TRL Reason for TRL level
operator to access the devices on the
computer to perform desired functions.
CTE4 Control
Algorithm
SW The control algorithm which is developed
using the C programming language is at the
heart of the system. The algorithm collects all
the data via the Ethernet and ZigBee
interfaces. The algorithm then interprets the
information and makes load shedding or
restore decisions based on programmed
parameters.
TRL7 Demonstrated feasibility within
operational prototype scenarios.
Software fully integrated with
operational hardware and software
systems. The technological
capabilities of the software have been
measured against its required
capabilities.
Although some documentation has
been started not all documentation
has been completed.
CTE5 RDAc
Platform
HW The RDAc provides the platform the hardware
platform and computing power to collect
data, process data, execute programmes and
commands, log data, and communicate using
various technologies. It is central to the
system.
TRL8 The RDAc platform has proven to
operate successfully under expected
conditions providing satisfactory
results. The platform is not
experiencing any more base
development but slight incremental
improvements are made to
embedded code. The RDAc platform
meets its design specifications.
Although the platform operates as
required there are still some problems
encountered which are mainly related
to the embedded operating system
and software drivers.
CTE6 Smart
Metering
HW Smart meters are devices that can record the
consumption of electricity and capable of
two-way communications. Within the system
smart meters are deployed at select locations
such as the main point of supply from the
grid. This is the point where energy is billed
and also the main measurement point for the
system demand. The smart meters are
equipped with ZigBee communications for
local area network communication and the
relay of information back to the central
processing platform. The smart meters also
gave 3G modems which enable the
communication of data back to the central
data collection server.
TRL9 The smart meter technology has
proven to operate successfully under
operational scenarios. The platform is
not experiencing any more
development. Smart meter
technology has also been used
successfully within the metering
industry for a number of years (Aslam,
Soban, Akhtar and Zaffar 2015, Bago
and Campos 2015).
14
Technology Hardware/
Software
Function in Relation to System TRL Reason for TRL level
CTE7 TCP/IP SW Transmission Control protocol and Internet
Protocol (TCP/IP) is the basic communication
language used by private networks as well as
the Internet. TCP/IP offers what is called end-
to-end communications and specifies how
data is packaged, addressed, sent, routed and
the received at the desired destination.
Within the system TCP/IP is used between the
RDAc platform and the ZigBee devices. The
ZigBee network coordinator collects data
from the ZigBee Mesh Network and then
encapsulated the data and transports it
within the TCP protocol over an Ethernet
connection to the RDAc platform. The RDAc
also uses 3G technology which used TCP/IP.
TRL9 TCP/IP technology has been available
for many years and has been
integrated into everyday life. The
TCP/IP software is readily repeatable
and usable and completely integrated
into the operational hardware,
software and environment. TCP/IP has
been documented and verified.
TCP/IP has had successful operational
experience with sustainable
engineering support.
Table 1: Technology Readiness Levels for Demand Control System
3.6 Review of TRL Findings
The summary of the technology readiness assessment can be seen in Table 2. A general
observation is that most technologies are relatively mature with the selected technology
readiness levels ranging from TRL7 to TRL9. All technologies have proven to work within an
isolated environment as well as within the system environment. The conclusion drawn from
Table 2 is that there are no technologies within the system which falls below TRL7 and the
project can thus be approved for Milestone C, instead if Milestone B, which is the approval to
enter into low production.
Identified CTE CTE Technology Hardware/Software Technology Readiness Level
CTE1 ZigBee Technology HW TRL8
CTE2 RDAc Platform HW TRL8
CTE3 3G Technology HW TRL9
CTE4 Linux Technology SW TRL9
CTE5 Control Algorithm SW TRL7
CTE6 Smart Metering HW TRL9
CTE7 TCP/IP SW TRL9
Table 2: Summary of CTE Technology Readiness Level
According to TRL Desk book (DoD 2009), Milestone C marks the point at which low rate
production can be imitated with limited deployment of in order to test operational readiness.
15
Milestone C is important as it should show how technology deficiencies have been resolved
during the Engineering and Manufacturing Development (EMD) phase. It also indicates that
all CTEs are maturing as anticipated and will continue to mature through the use of
Technology Maturation Plans. Considering software, TRL7 indicates that all the source code
has been developed and has been tested to ensure that it integrates into the entire system
and that it can successfully be ported to a different host platform (DoD 2009).
How does this assessment compare to a real-world situation? The demand control system
described in previous sections for the document was deployed at three commercial
businesses in Mpumalanga after it went through the initial development process. The initial
development process took substantially longer than anticipated as there were two major
issues. The first challenge was that the ZigBee product platform developed by a Danish
company was still undergoing changes. The current products they had available worked as
required but after introducing the changes, which were required to operate on the ZigBee
Smart Energy Profile, their platform experienced some problems. This was mainly due to the
added security on the Smart Energy profile which required that the hardware had more
processing and storage capacity. This delayed the initial deployment schedule by 6 months as
they were upgrading and refining their solution. The second challenge was that the initial
selected central processing platform was not flexible and capable enough of handling all the
required system functionality which evolved with the project. The RDAc was then selected as
the best available platform but was never deployed in this particular solution or system. The
combined delay in delivering the first system was 12 months. All the technologies were tested
individually and tested as an integrated system and deployed to commercial customers. After
1 month of successful operation the ZigBee platform, more specifically the software code on
the ZigBee hardware, crashed leaving the entire system non-operational and after another 6
months of effort the system was upgraded in order to offer more stability which incremental
changes to other parts of the system as well.
The TRL alone does not seem adequate enough to ensure that a technology within an
integrated system can be successfully deployed. The literature finds many cases where TRL as
16
a metric is criticised and according to Cornford and Sarsfield (2004) it is not accurate,
subjective with no clear definition and does not have much value in supporting certain
decisions. Tetlay and John (2010) also debate that that the term “System Readiness” and
“Maturity” should not be used in an interchangeable manner. Other concerns are that it does
not offer an encompassing way of understanding the difficulty of integrating individual
technologies or subsystems into a complete operational system (Mankins 2002, Valerdi and
Kohl 2004) and that it does not provide a way of comparing alternate TRLs (Valerdi and Kohl
2004, Smith 2005, Mankins 2002). The inference made in the literature is that when a system
is considered instead of a single technology there are a more comprehensive set of metrics
required to assess system readiness. Sauser, Verma, Ramirez-Marquez and Gove (2006)
propose that an additional readiness metric called the Integration Readiness Level (IRL) needs
to be used on conjunction with the Technology Readiness Level (TRL) metric in order to
determine the system readiness which is then in turn measured by a System Readiness Level
(SRL). Bilbro (2007) also proposes using two different metrics when assessing technologies;
the first is TRL scale developed by NASA as well as another scale consisting of nine levels which
is called the Advancement Degree of Difficulty.
Azizian, Sarkani and Mazzuchi (2009) provide a summary of the numerous readiness levels and maturity assessments and
can be seen in
Table 3 below.
17
Table 3: Techniques for Assessing Qualitative Maturity. Source (Azizian et al. 2009)
4. Review of Demand Control System Using Qualitative
Maturity Multi Metric Technique
Bilbro (2007) regards the technology assessment as a two tiered process which first considers
the current technology maturity though assessing TRLs and then secondly; follows another
process by which it is determined how challenging it will be to take move a technology from
its current TRL to the next TRL by using Advancement Degree of Difficulty (AD2). The iterative
process can be seen in Figure 11 below.
18
Figure 11: Technology Assessment process proposed by Bilbro (2007).
Although a Microsoft Excel tool does exist to perform an assessment using the method
proposed by Bilbro (2007), it was found that the tool is very complex and does not offer real
guidance in terms of interpreting the results. For the purpose of this paper the method
proposed by Sauser and Ramirez-Marquez (2007) will be assessed from a practical
perspective. According to Sauser and Ramirez-Marquez (2007), the inefficiencies of TRLs, as
discussed in the previous section, can be addressed by a composite method, which they have
termed the System Readiness Level (SRL), to act as a quantifier in assessing system maturity.
The SRL is a function of the existing TRL of each technology within a system and IRL which is
a metric to assess the complexity involved with integrating the relevant technologies. The
definition of each IRL can be seen in Table 4 with Figure 12 graphically representing the
rational by which SRL is developed.
Figure 12: Relationship between TRL, IRL and SRL. Source (Sauser and Ramirez-Marquez 2007)
19
Table 4: Integration Readiness Levels. Source (Sauser and Ramirez-Marquez 2007)
The SRL metric can thus be calculated as a TRL and IRL pair-wise comparison matrix that is
normalised and then interpreted as index of maturity between 0 and 1 (Sauser and Ramirez-
Marquez 2007). The single column [TRL] matrix is defined in Equation 1 where a system of n
technologies.
Equation 1
20
The possible integration of all technologies are then represented in a symmetric (n x n) square
matrix. In a system which has n technologies the [IRL] matrix is defined in Equation 2 with the
IRL between technologies i and j is indicated by IRLij. In the event that no integration is
planned between two technologies it is given a IRL of 9 (Sauser and Ramirez-Marquez 2007).
Equation 2
The values that were originally obtained from the TRL and IRL can be used but the normalising
the values provided a more accurate comparison when the use of competing technologies are
considered. The original [TRL] and [IRL] matrix values are thus generally normalised from their
1 to 9 levels to 0 to 1. The [SRL] matrix is then calculated by multiplying [TRL] and [IRL] as seen
in Equation 3.
Equation 3
The calculated [SRL] matrix contains a single element for every fundamental technology
regards to integration. The readiness of level of each specific technology in relation to another
technology is quantified while simultaneously accounting for the state of development of
each respective technology through its TRL. The [SRL] calculation for a system with n
technologies can be seen in Equation 4.
Equation 4
The resulting SRL values that are calculated via Equation 4 will reside in the interval 0 to n but
from a consistency perspective should be normalised to the interval 0 to 1 by dividing by n.
21
Interestingly the [SRL] matrix can also be used as a tool to assess which elements should be
prioritised in terms of system technology integration and expose paucities within the maturity
process.
Lastly the SRL for the entire system is calculated as the average of all the [SRL] values that
were normalised and the calculation can be seen in Equation 5.
Equation 5
4.1 SRL Calculation for the Demand Control System
The demand dontrol system that was assessed in previous sections will now be assed
according to the multi metric techniques proposed by (Sauser and Ramirez-Marquez 2007).
After selecting the appropriate Integration Readiness levels according to Table 4, the [SRL]
matrix and SRL of the demand control System is calculated using an online tool developed by
the Stevens Institute of Technology (SIT 2010). The results can be seen in Figure 13. Firstly,
the [SRL] results for each individual technology which is indicated as the Integrated
Technology Readiness Level for each technology in Figure 13 is considered. The SRL for the
RDAc has the lowest score at 0.83 indicating that the technology has deficiencies within the
maturity process and needs to be prioritised. Interestingly there are other technologies such
as the control algorithm which and TCP/IP with a SRL of 0.85 which is lower than the ZigBee
SRL of 0.86. Intuitively it would make sense that the RDAc, which is at the heart of the system,
would have the lowest SRL score. However the scores for the TCP/IP and ZigBee technologies
were not was expected. TCP/IP is a very mature technology and is standard almost equipment
for most technology based projects while the most demand control system failures were due
to the ZigBee platform.
22
Figure 13: System Readiness Level Calculation. Source (SIT 2010)
The system SRL can be mapped against the engineering lifecycles shown in Figure 14. The SRL
score of 0.86 would indicate that the system is ready for use according to all of the
engineering lifecycles shown in in Figure 14.
23
Figure 14: Different Engineering Lifecycles and how the System Readiness Level (SRL) is mapped. Source (Sauser and
Ramirez-Marquez 2007)
This would verify the decision to have had deployed the demand control in an operational
environment. However, the system would often fail and remain offline for extended periods
(2 – 4 weeks) due to problems with the ZigBee technology. Once the ZigBee component was
restored due to upgrades, the system would function for extended periods of time (6 – 9
months). However Sauser and Ramirez-Marquez (2007) does state that a system will hardly
ever achieve an SRL of greater than 0.9 as systems are generally deployed with technology
and integration which is not completely mature.
5. Conclusion
The demand control System was first tested using the TRL method developed by the U.S DoD
(DoD 2009). The obtained results indicate that all the technology was mature enough to reach
Milestone C which dictated limited system deployed in order to test whether the system was
ready for operations and does not mean that the system is completely ready for in-field
deployment.
24
In reality, the demand control system was deployed on a limited scale with three installations
but as a commercially ready solution and with no Technology Maturation Plans, which goes
against the recommendation of the book (DoD 2009). The demand control system, initially
operated according to the initial specifications but after a few weeks of operating,
experienced a major failure due to the ZigBee technology. The problems with the ZigBee
technology were mainly due to the ZigBee equipment not being stringently tested in a similar
condition with a weak ZigBee network and large volumes of data constantly having to retry
and find alternate routes. When considering the TRL of the ZigBee technology alone, it would
seem that is has matured enough to be deployed as a technology which was not the case. The
TRA where only the TRL is used as a metric thus seems to have limitations when integration
between technologies is required.
Literature revealed that TRL metric alone does not seem adequate to ensure that a
technology within an integrated system can be successfully deployed and that there is a
requirement for a more comprehensive set of metrics required to assess system readiness
(Cornford and Sarsfield 2004, Tetlay and John 2010, Valerdi and Kohl 2004, Smith 2005,
Mankins 2002). In order to address the deficiencies in the sole use of the TRL, the SRL multi
metric assessment method, proposed by Sauser et al. (2006), was used to assess the demand
control system. The SRL which uses the conventional TRL metric in conjunction with another
metric called the IRL to derive a SRL providing a slightly more comprehensive result which
indicated where priority should be given to increasing integration maturity. However
although the SRL provided a more comprehensive analysis of where the integration focus
should be some of the results were slightly counter intuitive leaving some doubt around the
accuracy or the interpretation of the TRL and IRL level definitions. The overall System
Readiness calculated at 0.86 indicates that the system is ready for operational use and
reinforces the decision to have deployed the demand control system. However, due to
practical challenges faced after deploying the demand control system into its intended
operational environment it is questionable how useful these metrics are and it is clear that
there are gaps within the methodology.
25
Integration is a complex subject and interpreting the TRL and IRL levels are not always easy.
Although the case study of the demand control system indicates that the IRL is capable of
revealing integration maturity apprehensions regardless of the high TRL levels of the
integrating technologies. There are however still a few uncertainties surrounding the IRL such
as during which level of integration should the IRL be applied, how IRL handles emergent
system behaviour, its inability to assess R&D effort or costs and schedules (Sauser et al. 2010).
It would seem reasonable to conclude that the metrics are not able to guarantee the complete
success of an integrated system.
26
Appendix A
Table 5: Hardware Technology Readiness Definitions, Descriptions, and Supporting Information. Source (DoD 2009)
27
Table 6: Hardware Technology Readiness Definitions, Descriptions, and Supporting Information. Source (DoD 2009)
28
`
29
Table 7: Additional Definitions of TRL Descriptive Terms. Source (DoD 2009)
30
Appendix B
Figure 15: Descriptive Requirements for Technology Readiness Assessment Document. Source (DoD 2009)
31
Bibliography
ABU-ZEID, N. S. & AL-SHAKARCHI, M. R. G. 2002. A Study of Load Management by Direct Control for Jordan's Electrical Power System. Science & Technology, 7.
ASLAM, W., SOBAN, M., AKHTAR, F. & ZAFFAR, N. A. 2015. Smart meters for industrial energy conservation and efficiency optimization in Pakistan: Scope, technology and applications. Renewable and Sustainable Energy Reviews, 44, 933-943.
AZIZIAN, N., SARKANI, S. & MAZZUCHI, T. A comprehensive review and analysis of maturity assessment approaches for improved decision support to achieve efficient defense acquisition. Proceedings of the World Congress on Engineering and Computer Science, 2009. 20-22.
BAGO, R. & CAMPOS, M. 2015. 16 - Smart meters for improved energy demand management: The Nordic experience. In: BESSÈDE, J.-L. (ed.) Eco-Friendly Innovation in Electricity Transmission and Distribution Networks. Oxford: Woodhead Publishing.
BILBRO, J. W. 2007. Systematic Assessment of the Program/Project Impacts of Technological Advancement and Insertion. George C. Marshall Space Flight Center.
BOLAT, S. 2014. Technology Readiness Level (TRL) math for innovative SMEs [Online]. Available: http://serkanbolat.com/2014/11/03/technology-readiness-level-trl-math-for-innovative-smes/ [Accessed 14 February 2015].
BUEDE, D. M. 2000. The engineering design of systems: models and methods, Wiley. CALMEYER, J. 2012. ZigBee as a technology platform for advanced metering infrastructure (AMI).
63rd AMEU Convention. CORNFORD, S. L. & SARSFIELD, L. Quantitative methods for maturing and infusing advanced
spacecraft technology. Aerospace Conference, 2004. Proceedings. 2004 IEEE, 6-13 March 2004 2004. 681 Vol.1.
DOD 2009. Technology Readiness. Assessment (TRA)Deskbook. DOD 2011. Department of Defense Technology Readiness Assessment (TRA) Guidance. EADY, F. 2005. Chapter 17 - A New Kid in Town Who Calls Himself ZigBee. In: EADY, F. (ed.)
Implementing 802.11 with Microcontrollers. Burlington: Newnes. MALIK, A. S. & AL MATA’NI, F. S. 2007. Load Management In Industrial Sector And Its Impact On
Power Planning – A Case Study Of Oman. International Conference on Communication, Computer & Power. Muscat.
MANKINS, J. C. 2002. Approaches to strategic research and technology (R&T) analysis and road mapping. Acta Astronautica, 51, 3-21.
MANKINS, J. C. 2009. Technology readiness assessments: A retrospective. Acta Astronautica, 65, 1216-1223.
PROFFITT, B. 2009. What Is Linux: An Overview of the Linux Operating System [Online]. Available: https://www.linux.com/learn/new-user-guides/376-linux-is-everywhere-an-overview-of-the-linux-operating-system [Accessed 15 February 2015].
SAUSER, B., GOVE, R., FORBES, E. & RAMIREZ-MARQUEZ, J. E. 2010. Integration maturity metrics: Development of an integration readiness level. Information, Knowledge, Systems Management, 9, 17-46.
SAUSER, B. & RAMIREZ-MARQUEZ, J. E. 2007. A System Maturity Index for Decision Support in Life Cycle Acquisition. DTIC Document.
SAUSER, B., VERMA, D., RAMIREZ-MARQUEZ, J. & GOVE, R. 2006. From TRL to SRL: The concept of systems readiness levels. Conference on Systems Engineering Research, Los Angeles, CA.
SHARIFF, F., RAHIM, N. A. & HEW, W. P. 2015. Zigbee-based data acquisition system for online monitoring of grid-connected photovoltaic system. Expert Systems with Applications, 42, 1730-1742.
32
SIT. 2010. System Readiness Level Calculator [Online]. Available: http://personal.stevens.edu/~bsauser/SysDML/Methods,_Processes,_Tools_files/index.htm.
SMITH, J. D. An Alternative to Technology Readiness Levels for Non-Developmental Item (NDI) Software. System Sciences, 2005. HICSS '05. Proceedings of the 38th Annual Hawaii International Conference on, 03-06 Jan. 2005 2005. 315a-315a.
TETLAY, A. & JOHN, P. 2010. Clarifying the Concepts of System Maturity, System Readiness and Capability Readiness through Case Studies.
VALERDI, R. & KOHL, R. J. 2004. An approach to technology risk management. Engineering Systems Division Symposium, 2004 Cambridge. MIT.