designing a maintenance support system interface:...
TRANSCRIPT
Linköpings universitet/Linköping University | Institutionen för datavetenskap / Department of Computer and
Information Science
30 credits, Master thesis | Cognitive Science
Spring term 2016 | ISRN: LIU-IDA/KOGVET-A--16/008--SE
Designing a Maintenance Support System
Interface: Using the Activity Checklist
Combined With the Sequence Model
Jonna Magnusson
Academic supervisor: Mattias Arvola
Supervisor at Saab: Mattias Holmström
Examinator: Arne Jönsson
ISRN: LIU-IDA/KOGVET-A--16/008--SE
Upphovsrätt
Detta dokument hålls tillgängligt på Internet – eller dess framtida ersättare – under 25 år från
publiceringsdatum under förutsättning att inga extraordinära omständigheter uppstår.
Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka kopior
för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervisning.
Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. All annan
användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten,
säkerheten och tillgängligheten finns lösningar av teknisk och administrativ art.
Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som
god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att
dokumentet ändras eller presenteras i sådan form eller i sådant sammanhang som är kränkande för
upphovsmannens litterära eller konstnärliga anseende eller egenart.
För ytterligare information om Linköping University Electronic Press se förlagets hemsida
http://www.ep.liu.se/.
Copyright
The publishers will keep this document online on the Internet – or its possible replacement – for a
period of 25 years starting from the date of publication barring exceptional circumstances.
The online availability of the document implies permanent permission for anyone to read, to
download, or to print out single copies for his/hers own use and to use it unchanged for non-
commercial research and educational purpose. Subsequent transfers of copyright cannot revoke
this permission. All other uses of the document are conditional upon the consent of the copyright
owner. The publisher has taken technical and administrative measures to assure authenticity,
security and accessibility.
According to intellectual property law the author has the right to be mentioned when his/her work
is accessed as described above and to be protected against infringement.
For additional information about the Linköping University Electronic Press and its procedures for
publication and for assurance of document integrity, please refer to its www home page:
http://www.ep.liu.se/.
© Jonna Magnusson
i
Abstract
This thesis aims to investigate how a new interface of the Maintenance Ground Support System
(MGSS) can be designed with insight generated from combining the Activity Checklist with the
Sequence Model. Through observation of an expert user, deficiencies and qualities of the system
should be raised. The combination of the activity checklist and the sequence model generated
fruitful insight over the deficiencies leading up to a troublesome event in the interface. A rich
understanding generated of what led up to a certain activity and how that action could be improved.
The activity checklist explains the users’ behavior and why they are. It provided beneficial
knowledge over the whole operation in the system and not just one functions. The insight from
these two methods set up a ground pillar for a solid design, where the deficiencies are improved
and the qualities are isolated from an untroublesome event. With the knowledge from the analysis
a prototype of a new interface was made in Microsoft Blend. The new interface was evaluated with
surrogate users at Saab AB. Though the result was not significant the result showed a higher SUS-
score and faster total mean time of the performing of the tasks. The result showed a trend with a
better usability and can be generalized and interpret that it would be the same output with real end
users.
ii
iii
Acknowledgement
This thesis would not have been done if there was not a good cooperation with Saab AB. A big
thank you to all that have been supporting me with your knowledge and inspiration. A special
thanks to my supervisor at Saab Mattias Holmström, for teaching me all about MGSS and all your
answers to my questions. I will also give a big thank you to Peter Bohnsack for the opportunity to
visit you at F17 in Ronneby, without the observation the knowledge would have been mostly
wrong! Mattias Arvola my academic supervisor at Linköpings University should also have a big
grateful thanks for the guidance throughout the thesis. You have given me inspiration and self-
confidence during this spring, thank you! The biggest thank you goes to my family and friends,
without you I would never have come this far, I love you all!
Linköping June 2016
Jonna Magnusson
iv
v
Contents
1 Introduction ........................................................................................................................... 1
1.1 Aim and purpose .............................................................................................................. 2
1.2 Delimits and limitation ..................................................................................................... 2
1.3 Saab AB ......................................................................................................................... 3
1.3.1 Maintenance Ground Support System - MGSS .............................................................. 3
1.3.2 MGSS walkthrough .................................................................................................... 4
2 Theoretical background ........................................................................................................... 9
2.1 Behavior of expert user ..................................................................................................... 9
2.2 Affordances ..................................................................................................................... 9
2.3 Signifier ........................................................................................................................ 10
2.4 The designers’ obligation ................................................................................................ 11
2.5 Activity theory ............................................................................................................... 11
2.5.1 Activity checklist ..................................................................................................... 13
2.6 The Sequence Model ...................................................................................................... 15
3 Pre-study ............................................................................................................................. 17
3.1 Needs analysis ............................................................................................................... 17
3.2 Observation ................................................................................................................... 17
3.2.1 Procedure ................................................................................................................ 18
3.2.2 Data analysis ........................................................................................................... 18
3.2.3 Observation summary ............................................................................................... 18
3.3 Result of the Activity Checklist ....................................................................................... 19
3.3.1 Decompositions of goals ........................................................................................... 19
3.3.2 Troubleshooting strategies ........................................................................................ 20
3.3.3 The use of artifacts ................................................................................................... 21
3.4 Result of the Sequence Model .......................................................................................... 22
3.5 Result of combining the Activity checklist and the Sequence Model ..................................... 25
4 Design process ..................................................................................................................... 27
4.1 Idea generation .............................................................................................................. 27
vi
4.1.1 Design decisions on theory adaption ........................................................................... 27
4.2 Design result ................................................................................................................. 28
4.2.1 Prototype walkthrough ............................................................................................. 28
4.3 Evaluation of the prototype ............................................................................................. 34
4.3.1 Measurements ......................................................................................................... 34
4.3.1.1 SUS ..................................................................................................................... 34
4.3.1.2 Time on Task ........................................................................................................ 35
4.3.2 Pilot study ............................................................................................................... 35
4.3.3 Procedure ................................................................................................................ 35
4.3.3.1 Participants ........................................................................................................... 36
4.3.3.2 Ethics .................................................................................................................. 36
4.4 Result of the evaluation .................................................................................................. 36
4.4.1 SUS-score ............................................................................................................... 37
4.4.2 Time on Task .......................................................................................................... 38
5 Discussion ........................................................................................................................... 41
5.1 Result ........................................................................................................................... 41
5.2 Interface ........................................................................................................................ 42
5.3 Method ......................................................................................................................... 43
5.4 Limitations .................................................................................................................... 45
5.5 Future research .............................................................................................................. 46
5.6 Implications .................................................................................................................. 46
References ............................................................................................................................. 49
Appendix ............................................................................................................................... 51
1
1 Introduction
Saab AB is a Swedish company developing aerospace and defense solutions. Their most famous
product is the fighter aircraft Griffin, which is a multirole aircraft. To keep a low cycle cost and a
high sustainability the knowledge about the wear of the aircraft is crucial. In order to generate a
longer lifecycle Griffin is using advanced techniques to provide the ground crew with insight and
understanding of the aircraft wellbeing. Over 6000 variables are logged every cycle to keep track
of the wear of the aircraft. There are a several types of cycles, for instance a ground cycle, when
the aircraft has started but has not left the ground, or a flight cycle which is a logged cycle when
the aircraft is in the air. The variables that are recorded are for instance flight hours, height, speed,
pressure and temperature. These variables are later analyzed by the ground crew staff through their
maintenance system called: Maintenance Ground Support System (MGSS). This system, explained
in detail later in the thesis, provide the ground crew staff with information that helps them improve
the wellbeing of the aircraft. The system enables the staff to perform advance troubleshooting in
the logged data. If any abnormal events have occurred during a cycle MGSS present the fault codes
that represent the appropriate actions for the ground crew staff. The priority of the maintenance
system is to offer a longer mean time between failures and decrease the mean time to repair (Saab,
2016). The results from the analyzed data contributes to a secure and well-being aircraft with low
cost over a long lifetime.
Over the years MGSS has been updated with new functions but the interface design has not been
in focus during this time. The existing interface of MGSS is perceived to be too complicated for
easy control and it is difficult to get an easy overview of the data. Systems that are perceived as
having a poor interface design have issues with loss of efficiency, productivity, and most
importantly, safety (Stone, 2005 p.9-10). The interface should encourage users to carry out their
required tasks in an easy and engaging interaction (Stone, 2005 p.6). In order to create a well-
designed interface of MGSS knowledge about users’ actions and intuition are crucial. This
knowledge will affect the interface design and let the system support rather than limit the users
(Preece, Rogers & Sharp, 2016 p.410). Not only is it crucial to know which actions and activities
that are needed, but an understanding of how these activities are related to each other in order to
see the entire development process is equally important (ibid.). This thesis will investigate if it is
possible to get an understanding of the users’ actions and their relations by combining the Activity
2
Checklist with the Sequence Model. The Activity Checklist is an analytic tool used to get a better
understanding of the user’s activity in the design and provides a guidance and structure for
empirical work on design (Kaptelinin et al., 1999). In order to analyze how the actions are related
and how they unfold over time the actions are modelled, step by step, and analyzed with a Sequence
Model (Beyer & Holtzblatt, 1998 p.89).
The goal is to examine if the troubleshooting of Griffin can be more efficient with a new interface.
The design will focus on usability and not just adding additional functionality, as in the past, to
create a quicker and smoother workflow for the user (Stickdorn & Schneider, 2011 p.82-83). Higher
usability would provide faster and more confident troubleshooting and reduce ambiguity in the
interface for the ground crew staff.
1.1 Aim and purpose
The aim of this thesis is to develop a prototype of a new interface of MGSS, by using knowledge
and insights from the Activity Checklist and the Sequence Model. Can these two method be
combined and generate useful knowledge in the field of design. The main questions in the thesis
are the following:
● How can an interface for maintenance system for aircraft be designed to increase
the usability?
○ Which are the qualities and deficiencies in the existing system?
○ In which way can desirable qualities be improved or implemented?
○ How can the daily work routine be more efficient for the user?
● How can a combination between the Activity Checklist and the Sequence Model
contribute when analyzing a workflow in an interface?
1.2 Delimits and limitation
MGSS is a complex system, to consider the system as a whole in a prototype was not possible.
Delimitations of the thesis were therefore crucial. Agreed upon with the supervisor from Saab AB,
the focus of the thesis should mainly consider the primary use of a daily work routine by the ground
crew staff.
3
End-user of MGSS is not the customer. The Swedish Defense Materiel (Swe. Försvarets
materielverk, FMV) supplies the Swedish armed forces with equipment, hence FMV is the
customer to Saab, but they are not end-users. For this thesis one visit to one of the end-user was
possible, to do an observation. Going back to test and evaluate the prototype was not possible. The
prototype is therefore only evaluated with surrogate users, the development team of MGSS at Saab
AB.
1.3 Saab AB
Saab AB was founded in 1937 and is a Swedish aerospace and defense company. Today they
administrate solutions in fields of air, land, naval, security and commercial aeronautics (Saab,
2016). Saabs main focus is the aircraft Griffin which is a multirole fighter aircraft capable of
performing different missions of air-to-air, air-to-surface and reconnaissance. Four versions, one
launching 2016, have been produced over the years. The focus for this thesis is to redesign the
complex maintenance system for Griffin called Maintenance Ground Support System (MGSS).
1.3.1 Maintenance Ground Support System - MGSS
MGSS is mainly used by the ground crew staff for Griffin to conduct advanced fault analysis after
a cycle. During a cycle over 4000 variables of data are recorded in Griffin, a few examples of
variables are flight hours, height, speed, pressure and temperature. Analysis of these data points
provides control over the aircrafts well-being. For instance, after a certain time of flight hours some
components need service or replacing. Knowing the wear of the aircraft creates control and the
ability to achieve low operating costs, and a longer life cycle by only replacing necessary parts.
The data is recorded on a data transfer unit (DTU) placed in the cockpit during a cycle. The DTU
is a portable device, and after a cycle the DTU is disconnected from the cockpit and connected into
a computer with MGSS. The process is similar to using an usb-stick for transferring data between
two computers, though the DTU is bigger and more robust. When the transfer is finished the system
will unpack the files and provide the ground crew users with big datasets from the cycle. MGSS is
mainly used after a cycle with Griffin to analyze the data and confirm that the aircraft is okay to
perform another cycle. If a fault event occurs during a cycle the pilot receives information about it
directly in the cockpit in real time. All fault events can later be seen in MGSS and the ground crew
staff can troubleshoot the events. After an analysis the staff can verify if the aircraft is okay or if
something needs to be fixed. Some fault events are more common than others and the staff learns
from experience what these events mean and what action they warrant. Sometimes an unusual code
4
appears and further troubleshooting to find what the code means is needed. Today, this is
complicated since there is no easy way for the user to know where to find correct information about
fault codes. Fortunately, majority of cycles does not have any fault events. The data however still
needs to be analyzed by one of the ground crew staff in order to make sure that the aircraft is
functional.
1.3.2 MGSS walkthrough
In this section MGSS will be presented and explained along with pictures from the interface. MGSS
can contain secret information and cannot presented with an accurate walkthrough, the pictures in
this thesis are approved by Saab AB.
MGSS is designed as a traditional window application, instead of changing the view of the interface
a new window opens up containing new information. The start view of MGSS consist a lot of
information of all cycles (Figure 1), one row represents one cycle. A row consists of 19 columns
containing summed up information from the cycle such as time, cycle type and station. Farthest to
the left is a box showing the ID-number of the different aircrafts, allowing the user to only show
data from one select aircraft. In the start view there is also information about the different
administration actions, six small columns indicate different information depending on the letter. In
Figure 1 there are a few rows presented in a red color which means that the cycle has been created
manually. If a cycle has not been logged on the DTU for some reason, the ground crew staff creates
a cycle manually so that at least known variables can be documented, such as flight hours.
5
Figure 1. Start view of MGSS
Column number six is called “Fault Events” and contains the events that occurred during the cycle.
As the system is today there are fault events in almost every cycle, although they are not critical.
The user need to analyze to interpret the criticality of the events and to get more information about
a cycle the user needs to double click on the corresponding row. Then, because of the traditional
window application, a new window opens up containing more information from the cycle (Figure
2).
6
Figure 2. Presenting the new window displaying cycle information
The new window in Figure 2 includes nine tabs, each showing different information from the cycle.
The start view of this pop up window is the second tab Events, containing information of data
collected over time in a cycle presented in a sequence of events. The user gets information of when
the event happened, the event number and name of the fault event. It also provides the user with
the estimated criticality of the fault event, which type of event and a MG number that represent to
which material group the event belongs to. A trained eye can in this view see the event number and
interpret the fault event from that, but sometimes an unfamiliar event number occurs and the user
need to troubleshoot further to get explanation of what cause the event. For instance, at the time
260s in Figure 2 there has occurred a fault event with event number 10, with the name ‘Fault Report
DTU’, and has an Event Type of ‘Fault Report’ and MG number 7. All events with the event type
‘Fault Report’ is listed under the tab ‘Fault Reports’ (Figure 3).
7
Figure 3. The same window as in Figure 2, showing another tab for more information
of a fault event from the cycle
From the view in Figure 3, the user gets the same information as in the previous view in the left
section and new information to the right. Here are three tabs which contains additional information.
In order to interpret the information shown to the right a separate document called Aircraft
Maintenance Publication (AMP) is needed. In this version of MGSS the user does not get any
further information of what the fault event means, it is only providing codes that can be interpreted
with help of the AMP.
8
9
2 Theoretical background
This chapter explains the theoretical framework and background needed to understand the result in
the thesis.
2.1 Behavior of expert user
From start all individuals are at some point novices, but expertise develops over time as a person
matures with training and education in a certain field (Cross, 2004). During the development from
a novice to an expert something happens. Studies has shown that there are differences in how
experts and novices categorizes problems and how they reflect about them have been shown. It
also has shown differences in the brain activity between novices and experts (Sheridan & Reingold,
2014). Further results shown differences in perceptual skills while playing chess, an expert can
distinguish between relevant information faster and therefor spending more visual time on these
areas (ibid.). The novices tend to focus on shallow parts of the problems, whereas the experts tend
to study problems more profound (Chi, Feltovich, & Glaser, 1981). An expert is aware of
limitations and the criticality to distinguish between important from the less important problem.
The experts take advantage of explicit artefacts and restructure the problem into smaller sub
problems to easier solve the task at hand. Additionally, experts tend to be more capable to
generalize knowledge and information in a higher cognitive process than novices (Cross, 2004).
Because of this, experts can recognize and detect underlying principles, rather than focusing on the
shallow parts of the system (Cross, 2004: Ball et al., 2004). The novices on the other hand tend to
use more general principles (Ball, Ormerod, & Morley, 2004).
2.2 Affordances
The term affordance is coined by James J. Gibson within his theory of ecological approach to
perception. He argued that a physical object in the environment contain all information about how
an agent, for instance a person, can interact with it. Affordances refers to the relationship between
an object and a person, and it implies all the properties that the object has and all the possible
actions. In short, affordances determine what actions are possible (Norman, 2013. p.14, 145). The
relationship varies between individuals, the same object can have different affordance to different
people. A knee-high firm surface does have the properties to afford sitting on, but the knee-high
height is however individual, hence is the relationship of affordances (Gibson, 1979. p. 120).
10
Norman introduced the term affordances into the field of human-computer interaction (HMI). For
designers’ affordances provides strong clues to the operations of things and can therefore be fruitful
in design (Norman, 2013 p.12). For example, a door with a flat plate mounted on it affords pushing
and a door with a vertical door handles afford pulling. Users should be able to figure out what
actions are possible without the need for labels or instruction, for instance a label at the door saying
‘Push’ or ‘Pull’. When talking about affordance within interface design, a button that is designed
to protrude from the display do have affordance to pushing and not moving or editing (Gaver,
1991). When taking advantage of affordances, the user knows what to do by just looking and no
further instructions are required. The term was loved in the field of design, and became used to
almost everything, but according to Norman (2013) the usage of the word had nothing to do with
its original definition. An indication of where the user should touch in the interface is not an
affordance, it is just a way of communication where to do the touching (ibid.). Affordance of
touching is existing on the entire screen. So for example designing a circle in the interface is to
signify where the touch should take place, and that is not the same thing as saying what actions are
possible (ibid.). There have been discussions for a further development of the concept, introducing
such a powerful concept but limiting its scope is a failure of its full potential (Kaptelinin & Nardi,
2012). To disentangle the ambiguity Norman coined a new term, Signifier (Norman, 2013 p.14).
2.3 Signifier
Instead of using the ambiguous term affordances, Norman proposed to use the term signifier. A
signifier should not replace affordance but rather be seen as a complement, designers need both of
them. Affordances determine which actions are possible (e.g. display afford touching), and
signifiers communicate where the action should take place (e.g. the circle indicate where to touch)
(Norman, 2013 p.14). Users search for clues and signs in the interface that can help them understand
what to do. A signifier should provide the user with the information that is needed. A signifier does
not need to be intentional, it could be accidental for instance a visible trail in a field after people
had walked (ibid.).
11
2.4 The designers’ obligation
What the term is called is not crucial for the designer nor the users, the important thing is how the
users interpret the designers’ system. There is no need to talk about putting affordances and
signifiers in the interface, but rather of how the system speaks for the designer. Getting the interface
to talk for the designer is a way of metacommunication (de Souza, 2005 p.90). The designers are
trying to represent their understanding in a simplified way, so the users can determine what they
mean in the system, it is a one-shot message from the designers to the users (ibid.).
Norman underline the significance of cognition as central to the users’ activity, and de Souza
complement that we need to emphasize the fact that cognition is also central to the designers’
activity (de Souza, 2005 p.89). The fact that the designers says “I put an affordance there”, to
describe for example a circle to indicate where to touch is a conceptual mistake (Norman, 2013
p.13; de Souza, 2005 p.91). Whether the term is called affordances, signifiers or the designers’
deputy, it is the designers’ obligation to make the interface understandable for its users.
The design need to have clues to enhance the likelihood for the users to make the correct
interpretation. They need to have the not only the answer of “where can I go next”, but rather,
“where is the most beneficial next step to go”. The interface must follow the natural evolution,
rather than random choices of the users (Spence, 2007 p.160). If the next step is not clear for the
user, the interpretation involves a high-order cognitive process.
2.5 Activity theory
Vygostsky, a Russian psychologist, presented that human thinking is not a single mental entity set
apart from the outer world, it is a product of social interaction (Garbin, 2002 p.40). Our cognition
is situated within our social and cultural context. Activity theory is a general conceptual approach
that explains the activity which is a human interaction with the objective reality. An activity can
be anything from shopping in a store to working with a computer task. To understand the human
activity requires a complex analysis (Kaptelinin, Nardi & Macaulay, 1999). The basic unit of this
theory is an activity, which is composed of subject, object, action and operations. A subject consists
of an individual or a group engaged in an activity. An object, or sometimes referred as a motive, is
something pursued by the subject. It can be a material thing, but also more abstract like a plan, as
long as it can manipulate the individual's activity. The object is not a permanent structure, it can
12
change during the activity. An action is a goal-directed process that must be used to complete the
object. An operation refers to all the actions performed by an individual. According to activity
theory there are different artifacts employed in an activity. Individuals mediate their relationship
with the world through these artifacts (Garbin, 2002 p.40). Figure 4 shows a triangle that is
representing the relationship between subject, object and artifact.
Figure 4. The basic triangle of the Activity Theory
(Source: The Cognitive Use of Artifacts in Cooperative Process Management: Rescue Management and Underground
Line Control. (Linköping Studies in Arts and Science, 2002) p.40 Figure 3.1)
Activity theory pose that there are differences between internal and external activities (Kaptelinin
et al., 1999). As mentioned before cognition cannot be analyzed separately in isolation from the
external activities. There is a constant transformation between external and internal that stands as
the basis of cognition (ibid.). These transformations have two directions. The first is when external
activities are transferred to an internal, and is called internalization. An example of this is a person
going from having to look at the keyboard while writing on a computer to being able to write
without having to look. The second direction, externalization, is when internal activities is
transferred into external ones. This could be when a calculation is too hard for mental calculation,
and need to written down or solved with a calculator.
Activity theory have developed over the years and several branches of it has emerged. The original
three connection triangle in Figure 4 was deemed too simple to fulfill the needs of a theory
explaining the relationship between an individual and the environment in an activity. The Finnish
researcher Engström extended the triangle and developed an activity theory toward organizational
change and development (Garbin, 2002 p.41). He complemented it with more connection, by
including communities, social roles, and the division of labour (Figure 5). According to this
expanded view, interaction between a subject and the community is mediated by social rules.
Similarly, the interaction between community and an object is mediated through division of labour
(Garbin, 2002 p.41).
13
Figure 5. The expanded triangle of Activity Theory
(Source: The Cognitive Use of Artifacts in Cooperative Process Management: Rescue Management and Underground Line Control. (Linköping Studies in Arts and Science, 2002) p.41 Figure 3.2)
The concepts and the theoretical foundation of activity theory have been further developed and
oriented into the domains of Human Computer Interaction (HCI). This context makes it possible
for activity theory to be a powerful multi-disciplinary framework within cognitive, social,
technical, and organizational aspects and how these can be studied together (Garbis, 2002 p.41).
The activity theory however does not provide a designer with a solution that can be applied on
specific problems (Kaptelinin et al., 1999). The activity theory has been seen as a promising
framework to use when analyzing work settings of complex technologies, but have been found as
an extensive and difficult to use. It helps the researcher to orient thoughts, but can for instance be
too abstract when it comes to working on a design. For this reason, the Activity Checklist was
introduced, an analytic tool of activity theory (ibid.).
2.5.1 Activity checklist
The activity checklist is an analytical tool established from the activity theory to get a better
understanding of design activities, and is often used during the early design phases for a system, or
for evaluating existing systems (Kaptelinin et al., 1999). It provides guidance and structure for
empirical work on design and evaluation of computer systems in workplaces (Garbis, 2002 p.41).
Further the checklist can help the designer to identify the most important issues and potential
trouble spots (Kaptelinin et al., 1999). The checklist exists in two slightly different versions: the
evaluation version, and the design version, both used to help organize sets of item that potentially
influence the use of a system. This thesis used the design version from 1999 by Kaptelinin, Nardi
14
and Macaulay. The checklist has potential to identify the most important issues or trouble spots
that designers can encounter. It starts general, by examining the whole area of interest, then
focusing on identifying areas in as much depth as possible. When those areas are established they
are analyzed with a breadth-first consideration, followed by narrowing it down and isolate the
problems (Kaptelinin et al., 1999). This breadth of coverage in the checklist will help to ensure that
designers do not miss any areas that might be important to the understanding of the systems they
are working on. The checklist reflects on the core of activity theory but is intended to apply on the
analysis of how people are using, or will use, a system. For this reason, the activity checklist had
divided into four perspectives (ibid.): Means and Ends: Environment: Learning, Cognition and
Articulation and: Development.
Means and Ends:
This perspective involves to which degree the technology facilitates and constraints the
achievements and how it affects obtainment of the users’ goals.
Environment:
This perspective considers that all human lives in a social and cultural world. They achieve their
goals by active transformation of objects through activity. It is an integration target technology
with requirements, artifacts, and social rules of the environment.
Learning, cognition and articulation:
This perspective involves how artifacts affects the mutual transformation between internal and
external activities, which also can be transferred into each other.
Development:
Activities undergo constant developmental transformation. All practice is a result of certain
developments under certain conditions.
The checklist is not a tool to be used in isolation, it is an appendage to the basic principle of activity
theory. Furthermore, the activity theory itself does not provide straightforward solutions that can
be applied directly on specific problems (Kaptelinin et al., 1999). The activity checklist interprets
as top-down approach, and is used to get richer knowledge. It is complemented with a bottom-up
approach, hence introducing the Sequence Model below.
15
2.6 The Sequence Model
One way to make an analysis from the observed data collected during the visit is to modelling it,
for instance with a work model. Work models are built to present and to analyze individual
perspectives of the observed actions. They are not intended to represent the whole organization and
everything that one person does (Beyer & Holtzblatt, 1998 p.89). The model used in this thesis is
called The Sequence Model, and analyses the steps people take to accomplish an intent in work
which unfolds over time - step by step. The action people take are not random they happen for a
reason and individuals do them for a purpose. These action reveal their strategy, and what intention
they have and what matters. Understanding the individual's real intentions are a key to improve
their work practice (ibid.). With this knowledge one can redesign, modify, and remove steps as
long as the user can achieve their underlying intentions. When changing the work tasks, the system
need to support all intentions concealed in the work, the goal is to make the work process more
efficient. A sequence model represents the steps that unfolds over time during work, it becomes a
sequence of actions - steps to achieve an intention. The visualization of the sequence model become
a detailed map to the work that is actually done. In this way the designer can easier see what actions
needed change, and what these changes can contribute with (ibid.).
A sequence model starts with the overall intent of work, and this intention triggers new actions in
the forms of the work steps. A list of all the steps that follows after each other are visualized, at the
detailed level of the collected data. Steps that are causing problems are labelled with a lightning
bolt. When the work of an individual is modeled the sequence model does not attempt to show
pattern or repetition, they will be identified during consolidation.
16
17
3 Pre-study
This section contains the methods and result from the pre-study. The observation and the procedure
are introduced first, followed by the result of both the Activity Checklist and the Sequence Model.
3.1 Needs analysis
In order to design a customized interface, an understanding of the users’ tasks and goals in the
system provides a solid base to stand on during the development. An understanding of which
functions they want and need in the system are crucial (Allen, 1996 p.55). The requirements in a
needs analysis elaborates from data collected of how the system is used by real users. The aim of
a needs analysis is to create a comprehensive understanding of the users’ needs and issues which
lays the foundation during the design development and design decisions (ibid.). In this thesis an
observation at F17 was made to collect as much information as possible of the daily usage of the
system for making a need analysis. The need analysis can be seen as a ground pillar of the
development throughout the entire design process.
3.2 Observation
To gather data of MGSS’s daily usage observations was made for two days at F17 in Ronneby
Sweden. F17 has since 2004 had the responsibility for The Swedish Air Force Rapid Reaction Unit
(SWAFRAP) and they always have a group of Griffin ready to fly. After an aircraft has been on an
operation the cycle needs to be analyzed to see if the aircraft is approved for another mission.
Although the pilot gets all fault events in the cockpit that appears during the cycle, if any critical
event occurs the pilot can tell the ground crew what happen after landing. A crucial event may be
analyzed directly, otherwise the fault search is in no hurry.
Through a contextual inquiry detailed perspectives can be elaborated on how the system is used.
During the observation interviews were simultaneously held with the expert. The interviews were
based on a semi-structured approach, hence it is difficult to follow a strict questionnaire
simultaneous as the situation develop. Observations and interviews are in this inquiry combined to
collect as detailed perspective over the work process or daily activity as possible (Arvola, 2014
p.50). To learn as much as possible during the observation an apprentice approach was applied. In
this approach the observer aims to learn from experts, like an apprentice learns skill from a master.
18
The observer wants to learn about the users’ work, and together define the functions that the system
need to address (Beyer & Holtzblatt, 1995).
3.2.1 Procedure
The observation lasted for two days by following the MGSS expert who has the responsibility for
the data analysis in MGSS at F17. The first day started with a guided tour of the wing, after that
the work with MGSS started. The expert started the daily work routine and simultaneous explained
all events and actions. Though the observation was made from an apprentice approach taking notes
at the same time was not always easy. The learning and doing approach took more time so there
was little time to the sit down and passively observe approach and taking notes. Notes were written
during the observation when the moment allowed it, or sometimes it was asked to get a moment of
silence to take notes. In order to get the information as organized as possible, the notes were
rewritten and analyzed once again later that day. For confirmations about the notes, the expert was
asked if the notes were understood correctly. If there was some misunderstandings or ambiguity
the expert went through the descriptions one more time. Detailed descriptions were always
encouraged during explanations or demonstrations of functions in MGSS. On the second day of
the observation it was the apprentice turn to make the daily routine work, with support from the
expert.
3.2.2 Data analysis
All notes during the observation was made in a notebook size A5 with lines. To get the notes better
organized and analyzed they were rewritten and analyzed in a mediated action sheet and a persona
sheet (Arvola, 2013). The persona sheet can be modified and function as a checklist for developing
themes during a contextual inquiry. This was made to compensate for the risks of losing important
data. To facilitate the analysis a draft timeline over the users’ work routines was created
continuously as the observation proceeded. To help the analysis further a drawn map analysis made
with notes of the user’s artifacts and what they were used to. The data was further analyzed through
the Activity Checklist and the Sequence Model.
3.2.3 Observation summary
The thesis focus on the daily work routine within MGSS, so the observation had the same focus.
There are probably differences from one MGSS user to another. In the observed expert’s daily
routine all cycles from the day before are analyzed the first thing in the morning, at least all cycles
19
that the pilot did not think was a critically fault event during the cycle. All events that the pilot
interpret as critical are analyzed as fast as possible, the regularity is though that no critically events
occur. Every cycle is logged on a stick in the cockpit called a DTU, after a cycle is finished the
DTU’s are stored in a safety locker waiting to be analyzed. Every cycle has an attached log sheet,
containing information from the cycle. On the log sheet the pilot has noted the time of the cycles
and how many cycles has been done. The information is confirmed through a signature by the
turnaround service manager and the pilot. The DTU contains data of all cycles that has not been
transferred into MGSS. When the DTU is inserted in a computer all data is transferred into MGSS
and the DTU clears automatically. It is rare but sometimes the cycle has not been logged on the
DTU. In these cases, the MGSS leader need to create a cycle manually from the information in the
log sheet. The MGSS expert stated that he needs to create a cycle manually every two months or
more rarely.
After the expert had transferred all data from the DTU’s into MGSS troubleshooting started. The
expert took one log sheet at the time, and analyzed all the cycles from that log sheet. To get the
correct information the expert chose to see data only from the current aircraft. The expert then
interpreted the fault events if there was something unfamiliar with the presented codes. The expert
does have a lot of the codes memorized and knows how interpret them. If the code is unfamiliar to
the expert a further analysis and troubleshoot in MGSS is needed. If the cycle was free from fault
events the expert signed the log sheet and moved on to another log sheet and started over. After all
cycles had been analyzed the data was exported into another system that saves the data for a longer
period than MGSS.
3.3 Result of the Activity Checklist
The activity checklist results in three categories, presented below, that was analyzed deeper. These
three categories together with the sequence forms the basis of the requirement that was used during
the design process.
3.3.1 Decompositions of goals
When talking about the goals with MGSS one is roughly talking about transferring data into MGSS,
make a fault analysis and then export the data. To understand what the user wants and need in the
system this big picture must be narrowed down into sub goals, which are presented in more detailed
in the sequence model.
20
A major sub goal in the system was the comparing the information on the log sheet and the data in
MGSS to ensure that they matched. This was the first thing the expert did, was to check the log
sheet to see how many cycles had been done, compered the information in MGSS, and then started
the fault search of these cycles. The expert thought it was much easier to select and sort the cycles
by aircraft name to get an overview. Then he knew that he cannot be mistake the data with another
aircraft. This activity seemed very important for helping the user to feel like they have control over
the situation, and became a requirement for the design process.
In the situations where the DTU did not log a cycle the user must create a cycle manually with the
information from the log sheet. The data is needed later for further analysis over the aircraft, such
as flight hour for the ability to calculate the wear of the material. The user must be able to
accomplish this activity, which lead it to a requirement.
Different kinds of exports are needed for various addressees, though MGSS sort out relevant
information on its own depending on which export is selected. The participant in this thesis did,
some exports more frequently like once a day and others once a month.
3.3.2 Troubleshooting strategies
If there is a fault event during a cycle it is tagged with codes. For instance, a fault event is tagged
with a MG number, an event number and also given a binary code. As it is today MGSS just shows
the user a fault event with different tags, but it does not give the user any information of what these
tags mean. To interpret the tagged codes, they use an external document containing all information
needed. Recently F17 gained access to have two screens, before they needed to tab between the
views on one screen. The participant thought this was a considerable improvement.
The tags from a fault event are represented with different tags, the MG number represent a material
group and the binary code can be translated through the external manual into errors. Knowing all
fault events is not possible, but an expert user learns the fault events that are more common. This
is an example of when an action has become internalize, as mentioned above within the activity
theory.
21
The user at F17 have on own initiative developed a cheat sheet with all of the most frequently fault
codes. They are using this cheat sheet as an external memory but also to avoid the complex
troubleshooting if it is possible. The cheat sheet however belongs by itself in next category the use
of artifacts.
3.3.3 The use of artifacts
To facilitate the fault analysis, the users at F17 use some artifacts. For instance, they use their
homemade cheat sheet mentioned above that contains a summary of the most frequently occurring
error codes. This cheat sheet was a printed on a piece of paper and placed in the physical regular
manual. The physical manual is almost the same as the digital AMP on the computer, but not as
complex. Most of the users do not use the physical manual only the cheat sheet and the manual on
the computer.
Another artifact, not only affected by the MGSS user, but also the pilot and the clarifying leader is
the log sheet. As mentioned above the pilot take some notes from the route, like aircraft number
and time in the air, and then sign it. The MGSS user compares this information with the data in
MGSS and confirms it is correct. Then the user also signs the log sheet confirming the information
is correct. For an easy overview of the information the participant chooses to only look at the
aircraft that matched the log sheet. To simplify and minimize the risk of looking at another aircraft
data, therefore it is a requirement to have the ability to sort out and just look at data from one
aircraft.
Another interesting artifact the participant had was a board on the wall in the workroom that helps
to keep track of all the different exports. After an export the user signed the whiteboard with which
export he had done and when. The whiteboard is an organizer over all the exports that are
distributed over time. In the existing interface of MGSS there is no information about which of the
different exports that had been done, nor when the latest export was. Feedback of the exports
became a requirement for helping the user by taking away concerns and doubt about which and
when the latest export was made. The user should in the new interface be able to find information
about the different exports, when it was done and which of the different exports are done or not.
22
3.4 Result of the Sequence Model
To get a more detailed analysis of the results from the activity checklist the data was also modelled
through a sequence model, described earlier. This to generate a better understanding of the
workflow over the daily work routines. For being able to create a sequence model all activity on a
workday was broken down into smaller steps of actions. From these smaller steps and with the
knowledge from the activity checklist sub goals appeared, and they are in organized in the sequence
model as they appear over time. The sequence model generates a further explanation of how crucial
every step is for the result of this thesis. Figure 6 - 8 are showing the result of the sequence model
analysis.
Figure 6. Result from the Sequence Model of the experts’ work routine, presenting the start of the morning routine
23
Figure 7. The routine of the daily analysis routine
24
Figure 8. The work process if any fault event had occurred during a cycle
25
3.5 Result of combining the Activity checklist and the Sequence
Model
A requirement specification was made from the data collected and analyzed from the observation.
Using the activity checklist and the sequence model combined gave a deeper insight of the sub
goals. It provided information of when they occurred in the daily routine and which steps and
activities that precede and follow them as well as how they may be effected by changes. Further, it
lets the designer see what changes does to related sub goals, an improvement for one step may
influence the outcome of other sub goals. The following requirements stand as a ground pillar
during the design process and has developed through the analysis:
● Have the ability to see information from one specific aircraft
● Clearly see fault events from a cycle
● Get a simple overview
● Export of data, through different exports
● Get feedback on which exports has been done and when
● Being able to create a cycle manually
26
27
4 Design process
In this section the prototype will be presented, along with explanation of the idea generation and
the design decisions.
4.1 Idea generation
The idea generation started from the need analysis, these aspects are important to consider during
the sketching. During the design development all of the requirements are not needed to be
considered in every sketch. The requirements are a good start to have them in mind, but they should
not disturb the creativity. The sequence model contributes with knowledge to consider for the
sketching as well, what actions that are important and in what order they are needed to appear in.
4.1.1 Design decisions on theory adaption
One major design decision was to take advantage of the moto for the design in the cockpit on
Griffin: Don’t need, Don’t Show. The current interface of MGSS do not have this mentality, it
shows all information that the user can be interested in at some point. This leads to an interface
containing a lot of data that is presented all at once, displayed before in Figure 1 – 3.
To show all information at once is not necessary in daily routine work, for any user. Even an expert
users of the system get overwhelmed with all the information which makes it hard to interpret, even
though expert users can distinguish relevant information from irrelevant better (Cross, 2004).
Making a clear distinction between new cycles from an already analyzed cycle is one improvement
that will remove the need to present all information at the same time in the interface. Splitting the
interface into two views will also help all users, experts as novices, to not be overwhelmed by the
interface. A cleaner interface would be simpler to interpret and also provide a more efficient usage.
The actions of a daily work routine were shown to be fairly straightforward in the sequence model
(Figures 6 – 8), and so should the actions in the interface be as well. An interface with the right
clues, affordances, can be as straightforward as the natural work flow. In an efficient interface the
user should be able to figure out which action is the next step to take (Gaver, 1991). The interface
must follow the natural evaluation, as shown in the sequence model, rather than the user’s random
28
search for the right next step (Spence, 2007 p.160). An easier interpretation involves lower cognitive
processes.
One other design decision that went back and forth was the question to have feedforward and/or
feedback on the dates of the exports visible in the interface. The exports are one of the significant
actions of the daily work routine. The routine of the exports probably differs a lot from place to
place, due to their work culture. The user in this thesis did one of the different exports every day,
other exports once a week. The unawareness of how other users act makes it difficult to generalize
the knowledge from the observation. This led to a decision to only using feedback in the interface,
though the feedforward probably differs too much between users.
The existing system do not have a characteristic color that the prototype need to follow. The color
of the system was an easy, and yet rather difficult decision. The argument of using a grey scale
throughout the entire interface are that grey is timeless. Using a background of grey and black text
is not too distinct to the eye, as black text on white background can be. A decision was made to use
different shades of grey to distinguish different areas in the interface. All boxes that are interactive
are in a darker grey, if they contain information with texts and so on the areas is in a brighter grey
to make it easier to read.
4.2 Design result
This section introduces the prototype and explains the design in the new interface. One
improvement that is presented is the possibility to choose between two different views, one simple-
and one advanced view. Further, introducing the interface as a whole with explanation. The
prototype is made with Microsoft Expression Blend 2013. Microsoft Blend is a production tool
created by Microsoft and is a part of Visual studio. Blend is a design tool suited to create graphical
interfaces. It produces Windows Presentation Foundation (WPF) applications, and is represented
in XAML.
4.2.1 Prototype walkthrough
During the observation it was noted that very few of MGSS’s functions was used on a daily basis.
The routine had an obvious sequential process, especially when no fault event appeared during a
cycle. To make the interface cleaner and more convenient to use the prototype has two views that
29
the user can choose between, one simple- and one advanced view (Figure 9). MGSS need to retain
all the advanced settings, but they do not need to be presented all at once. The idea to have a simple
view helps the user to easily do the daily routine; to check the cycle for any fault events, see what
the fault events means and the ability to do different exports. If the user would need to do a deeper
analysis or use some other functions they just need to switch to the advanced view. Two views
provide a simpler interface that is not overwhelming, which contributes with an easier overview
for the user.
Figure 9. The user can change between the simple and the advanced view in the upper right corner
In the beginning of the analysis the data from the DTU need to be transferred into MGSS. This is
done in same way as before, by docking the DTU into the computer and retrieve all files. Figure
10 shows the start view of the system after data is transferred into MGSS. Some qualities founded
in the observation are retained in the prototype. For instance, the ability to select and to see data
from only one aircraft is retained. However, in the prototype there is more focus on the ability to
choose between the aircraft than it is in existing MGSS. This also provides with a naturally distinct
bridge between the selected aircraft and the information to help the user know what cycles to
expect.
30
Figure 10. The start view of the prototype
With less information in the start view the prototype gives a more comprehensible overview, first
the distinction between the new cycles and the old ones. The new cycles are presented in first box
of the tab name “Events”, which is a tab connected to a box containing cycle history. The current
interface present both new and old cycles together in the start view with nothing to separate them
except the date tag. Another new feature in the start view is the indicators of the fault events.
Additionally, the current version represents fault events or alarms in text in one column at start
view. However, in the new prototype the information is summed up and categorized by its
criticality presented with three different symbols. The green square represents normal events that
are supposed to be logged. The yellow circle indicates events that are not critical, but needs to be
noticed. The red triangle indicates critical events that often needs to be dealt with.
31
Figure 11. Information from one selected aircraft
In Figure 11 only information from the aircraft 222 is selected and viewed. The button that is
selected has become an arrow indicating for the user what to expect. The user clicks on a cycle to
get more information about it, this opens a row that contains events in sequential order as they
appear over time. For instance, the cycle in Figure 12 has no fault events, but all the other non-
critical events are presented as they occurred over time. When a fault event occurs it will be
displayed in like Figure 13, the events are marked with a red triangle and the given event number.
32
Figure 12. When a cycle is selected all events from it are shown
Figure 13. A cycle where two critical fault event has occurred
33
An expert user automatically understands how to interpret the most frequently occurring event
numbers, and do not need more information about the given fault. Using the three new symbols is
a way to peel the interface by removing unnecessary perception input. If a fault event is critical it
will be marked by the red triangle, and the user do not need to interpret the criticality of a code by
itself. If an unfamiliar fault event number appears, the user can now search for more information
directly in the system, by using the search box to the right in the interface (Figure 13).
When the fault analysis is done and the cycles confirmed, the data needs to be transferred into
another system. As mentioned previously, there are different exports needed to be done for different
systems and purposes. In the existing interface of MGSS the exports are not gathered, they are
distributed over three places in the interface. In the prototype the exports are gathered in the same
view and, in order to get a natural transit from the “Event” tab to the “Export” tab they are placed
next to each other (Figure 14). The exports concern all aircraft and therefore the ability to choose
only one aircraft is faded. The user can export just one cycle manually directly from the view of an
event if that is needed.
Figure 14. The exports view, all exports are gathered and contains feedback from previous exports
34
To help the user feel in control of all the different exports, the prototype presents the user with
information from previous exports. It presents when the last export was done as well as how many
days ago this was, this is done to lower cognitive strain for the user. The box at the bottom of the
interface contains history over all the exports that has been done.
4.3 Evaluation of the prototype
The new interface needed to be evaluated against the existing system, using System Usability Scale
and Time on Task to present the differences and to see potential improvements in the new interface.
Because there was no opportunity to test the prototype with an end user, an evaluation was made
with surrogate users. The participants in the evaluation was the development team of MGSS at
Saab AB in Linköping.
4.3.1 Measurements
This section describes the two methods used in the evaluation: System Usability Scale (SUS) and
Time on Task. The result from which was analyzed in the program SPSS version 22 developed by
IBM. Both tests were analyzed through Levene’s Test for Equality of Variance, to see if the
variance is equal between the groups. Further investigation to see if the results are significant was
made through paired sample t-test.
4.3.1.1 SUS
SUS is a popular method for measuring and comparing usability in different interfaces. SUS is
often referenced as a quick and dirty method, which means that the method is cheap and easy to
utilize (Bangor, Kortum, & Miller, 2008). John Brooke introduced SUS in 1986, and it is a
questionnaire with ten questions, half are negative formulated and half are positive. All the
questions have an answer of a scale between one and five. These scales are then transformed into
scores, and is calculated into a total score that generates the overall usability of the interface
(Brooke, 1996). The calculation generates a SUS-score between 0-100, where 100 is a result for a
perfect system. A score under 50 is a non-acceptable interface, a score that lays between 50-70 is
on the marginal to fairly good interface, an acceptable interface should have a SUS-score over 70
(Tullis & Albert, 2013 p.139). The calculation of the total score follows: the score from the positive
formulated questions takes the score from 1 to 5 that the participants have grade the questions and
then takes the score minus one to get the SUS-score. The questions of a negative formulation takes
35
minus five on the given score. For a total SUS-score all these new values are multiplied with 2.5,
and the total score ends up between 0-100. The higher the score is, the higher one can say is the
usability in the interface is.
4.3.1.2 Time on Task
Another usability evaluation made in this thesis is Time on Task, it is simply the time the user need
to complete a task. This is, in most situations, a good way to measure the efficiency of a system
(Tullis & Albert, 2013 p.74). Reducing the time is particularly important in these kind of systems
like MGSS, where tasks are performed repeatedly by the same user. The more frequently a task is
performed in the system by the same user, the more important efficiency becomes (ibid.).
Collecting data with Time on Task is rather easy, it is simply timing the user when they start a task
and stop the time measure when the user has completed the task.
4.3.2 Pilot study
A pilot study was made with one participant from the development team for test the setup of the
evaluation of the two systems. Formulating the tasks was challenging, it is a big difference between
having knowledge about the system and knowing how the daily routine is done. All tasks were
rephrased except the last task. The tasks in the evaluation was formulated to resemble a work
situation, for instance task number three in the prototype was before the pilot study formulated as:
“Do a clarification of the aircraft 222 latest cycle”. This formulation was experienced as confusing
because the development team does not have knowledge about how the daily work routine is done
and how they determine clarification of an aircraft. This led to formulate the tasks to be more
exactly of what actions are expected of them in the interface. Task number three now reads: “Did
any fault events occur during aircraft 222 latest cycle?”. Except for the tasks changes the pilot study
went well, and no other changes with the setup of the computers was made.
4.3.3 Procedure
The evaluation was made with one participant at time, in small room close to the participants work
desks. During the evaluation two computers was needed, one that had MGSS, and one computer
with the prototype. It was not possible to use the same computer to both tasks because of the
security needed for the existing system. The laptops were similar in size and did not differ in
performance for these evaluation. Both laptops had an external mouse connected. The participant
performed four isomorphic tasks in the two system (see Appendix). Before the participants started
36
the evaluation they received information about the procedure of the evaluation. Information about
how they were going to test four similar test in both systems and the time for each tasks was going
to be measured. After the test of one system was finished the participants were asked to fill out a
SUS-survey, and then do the same procedure in the next system.
During the evaluation the participants received the tasks one at a time printed on a paper. They
were asked to tell when they had read and understood the task so that the time measurement started
first when they started the task. Finally, they were asked to tell when they had finished the task to
stop time measurement. Before the participants started the evaluation in the prototype they got an
explanation of the interface. This is because the system is not supposed to be used by novices, the
actual end users are experts and gets education in the system before they use it. They obtained an
explanation of the three new symbols and the idea behind them, and what their purpose in the
interface are. Furthermore, the participants got information of the new possibility to search for
more information about the event number directly from the interface and that no external AMP is
needed.
4.3.3.1 Participants
The participants in the evaluation was no ordinary end-user and was so called surrogate users. All
six participants were employees at Saab AB. Some information in the existing MGSS is secret and
can only be seen from people with permission to use MGSS, which limits the number of
participants. All participants in the evaluation are from the development team of MGSS. They do
have a great knowledge about the systems functions, but not how the system is used in a daily
routine. Among the participants there was one woman and five men.
4.3.3.2 Ethics
Before the evaluation started all participants received a short introduction over the evaluation
scope. They obtained information about the data collection and that all data was handled
confidentially, that no data could lead back to the participant. A verbal approval was then made
before the evaluation started.
4.4 Result of the evaluation
Results are presented as they has developed over time, first a short summary over the results is
presented, after that follows the summary of the evaluation. Different qualities and deficiencies in
37
the system emerged during the observation and are presented in the result of the sequence model
and the activity checklist. These results generated a short requirement list. The most crucial
information that emerged from the observation was the overview of the daily work routine. It
provided insight and knowledge about which events that needed a shortcut or which events that
was experienced as problematics. All this fruitful information helped the designer to not implement
the same deficiencies in the prototype once again as well as the ability to keep the good qualities.
The sequence model also provided information about how straightforward the proceeding in the
interface is. Knowing the flow of the work routine also provide knowledge about what information
to show in a given context rather than showing all information again, which can provide for a
cleaner interface.
4.4.1 SUS-score
Levene’s Test for Equality of Variances of time on task showed no significant result. This verifies
the assumption for homogeneity in the group. The total mean of the SUS result, which is analyzed
after SUS-survey instructions, is shown in Figure 15. The prototype got a higher mean on the SUS-
score (n = 6, M = 71.25 & SD = 13.1) than MGSS (n = 6, M = 44.6 & SD = 23.3). MGSS’s mean
SUS-score was below 50, which interprets as a non-acceptable interface. For the prototype the
SUS-score precisely got over the threshold, 70, for being an acceptable interface. A further analysis
with paired sample t-test did not show any significant results between the SUS-score (p > 0.05).
Figure 15. The mean of the SUS-score for the two systems, error bars with confidence interval at 95%
38
4.4.2 Time on Task
Levene’s Test for Equality of Variances of time on task showed no significant result. This verifies
the assumption for homogeneity in the group. A paired-sample t-test showed a total mean time on
25.4 seconds in MGSS, (n = 24, M = 25.4, SD = 18.17) and for the prototype the total mean time
was 21.9 seconds (n = 24, M = 21.92, SD = 26.1), see Figure 16. Further analysis between the total
means did not show a significant result (p > 0.05).
Figure 16. The mean of the total time it took for the participants to complete all tasks, error bars with confidence interval at 95%
The result for the time on every task individually is seen in Figure 17. Throughout the evaluation
the participants mainly finished the tasks faster in the prototype than in the existing one. Only task
two took longer time for the users to complete, but as mentioned above the mean time to complete
the tasks was faster in the prototype. Paired sample t-test was also conducted between the total time
between each task. The result from task one showed that the prototype was significantly faster (t(5)
= 2.97, p < 0.05). In task two existing MGSS did not show a significant faster total time (t(5) = -
2.24, p < 0.05). For task three the prototype showed a significant faster time (t(5) = 3.00, p < 0.05).
In the last task, task four, there was no significant result between the two systems (t(5) = 0.955, p >
0.05).
39
Figure 17. The mean time on individual tasks
40
41
5 Discussion
This section is divided in different sections, first it focused on the result, method and interface. It
also contains discussion about limitations, future research and implication.
5.1 Result
The results of the SUS-score was higher in the prototype than in the existing interface, which points
towards that the prototype do have a higher usability than the existing system. Using a paired-
sample t-test did however not show any significant result, this could be a result of having few
participants. The result of the prototype is more consistent, a lower standard deviation (SD = 13.1),
compared with the SUS-score of MGSS where the standard deviation varies greatly between
participants (SD = 23.3). Even though the participants are surrogate users the prototype seems to
have a higher usability according in SUS measurements.
The prototype got a faster total mean on time on task. To save time on tasks that are performed
frequently by the same user is an important improvement for the usability and the efficiency of the
system (Tullis & Albert, 2013 p.74). MGSS is used daily, and the troubleshooting is almost always
done with the same routine. As mentioned before the more frequent a user performs a task, the
more important efficiency becomes (ibid.). The participants did accomplish the tasks faster in the
prototype, though with no significant result of the total time. A further analysis on the tasks
individually showed significant result on all tasks except the fourth. Task four did not take many
seconds for the users to finish, and with so few participants it is difficult to get a significant result.
The time on task two however was the mean time significantly faster in MGSS rather than the
prototype. The task concerned interpretation of fault events, the faster time can be explained by
that the participants are familiar with the existing MGSS and knew were to get the right
information. The prototype however does have new symbols that indicates the fault analysis, which
the participant was not familiar with. To accomplish task three similar actions are needed from
both programs and the prototype have a significant faster time than the existing system. With this
result it is argued that there is a very low learning curve in the prototype, once the user knows how
to operate in the system it is faster to use and accomplish the daily routines. The users did however
finish all of the tasks on a faster mean time in the prototype, which indicates a more efficient
interface. This prototype is one way to make the daily work routine more efficient to use.
42
The result shows both a higher SUS-score and a faster mean on time on tasks in the prototype.
Even though the evaluation was made with surrogate users the results indicates a trend of a more
efficient interface with a higher usability in the prototype compared to the existing system.
5.2 Interface
One of the most important improvements in the new interface is the single view functionality.
Existing MGSS is designed in a more traditional windows application, instead of changing the
view, a new window opens up. For instance, if the user wants to see the fault events in the existing
MGSS, a new window opens up containing further information. The single view application in the
prototype contributes to actions being more fluent and natural for the user. For example, to get
more information about the fault event in the prototype the row expands with a click and shows all
information in the same view. To create a more natural flow of the actions in the interface the
exports in the prototype are visible directly from the start view, and not distributed as tool functions
in the menu structure as it is the existing interface. The export in the interface now follow the
naturally flow of actions, rather than random searching the menus for the right export (Spence,
2007 p.160). If users know from the interface where to go next, they will experience less cognitive
load (Sheridan & Reingold, 2014). To tell the user what to do next the prototype uses affordances
(Norman, 2013 p.14) For instance, the button for the exports does not look active in the start view,
but do have the affordance of the possibility to be active (Gaver, 1991). An expert user is also better
in riddle information and can more rapidly distinguish important from less important information
(Cross, 2004). Which is the argument for only showing the fault code and not the description when
a fault event occurs. They know how to interpret the most frequently common codes, when an
unfamiliar code appears they need to search manually for a description. Scaling the information is
an improvement when it is certain what the users can and what they cannot. Taking away
information that is obvious for an expert user creates a cleaner interface which provides an easier
overview that is more efficient for an expert user to use. Although an expert user is better on
distinguish relevant information from irrelevant information (Cross, 2004: Sheridan & Reingold,
2014) the interface can help the user to focus more on the interpretation of the fault code rather than
using cognitive processes on sorting out important information.
The interface of the prototype was needed to be less complex, but access to all the functions of the
system is still important. This led to the decision to have two different views that contains levels
of functionality. During the observation the expert user showed the daily routine which mainly did
43
not include the advanced functions. The daily routines in MGSS showed to be straightforward and
not that complicated, as is presented in the sequence model described in section 3.4. A simple view
however does need to have access to the more complex view, when more advanced troubleshooting
is needed. Which led to have a switch at the top of the interface (Figure 9) so the user easily can
change between the views.
The vision to create simplicity in the interface requires decisions about what information is
important in the given context. MGSS which is a maintenance system for Griffin need to adapt the
mentality that Griffins has: “Don’t Need, Don’t Show”. The design in the prototype is built on
sequential actions, to make navigation in the interface as clear as possible. The sequential actions
are based on the daily routine that proved to be straightforward during the observation. For instance,
the expert user first chooses which aircraft he wants to analyze, and then selects to see only
information from that one. This led to the design decision to have a more distinct area for the ability
to do this action. The new design does also clearly show which aircraft that is selected with the
button changing into an arrow pointing at the data in the adjacent area.
Another design decision was to keep the interface clean from all inputs in existing MGSS and to
distinguish new cycles from the older already analyzed cycles. Though the old cycles need to be
saved within the system for a longer period to make export at a later time possible. The data
however do not need to be clustered together in the same view as in the existing interface.
During the observation the need of overview of the exports was noticed. Today the user had all the
latest export noted on a whiteboard in the workroom to keep track of all different exports. Every
time a new export was made the user manually noted which export it was and the date on the
whiteboard. To help the user keep track of the exports the prototype shows when the latest export
was made, with the date but also with a counter telling how many days ago it was to minimize the
cognitive load for the user. History from exports is also shown in the prototype to help the user to
feel confident and to be able to double check that no exports have been missed.
5.3 Method
The thesis did not only have the purpose to create a prototype of a new interface, it also analyzed
the potential advantages of combining the Activity Checklist with the Sequence Model. Combining
these two methods led to a lot of profitable insights of the system. First the activity checklist was
44
used during the observation, for question inspiration and later for a more fruitful analysis of the
collected data. The activity checklist was needed to be analyzed iterative for a rich understanding,
every iteration gave more useful information and helped to organize sets of items that did not
surface earlier (Kaptelinin et al., 1999). Analysis of the activities the user performed during the
daily work routine gave information and hints of what actions the interface needed to support. The
observed user was an expert of the system and knew where all the functions was therefore the
collected data transmitted knowledge about the functions qualities and deficiencies and how they
could be improved. Furthermore, the sequence model pictured the work flow and gave a clearness
over the actions, and where in the system the actions seemed difficult and troublesome. The
sequence model generated a solid ground pillar to lean on during the design process. The model
showed what step the user would take next, expanding on this knowledge provides the interface
with a seamless flow with a natural evaluation (Spence, 2007 p.91).
Combining these two methods gave a deep understanding of the users’ actions and behaviors in the
system. The activity checklist gave inspiration during the observation over small details, it also
helped to find the deficiencies that needed to be changed and qualities that needed to stay in the
interface. Later modelling of the usage process in the sequence model gave a richer and an easier
way to understand all the information that appeared from the activity checklist. All sub goals that
was found in the activity checklist can now be seen in the sequence model in section 3.4. It also
provided a modeling of all actions that is leading up to a sub goal, and which of these actions that
seems awkward for the user. Using these two methods together was very useful, it gave insight and
clarified both deficiencies and qualities of the existing system. Knowing what functions needed
after each other, (i.e. looking at the operation in the sequence model) made the prototype
development easier. The activity checklist also gave insight in all sub goals, the troubleshooting
strategies and also how the user use artifacts for an easier operation in the system. Every iteration
of the two methods gave more fruitful information about the users’ behaviors.
As mentioned earlier the more frequently a task is performed in a system by the same user, the
more important the efficiency becomes (Tullis & Albert, 2013 p.74). Hence, measuring the time is
important for MGSS, it is used daily by one user. The result for the mean time demonstrated that
it took longer time for the participants to finish all the tasks in existing MGSS, even though the
participants never have seen the prototype before the evaluation. This result can be interpreted to
mean that an expert user with years of experience of the new interface should be able to perform
the tasks even faster.
45
The idea of the measuring with SUS was to get a quick validated insight to see which system the
participant experienced to be the most usable interface. The participants did perceive the SUS
survey a bit tricky, for instance with the question: “I think that I would like to use this system
frequently”. They were then asked to answer how the interface spoken to them, and the experiences
of the systems. The SUS survey was used after both evaluation, so the user that did find the
questions tricky did it with both systems which concludes that none of the systems influenced
negative from this aspect.
The evaluation was made with surrogate users, since it was not possible to test with real end-users.
Neither the tasks nor which system they started with were balanced in the evaluation. The tasks
were not randomized with the argument that the evaluation should resemble a real work situation,
and the flow of actions is in a certain way as the sequence model showed. Hence the order of the
tasks starts with check for fault events and finishes with an export. The argument for not balancing
the systems are the notion of no effect on it. All the participants are developers of the existing
system and have a great knowledge of the system and they are working with it all day. The
evaluation was made during work hours and they were all working with the existing system before
they came in and did the test. Switching the start system on half of the group would not give any
difference in result, because their ability to generalize their skills (Charness, 1976) to the new
interface would not change the result. So the bigger struggle was to get the participants to
understand the question, because they do not have a knowledge about the work routine in the
system. This lack of knowledge could probably provide a misleading time for the participants that
started the evaluation with the prototype.
5.4 Limitations
The most crucial limitation in the thesis is the missing of end-users during the evaluation of the
prototype. Using surrogate users with minor knowledge about the daily routine is not optimal.
Though the evaluation was made with the development team which led to a more generalizable
result than using participants with no experience of the system. Although using random participants
was not an alternative since the existing system is not censored of the secret information, which
means that only confirmed people are allowed to use the system. This is also the reason that there
were only six participants in the evaluation.
46
5.5 Future research
This thesis showed an example of an improved interface of MGSS, but the need of evaluation the
interface with real end-users are crucial to the ability to confirm the improved usability of the
interface. The idea of having two views, one simple- and one advanced view, need to be tested and
evaluated with end-users. This thesis did however not concern the advanced troubleshooting view.
A future research of how the interface in the advanced view should be designed is needed.
It would be interesting to combine this kind of study over different fields, it is not only aircraft who
use and need these kind of complex maintenance systems. To do an over field study would maybe
contribute with profitable insights.
A broader inspection about what every possible outcome, what can go wrong in every step in the
sequence model would give a fruitful insight of needed improvements if the system ever needed in
crisis (i.e. a situation of war). This analysis could be done by for instance with Functional
Resonance Analysis Method (FRAM) by Erik Hollnagel (2012). Which is a method for describing
what has happened or what may happen.
5.6 Implications
MGSS helps Griffin keep a low maintenance cost with high reliability for a longer life cycle.
Rigorous safety requirements are crucial in tense situations. Time is also crucial in a situation of
war, therefore saving time for the aircraft troubleshooting should be prioritized. Air force is a pillar
in military defense and in war every second counts. Griffins have a good rumor about the ability to
be repaired, now it is time to make the pre work (i.e. troubleshooting the aircraft) more efficient.
A new interface can provide less troublesome and confusing information in the system, which can
provide a quick and confident troubleshooting.
47
6 Conclusion
The thesis presented a new way to think when designing a maintenance system. It presented an
interface with simplicity, in the purpose for the user to not be overwhelmed with information. To
help the user to interpret data correct is crucial when handling big data sets. This maintenance
system is used frequently by the same user that in one point becomes an expert user of the system.
Expert users do not need to receive information repeatedly, they can interpret the situation from
the context. They memorize and categorize information differently from novices and by supporting
this expertise the work time can be reduced. It is shown in this thesis that a simpler interface tends
to provide a faster time to finish the daily routine work. Saving time can be crucial in a situation of
war. The evaluation of the prototype showed that the prototype has a higher usability than the
current interface. This new interface the user tended to finish the daily work routine faster and they
experienced it to have a higher usability. These two improvements are important when dealing with
a system that is used in similar way frequently by the same user.
The combination of the sequence model along with the result from the activity checklist provided
with what information that is crucial within different context. Together these two methods
generated an easy way to understand how behaviors and actions elaborates over time in the system.
To see which actions that solves a certain sub goal, which can be seen as qualities. The combination
of the two methods also provided information of which actions that needed improvement, but also
which actions that were not needed at all. The usage of these two methods together gave a solid
insight and helped to define both deficiencies and qualities in the system.
48
49
References
Allen, B. (1996). Information tasks: toward a user-centered approach to information systems. San
Diego, Calif: Academic Press, 1996.
Arvola, Mattias. 2014. Interaktionsdesign och UX: om att skapa en god användarupplevelse. n.p:
Lund: Studentlitteratur, 2014, 2014.
Arvola, M. (2013). The Mediated Action Sheets: A Framework for the Fuzzy Front-End of
Interaction and Service Design. In Crafting the Future: The 10th International European Academy
of Design Conference. Gothenburg, Sweden, April 17 - 19, 2013.
Ball, L., Ormerod, T., & Morley, N. (2004). Spontaneous analogising in engineering design: a
comparative analysis of experts and novices. Design Studies, 25, 495-508.
Bangor, A., Kortum, P., & Miller, J. (2008). An Empirical Evaluation of the System Usability Scale.
Intl.Journal of Human-Computer Interaction, 574-594.
Beyer, H., & Holtzblatt, K. (1998). Contextual design: defining customer-centered systems. San
Francisco, Calif: Morgan Kaufmann Publishers, cop. 1998.
Brooke, J. (1996). SUS-A quick and dirty usability scale. Usability evaluation in industry.
Chi, M., Feltovich, P., & Glaser, R. (1981). Categorization and representation of physics problems
by experts and novices. Cognitive Science, 5, 121-152.
Cross, N. (2004). Expertise in Design: An Overview. Design Studies, Vol. 25, No. 5, pp. 427-441,
2004
De Souza, C. S. (2005). The semiotic engineering of human-computer interaction. Cambridge, MA:
MIT Press, 2005.
Engeström, Y. (1987). Learning by expanding: an activity-theoretical approach to developmental
research. Helsinki: Orienta-konsultit, 1987.
Garbis, C. (2002). The Cognitive Use of Artifacts in Cooperative Process Management: Rescue
Management and Underground Line Control. Linköping Studies in Arts and Science.
Gaver, W. W. (1991). Technology affordances. In Proceedings of the SIGCHI conference on
Human factors in computing systems (pp. 79-84). ACM.
Gibson, J. J. (1979). The ecological approach to visual perception. Boston, Mass: Houghton Mifflin,
cop. 1979.
50
Hollnagel, E. (2012). FRAM, the functional resonance analysis method. [electronic resource] :
modelling complex socio-technical systems. Farnham, Surrey, UK England; Burlington, VT:
Ashgate, c2012.
Kaptelinin, V., & Nardi, B. (2012). Affordances in HCI: toward a mediated action perspective. In
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 967-976).
ACM.
Norman, D. A. (2013). The design of everyday things. New York, NY: Basic Books, 2013.
Preece, J., Rogers, Y., & Sharp, H. (2016). Interaktionsdesign: bortom människa-dator-interaktion.
Lund: Studentlitteratur, 2016
Saab AB (2016) Griffin Global Support – High availability greater operational effect. Retrieved
2016-03-23, from http://saab.com/air/support-solutions-and-services/airborne-platform-support-
solutions/Griffin-support/
Sheridan, H., & Reingold, E. M. (2014). Expert vs. novice differences in the detection of relevant information
during a chess game: evidence from eye movements.
Stickdorn, M., & Schneider, J. (2011). This is service design thinking: basics, tools, cases.
Hoboken, N.J. : Wiley, 2011.
Stone, D. L. (2005). User interface design and evaluation. San Francisco, Calif.: Morgan
Kaufmann, cop. 2005.
Spence, R. (2007). Information visualization: design for interaction. Harlow: Pearson/Prentice Hall,
2007.
Tullis, T., & Albert, B. (2013). Measuring the User Experience: collecting, analyzing and
presenting usability metrics, second edition. Waltham: Morgan Kaufmann.
51
Appendix
The tasks for existing MGSS:
1. Find out when the aircraft 3707 did its last cycle?
(define date and time, it must not be a manually created cycle)
1. During aircraft 3407 latest cycle a FM-Alarm occurred with Event No 563, find MG-number and
the binary code for this fault event.
2. Did it occur any FM-Alarm during aircraft cycle with time and date:
2010-08-02 08:14:40?
3. Do an OSIS export.
In the prototype the participants were asked to perform following four isomorphic tasks:
1. Find out when the aircraft 222 did its last cycle?
(define date and time, it must not be a manually created cycle)
2. During aircraft 224 latest cycle two critically fault events occurred, which was the first fault event
that occurred?
(define MG-number and the binary code for the event)
3. Did it occur any fault event during aircraft 222 latest cycle?
4. Do a Fenix export.