* Université catholique de Louvain (UCL) Belgian Laboratory of Computer-Human Interaction (BCHI)

{juan.m.gonzalez, jean.vanderdonckt}@uclouvain.be† OFFIS e.V., Escherweg 2, 26127,

Oldenburg, Germany{luedtke, osterloh}@offis.de

Towards Model-Based AHMI Automatic Evaluation

Juan Manuel Gonzalez-Calleros*, Jean Vanderdonckt*, Andreas Lüdtke†, Jan-Patrick Osterloh†

• Existing work on automated evaluation of AHMI • Some studies suggested that AHMI and software user interfaces

share several similarities:• In terms of interaction techniques for input/outputWilliges, R.C., Williges, B.H., and Fainter, R.G., Software interfaces for aviation systems, in

Human Factors in Aviation, E.L. Wiener and D.C. Nagel (Eds.) 1988, Academic Press: San

Diego: pp. 463-493. • In terms of automatic analysis of workloadEldredge, D., S. Mangold, and R.S. Dodd, A Review and Discussion of Flight Management

System Incidents Reported to the Aviation Safety Reporting System 1992, Battelle/U.S. Dept. of Transportation.

• In terms of automatic evaluation based on goal modelsIrving, S., Polson, P., Irving, J.E., A GOMS Analysis of the Advanced Automated Cockpit, Proc.

of ACM Conf. CHI’94, ACM Press, New York, 1994, pp. 344-350

Introduction

10 April 2023 Page 2

• Several challenges for Aircraft cockpit design, such as: • Introduction of new technologies break previous user experience.

Introduction

10 April 2023 Page 3

• New challenges for AHMI: analysis, design, implementa-tion, and evaluation

• Integrating evaluation in the loop of AHMI life cycle involves the use of pilots and a physical simulator. • Shortcomings: availability, cost, time,…

• Human project: new ways to substitute pilots and physical simulator are explored by coupling them to virtual simulator

• Our goal: to introduce UI automated evaluation (workload, execution time, usability guidelines) in the life cycle

Introduction

10 April 2023 HMAT Workshop, June 2010 Page 4

• Interactive Cooperative Objects (ICOs)• Used to model aircraft interactive systems (air traffic workstations, civil

aircraft cockpit and military aircraft cockpit)• Formal description for interactive cockpit applications using Petri nets• Behavioural aspects of systems in the cockpit are modelled• Recent work on usability, reliability, and scalability

• Navarre, D., Palanque, Ph., Ladry, J.-F., Barboni, E., ICOs: A model-based user interface description technique dedicated to interactive systems addressing usability, reliability and scalability, November 2009, Transactions on Computer-Human Interaction (TOCHI) , Volume 16 Issue 4

• To be added • More widgets, more than system and task models• Usability guidelines• Notion of resilience

• David Navarre, Philippe Palanque, Eric Barboni, Jean-François Ladry, Célia Martinie, Designing for Resilience to Hardware Failures in Interactive Systems:A Model and Simulation-Based Approach Reliability Engineering and System Safety, vol. 95, n°6, 2010

State of the Art

10 April 2023 HMAT Workshop, June 2010 Page 5

• Interactive Cooperative Objects (ICOs)

State of the Art

10 April 2023 HMAT Workshop, June 2010 Page 6

• ARINC 661 Standard• Defines protocols to communicate the dialogue and the functional core

of cockpit display system:

• a user application is defined as a system that has two-way communication with the CDS (Cockpit Display System):

• Transmission of data to the CDS, possibly displayed to the flight deck crew• Reception of input from interactive items managed by the CDS

• A set of widgets is included as a recommendation

• Problems• No design guidelines for the widgets,

• No method to design UIs are considered in the standard

• Each manufacturer is free to implement their own understanding about the standard

• The ARINC standard is not used for primary cockpit applications (AHMI)

• Only deals with secondary applications involved in the management of the flight such as the ones allocated to the Multiple Control Display Unit

State of the Art

10 April 2023 HMAT Workshop, June 2010 Page 7

• A Methodology is suggested that is composed of:• Models (Traditional Widgets + AHMI specific UI elements)• Language (User Interface Description Language)• Method (Model-Based UI Design applied to the AHMI)• Software support

• A Language Engineering approach• Semantics as meta-models (UML class diagrams)• Syntax as XML Schemas• Stylistics is the visual syntax, in our context not used.

• A Structured Method compliant to the Cameleon Reference Framework for UI development• Just focus in the layer that concerns to the concrete description

model.

Model-Based AHMI Design

10 April 2023 HMAT Workshop, June 2010 Page 8

• The Concrete UI Model (CUI) allows:• Specification AHMI presentation and behaviour independently of

any programming toolkit

Model-Based AHMI Design

10 April 2023 HMAT Workshop, June 2010 Page 9

X3D OpenGL

• Automated evaluation of• Static aspects (UI layout, position of objects)• Dynamic concepts (state of a button during the interaction, color of

the label).

• Any UI is represented in a UsiXML model submitted to automated evaluation (automatic or manual)

• Usability guidelines over the UI objects (distribution of the widgets composing the UI) are evaluated.

• Special attention was paid to those guidelines for standard certification and quality assurance and to express them in the Guideline Definition Language (GDL)

Evaluating the AHMI User Interface

10 April 2023 HMAT Workshop, June 2010 Page 10

http://www.usixml.orghttp://www.w3.org/2005/Incubator/model-based-ui/

Integrating UI Evaluation in a Simulation Environment

10 April 2023 HMAT Workshop, June 2010 Page 11

UI timeline

10 April 2023 HMAT Workshop, June 2010 Page 12

• Evaluating the AHMI User Interface

• Execution Time

• Workload

• Guidelines

Outline

10 April 2023 Page 13

Evaluation-Execution time

10 April 2023 Page 15

Evaluation-Workload

10 April 2023 Page 16

Evaluation-Execution Workload

1.2 + 3x(1.5 + 0.075) + 1.5 = 7.4254.6 + 4x(4.6 + 2.2) = 31.8

WorkloadRemember dateSelect daySelect monthSelect year

Execution Time Select daySelect monthSelect yearValidate

(Use mouse to point at object on screen1.5 second)+Execute a mental “step”.075 secondRetrieve a simple item from a long-term memory1.2 second

4.6Evaluation/Judgment (consider single aspect) + 2.2Discrete Actuation (button, toggle, trigger)

Compilation

UsiXML Evaluation

Evaluation report

Formal rules file

Original ruleUsability guidelines

Smith & MosierRule interpretation

WCAG 2.0

Natural language

Formal language

…

A Rule

A Header

Contrainte A

Variable AVariable AA Variable

Contrainte AA constraint

Formal des-cription of A

A Widget

A Quantifier

UA: evaluation process

Compiled evaluationrules

Compilation

UsiXML Evaluation

Evaluation report

Formal rules fileSource file

Lexical analysis

Syntactic tree

Sequence ofLexical units

SyntacticAnalysis

Header:1, “White and yellow should not be used together for text“,high,partial, "Legibility and perception",must,enabled

rule:no inputtext i: i has_properties(bgcolor == "#ffffff") and i has_properties(fgcolor == "#ffff00")

UA: evaluation process

Compiled evaluationrules

Compilation

UsiXML Evaluation

Evaluation report

Formal rules fileCréation d’algorithmes testant le contenu de l’information des “GraphicalWidget”InputText AInputText A

InputText BInputText B

InputText C

UA: evaluation process

Compiled evaluationrules

Compilation

UsiXML Evaluation

Evaluation report

Formal rules fileInputText C

UA: evaluation process

Compiled evaluationrules

Some guidelines

• Cockpit display systems should at least be consistent with systems of our daily life [Singer 2002]

• Usability guidelines from ISO 9126 could be evaluated• Messages should follow always the nomenclature: first letter in

capital and the rest in lower case• AHMI display systems such as the consistency in the roll index in

the compass rose [Singer 2001]

10 April 2023 HMAT Workshop, June 2010 Page 21

• A model-based method for the development of the AHMI was presented allowing:• Structured development process (making design more explicit)• UI evaluation

• Traditional measurements can be assessed like UI workload and execution time,

• More complex automated evaluation based on guideines.

• Explore design options, for instance, modality of interaction of the UI can be object of evaluation.

• The original 2D rendering can be equally rendered in 3D.

• A future plan is to automatically generate the AHMI from its model and to submit it to run-time analysis.

Conclusion

10 April 2023 HMAT Workshop, June 2010 Page 22

10 April 2023 HMAT Workshop, June 2010 Page 23