projecting efficacy and use of business simulation games – ahfe 2016

Projecting Efficacy and Use of Business Simulation Games in the Production Domain using Technology Acceptance Models

Philipp BraunerRalf PhilipsenMartina Ziefle

Human-Computer Interaction CenterRWTH Aachen University, Germany

Philipp Brauner , Ralf Philipsen, Martina Ziefle, Projecting Efficacy and Use of Business Simulation Games in the Production Domain Using Technology Acceptance Models, Advances in Ergonomics of Manufacturing: Managing the Enterprise of the Future, Volume 490 of the series Advances in Intelligent Systems and Computing pp 607-620, Springer International Publishing (2016)

http://link.springer.com/chapter/10.1007/978-3-319-41697-7_52









RWTH Aachen University, Human-Computer Interaction Center

P. Brauner et al. – "Projecting Efficacy and Use of Business Simulation Games in the Production Domain using Technology Acceptance Models" - AHFE 2016

Context of the Work & Motivation What determines performance in production

environments? Study on the usage intention of business simulation

games in the production domain.– Determinants of

interaction performance acceptance

Discussion and Outlook

Agenda

page 2

2016-07-31


2016-07-31

page 3

Human-Human & Human-Technology communication Design, development, and evaluation of interactive systems

– Requirements analysis– Usability & User Experience– Technology Acceptance and risk perception– Human Factors, User Diversity & Aging population

Interdisciplinary team, 20 researchers

Human-Computer Interaction Center atRWTH Aachen University, Germany



Context: Part of the Cluster of Excellence“Integrative Production Technology for High-Wage Countries”

Goal: Strengthen competitiveness of high wage countries Engineering of future production systems

– New materials and processes– Improved and smarter machinery– Optimize assembly cells, shop floor, cross-company cooperation

> 25 Institutes, > 100 researchers, > 60M€

page 4

2016-07-31 P. Brauner et al. – "Projecting Efficacy and Use of Business Simulation Games in the Production Domain using Technology Acceptance Models" - AHFE 2016


Our research objective:Optimize cross-company cooperation

Optimize cross-company supply chains (SC)– Technical factors influencing performance of SCs– Human Factors influencing performance of SCs– Interface Factors on SCs performance– Interrelationship of technical, interface, and human factors

Why are humans considered?– Humans make final decision– Overview over not explicitly modelled relationships

(e.g., closed-world assumption)– Complexity and uncertainty increases,

less time for making decisions

Information flow

flow of goods

Supply Chain

page 5



Our research objective:Optimize cross-company cooperation

Information flow

flow of goods

Supply Chain

page 6

2016-07-31

Goal 1: Understand system and user factors that influence efficiency, effectivity, and user satisfaction in Enterprise Resource Planning Systems, Supply Chains and Quality Management.


Goal 2: Supporting decision makers in production networks by providing appropriate user interfaces, decision support systems and training tools.


What determines performance inComplex Cyber-Physical Production Systems?

2016-07-31

page 7

Domain expertisePersonality states & traitsTrust, Self-efficacy,Motivation, …

Uncertainty, randomnessNon-linear interactionsDisruptions & Seasonal changesFeedback loops, …

Interface DesignVisual complexityInformation visualizationDecision Support, …

Efficiency, Effectivity,Profit, Quality, Satisfactionof Workers and Customers

USERSYSTEM INTERFACE

PERFORMANCE



How can Human and Interface factors be investigated?Business Simulation Games!

Convergence between field and laboratory study Simplified & controllable (game-based) environments Experimentally manipulate complexity and interface Empirical methodology to quantify human performance

– Identify and measure influencing personality factors– Identify and measure influencing interface factors– Build a formal model that explains performance

Side-effect:Usable for game-based learning (GBL) ineducation and professional trainings

Test in the field(ecological validity)

Controlled experiment in

laboratory(internal validity)

We are here

page 8



Development of business simulation game Extends „Beer Distribution Game“ Based on System Dynamics model Includes product quality

– Product intact or broken– Supplier's quality varies– Internal production quality varies

Increased complexity (tradeoff 3 parallel tasks)– Management of stock levels– Investment in incoming goods inspection– Investment in internal quality management

Over 20 variables in the user interface Players must infer state of the production

Part of the game’s simulation model:Stock level at a given time t:S(t) = S(t-1) + O(t-1) – D(t)

Net profit:P(t) = R(t) – cStock×S(t) – Iigi(t) –

Iipq(t) – C(t-1) – …

page 9



The Quality Management Game’s interface

Web-based game environment– Accessible across the world (scale)– Laboratory studies (scope)

Captures all metrics and interactions Controllable conditions

– Varying supplier’s quality, own production quality, customer’s demand, user interface, …

Map game performance and other metrics to Human-Factors

– Identification of attitude, aptitude & motivation– Evaluation of the game, interface, …

page 10



Business Simulation Games as aResearch Lab for Understanding System, Interface, and User Factors Interactive Business simulations

– Forrester’s Beer Distribution Game, Goldratt’s Game– Quality Management Game

Several studies– Do System, Interface, and Human factors influence performance?

Questions addressed– Replication of similar studies? ✓– Raises awareness for Quality Management? ✓– Do human factors exist that explain performance? ✓– Which human factors influence performance? ❓– Do interface aspects influence performance? ✓– Which interface aspects influence performance? ✓– How can users be supported to make better decisions? ❓

2016-07-31

page 12



Business Simulation Games asTools for Game-based Learning (GBL) in Education and Professional Trainings

Research Questions:

– What are crucial factors for the use of serious games for knowledge dissemination in quality management, mechanical engineering, and production engineering?

– What are determinants for interaction with the game, performance and acceptance (behavioral intention BI)?

2016-07-31

page 13


Study approach based on technology acceptance model research. Survey-based study combined with playing the Quality Management Game.


Experimental Setup 7 Dimensions from UTAUT2 (Venkatesh et al. 2012)

– Performance expectancy, Effort expectancy, Social-influence, Price-value, Hedonic Motivation, Facilitating conditions, and Habit

4 Dimensions of SG-TAM (Yusoff et al. 2010)– Reward, Learner Control, Transfer of Skills, and Situated Learning

Time-Value Self-efficacy in interacting with technology (Beier 1999) Attitude Towards Serious Games Need for Achievement (Nicholls 1984)

2016-07-31

page 14

Interaction

Performance

Acceptance



UTAUT2

2016-07-31

page 15


Predict Use of consumer technology Intention To Use and Use governed by

– Performance Expectancy (PE)– Effort Expectancy (EE)– Facilitating conditions (FC)– Social Influence (SI)– Hedonic Motivation (HM)– Habit (H)– Price-Value (PV)

Mediated by– Age– Gender– Experience


Experimental Setup 7 Dimensions from UTAUT2 (Venkatesh et al. 2012)

– Performance expectancy, Effort expectancy, Social-influence, Price-value, Hedonic Motivation, Facilitating conditions, and Habit

4 Dimensions of SG-TAM (Yusoff et al. 2010)– Reward, Learner Control, Transfer of Skills, and Situated Learning

Time-Value Self-efficacy in interacting with technology (Beier 1999) Attitude Towards Serious Games Need for Achievement (Nicholls 1984)

2016-07-31

page 16

Interaction

Performance

Acceptance



Description of the Sample

66 participants– both experts and students in the field of quality management in

production Age

20 – 56 yearsMedian: 24 years

Gender8 female, 57 male, 1 unspecified

2016-07-31

page 17



Results

Qualitative findings

2016-07-31

page 18



Qualitative Feedback

mostly positive feedback on the overall game:

“[…]is really good and useful for understanding the concepts of PPC.” (Production planning and control)

“Game was great… need more such games with different concepts. It encourages involvement and understanding real life scenarios.”

“[…] is one of the best way[s] to learn practical industrial problems. This is [a] very nice Game.”

2016-07-31

page 19



Results

Quantitative findings

2016-07-31

page 20



Determinants for Increased Game Interaction

The number of changes to the controls in the game varied from 3 to 39 (mean 22.6±10.6, median 22).

Self-efficacy in interacting with technology single explanatory variable influencing the number of changes. People with higher SET did significantly more changes [ρ=.367, p=.028<.05].

2016-07-31

page 21


Determinants for Performance Most independent variables did not relate to the achieved Performance in the game. Only the attitude towards serious games ASG [ρ=-.327, p=.051<.1] and the need for

achievement NA [ρ=-.329, p=.050<.1] had a negative influence of performance.


Determinants for Acceptance

2016-07-31

page 22


EE PE HE FC SI PV TV REW LC TOS SL BIEE — .732** .780** .447** .466** .601** .693** .698** .684** .434* .603**

PE — .671** .502** .630** .516** .733** .372* .834** .790**

HE — .497** .419* .309 .782** .495** .720** .289 .619**

FC — .280 .357* .365* .498** .463** .548** SI — PV — .717** .417* .633** .317 .415*

TV — .594** .359*

REW — .417* .745** .595**

LC — .354* .432** TOS — .747**

SL — SET .588** .410* .574** .399* .419** .472** .486** .372* .534** .407* .401*

ASG .398* .477** .357* .309 .491** .682**

AM .319 .334* .349* .441* .480* .470** .405*

Spearman’s ρ-correlation coefficients for the user factors and the UTAUT2 / SGTAM model dimensions (listed if p<.1, *<.05, **<.001).



Strong positive relationship of most variables on the intention to use the game (BI).

The three strongest influencing factors:– Performance Expectancy PE [ρ=.790]– Transfer of Skills TOS [ρ=.747]– Hedonic Motivation HE [ρ=.619]

No influence on BI by:– Facilitating Conditions FC– Social Influence SI– Learner Control LC– Situated Learning SL

2016-07-31

page 23




Due to small sample size the application of multiple linear regressions yielded only in a single significant model for the user factors and for the models’ factors:

– User Factors: Attitude towards serious games (ASG) single predictor for intention to use the game, explaining 61.7% of the variance in BI (r2=.617, β=.792)

– Model Factors: Performance Expectancy (PE) explains 59.2% (r2=.592, β=.778) of the variance in BI.

2016-07-31

page 24



Subsumption and Outlook

Overall acceptance of serious games for conveying knowledge of production planning and control and quality management was high.

Combination of the UTAUT2 and the SG-TAM model to investigate the acceptance of the QM-Game and business simulation games in general seems to be very promising:

– both models complement each other and may in combination contribute more explained variance in BI and later USE than each individual model.

Further studies needed to research isolated influence of interwoven factors

2016-07-31

page 25



Subsumption and Outlook Guidelines for Business Simulation Games:

2016-07-31

page 26


Priority Guideline

1Provide clearly visible short-term benefits of using the

game. E.g., by making clear that the conveyed skills will be beneficial for an upcoming exam. (Based on PE)

2The player must perceive the presented environment as a

simulacrum of the reality. E.g., by portraying realistic production processes. (Based on TOS)

3Consider learner diversity, especially in regard to different

levels of inclination towards games. E.g., by augmenting the game environment with traditional forms of knowledge dissemination. (Based on ASG)


Subsumption and Outlook Guidelines for Business Simulation Games:

2016-07-31

page 27


Priority Guideline

4Create enjoyable learning environments. E.g., by including

potential players in the development to ensure target specific aesthetics and playfulness. (Based on HE)

5Avoid unnecessary complexity of the user interface and the

simulation model, and provide a focused learning experience. E.g., by reducing the perceived effort for mastering the game through guided tutorials, help functions etc. (Based on EE)

6Provide adequate and immediate feedback on the learning

performance. E.g., by linking the learning objectives with the company’s profit or by adding motivational incentives (badges, leaderboards, …). (Based on REW)


Brauner P, Runge S, Groten M, et al (2013) Human Factors in Supply Chain Management – Decision making in complex logistic scenarios. In: Yamamoto S (ed) Proceedings of the 15th HCI International 2013, Part III, LNCS 8018. Springer-Verlag Berlin Heidelberg, Las Vegas, Nevada, USA, pp 423–432

Brauner P (2014) Understanding Human Factors in Supply Chains and Quality Management by Using Business Simulations. In: Brecher C, Wesch-Potente C (eds) Proceedings of the Conference of the Cluster Of Excellence “Integrative Production Technology For High Wage Countries” 2014/1, 1st edn. Apprimus Verlag, Aachen, Germany, Aachen, Germany, pp 387–396

Hering N, Meißner J, Runge S, Brauner P (2014) Exzellenzcluster: Was bestimmt die Performance meiner Supply-Chain? – Eine Untersuchung technischer und menschlicher Einflussfaktoren im Hinblick auf die Effizienz von Lieferketten. Unternehmen der Zukunft - Zeitschrift für Betriebsorganisation und Unternehmensentwicklung 27–28.

Philipsen R, Brauner P, Stiller S, et al (2014a) The role of Human Factors in Production Networks and Quality Management. – How can modern ERP system support decision makers? First International Conference, HCIB 2014, Held as Part of HCI International 2014, Heraklion, Crete, Greece, June 22-27, 2014. Proceedings, LNCS 8527. Springer Berlin Heidelberg, pp 80–91

Philipsen R, Brauner P, Stiller S, et al (2014b) Understanding and Supporting Decision Makers in Quality Management of Production Networks. Advances in the Ergonomics in Manufacturing. Managing the Enterprise of the Future 2014 : Proceedings of the 5th International Conference on Applied Human Factors and Ergonomics, AHFE 2014. CRC Press, Boca Raton, pp 94–105

Stiller S, Falk B, Philipsen R, et al (2014) A Game-based Approach to Understand Human Factors in Supply Chains and Quality Management. Procedia CIRP 20:67–73. doi: 10.1016/j.procir.2014.05.033

Brauner P, Ziefle M (2015) Human Factors in Production Systems – Motives, Methods and Beyond. In: Brecher C (ed) Advances in Production Technology. Springer International Publishing, pp 187–199

Mittelstädt V, Brauner P, Blum M, Ziefle M (2015) On the visual design of ERP systems – The role of information complexity, presentation and human factors. 6th International Conference on Applied Human Factors and Ergonomics and the Affiliated Conferences, AHFE 2015. pp 270–277

Calero Valdez A, Brauner P, Schaar AK, et al (2015) Reducing Complexity with simplicity - Usability Methods for Industry 4.0. 19thTriennial Congress of the International Ergonomics Association (IEA 2015).

Ziefle M, Brauner P, Speicher F (2015) Effects of data presentation and perceptual speed on speed and accuracy in table reading for inventory control. Occupational Ergonomics 12:119–129. doi: 10.3233/OER-150229

Brauner, P., Philipsen, R., Fels, A., Fuhrmann, M., Ngo, H., Stiller, S., Schmitt, R., Ziefle, M.: A Game-Based Approach to Meet the Challenges of Decision Processes in Ramp-Up Management. Quality Management Journal. 23, 55–69 (2016).

Calero Valdez, A., Brauner, P., Ziefle, M.: Preparing Production Systems for the Internet of Things The Potential of Socio-Technical Approaches in Dealing with Complexity. In: Dimitrov, D. and Oosthuizen, T. (eds.) Proceedings of the 6th International Conference on Competitive Manufacturing 2016 (COMA ’16). pp. 483–487. CIRP, Stellenbosch, South Africa (2016).

Brauner P, Ziefle M. How to train employees, identify task-relevant human factors, and improve software systems with Business Simulation Games. Procedings of the International Conference on Competitive Manufacturing 2016, COMA ’16. Stellenbosch, SA; 2016. p. 541–6.

Brauner, P., Calero Valdez, A., Philipsen, R., Ziefle, M.: Defective Still Deflective – How Correctness of Decision Support Systems Influences User’s Performance in Production Environments. Proceedings of the Human-Computer Interaction International 2016. (in press)

Brauner, P., Philipsen, R., Ziefle, M.: Projecting Efficacy and Use of Business Simulation Games in the Production Domain using Technology Acceptance Models. Proceedings of the Applied Human Factors and Ergonomics Conference (AHFE 2016). (in press)

Brauner, P., Philipsen, R., Calero Valdez, A., Ziefle, M.: On Studying Human Factors in Complex Cyber-Physical Systems, Workshop HFIDSS 2016, Mensch &Computer 2016 (in press)

Publications

page 28



Thank you four your attention!

Questions?

Dipl.-Inform. Philipp BraunerHuman-Computer Interaction CenterChair for Communication ScienceRWTH Aachen University, GermanyeMail: [email protected]

page 29

2016-07-31

Funded by the German Research Foundation (DFG) within the Cluster of Excellence “Integrated Production Technology

for High Wage Countries” (EXC 128).



Appendix

Additional slides

page 30



Evaluation of user interfaces: Research question:

– Do interfaces influence player’s performance?

Interface optimizations based on user feedback– Better spatial layout– Key Performance Indicators (e.g., stock level)

Method– Study (N=40) with old vs. new interface, surveys

Results– Users preferred revised user interface– Higher profits and higher product quality w. new interface

Conclusion:– Good interfaces crucial for performance

(V = 0.263, F1, 38 = 13.548, p = .001 < .05*) revised interface

first interface

page 31


projecting efficacy and use of business simulation games – ahfe 2016

Data & Analytics