how we embraced risk to catalyse change

| How We Embraced Risk to Catalyse Change, Swiss-Air Rescue Rega, Georg Hauzenberger

How We Embraced Risk

To Catalyse Change

Case Study Presentation | 28/10/2020 | Zurich-Airport


Mass extinction events occur more frequently

than one might imagine. Throughout earth's

"recent" history, the so-called Big Five are the

most prominent.

These Great Resets always produce lots of

losers, but also the winners of tomorrow.

Winners embraced the risks and hardship that

arise from a global catastrophe and gradually

learned to thrive in a new, changed

environment. We should try to learn from that.

Millions of years before current time

Genera

extinction inte

nsity in % Our lucky

chance!

The dinosaurs'

opportunity

O-S

Ob.-D

P-Tr

Tr-JK-Pg


In January 2018, Swiss Air-Rescue Rega

started a project that aimed to introduce a

new helicopter mission control system.

The project was concluded in July 2019 on

time, on budget and with a very smooth

transition to operations.

We faced many risks along the way, which

we always confronted head-on. I want to

outline these risks, how we "embraced"

them and how we were able to catalyse

change by addressing them.


Shorthand for "Swiss Air-Rescue Rega"

Based in Switzerland

Privately run, non-profit organisation

Financed by around 3.5 million donors

Provides around-the-clock medical

assistance by air

12 helicopter bases throughout

Switzerland with 19 helicopters

3 repatriiation jets

16'782 missions in 2019


https://www.youtube.com/watch?v=n2JG-yrgvrM


Georg Hauzenberger, IT Project Manager

Helicopter Rescue

Originally born in Austria, now living in Zurich

Background in communication systems

technology

Worked in satellite communications,

telemetry systems design, requirements

engineering, systems architecture and cost

steering

Several years of experience in agile projects

and teams

Working for Rega since April 2018

Courtesy of www.bendik.ch


1. Moving from waterfall to agile

2. Making sure the users have their say

3. Which new communication system?!

4. Packing and leaving on-prem

5. Not everything is software

6. Testing the "real deal"

7. Going live and the aftermath



In Rega's context, the helicopter mission control system is used to

register incoming alarm calls, dispatch the "next best" rescue

helicopter, convey mission-critical information and store all mission

metadata for further processing.

The mission control system is operated by Rega's emergency

dispatchers, which act as a hub between the patient, the helicopter

crew and other emergency medical services, such as hospitals.


Rega's current mission control system was

already in operation since 2012. Zero downtime,

better usability and greater operational

efficiency together with diverse technical drivers

(e.g. Windows 7 phase-out) were the main

reasons for seeking a replacement.

In the summer of 2017, a tender was launched

in which several suppliers participated, including

the vendor of the current system. At the

beginning of November 2017, the decision was

made to switch to a new supplier. One key

agreement was that the project would be

conducted based on agile methodologies.


3+ Involved Suppliers (Dev, QM, Infrastructure, …)

Total project costs in the low seven digits (USD)

Core project team comprising 12 people

Project duration 18 months

"Scrumish" methodology


The mission control system enables Rega's

emergency dispatchers to register incoming

alarms, dispatch the "next best" emergency

helicopter and coordinate the mission involving

patients, Rega's helicopter crews and other

emergency service providers.


Map

Computer-aided

Dispatch (CAD)Rega GeoStore,

OSM, GWR, Swisstopo, …

PoI-Data

Mico

Radio system

Telephony and Radio

Helicopter positions, Helicopter status, "Panic button"

F24 SMS, FaxExchange

SAP Mission invoicing, helicopter and mission data

Mission Plattform

Crew alarms, Mission handover, helicopter status

BI Mission statistics

ETV Not

ETV Inside

Rega-App

Caller identification and position triangulation

Caller identification

Alarms via Rega-App

Client side: map web application

interacts with a fat client

dispatching application

Server side: georedundant,

virtualised application, geo object

and database servers

Whole range of systems needs to

be integrated

Interfaces to telephony, radio,

avionics systems, mobile apps,

etc.Rescuer & Mission

Database


ScopeCreep

Resources

User Acceptance

Integration

Test Coverage

New Data Centre

New Communication

System

New Client Infrastructure

New Methodology

Roll-out

You know those "Our" risks

12

34

56

6 7


1. Moving from waterfall to agile

New Methodology


Although it was agreed that the project should be conducted using agile

methodologies, it did not really start that way.

As I only joined Rega in April 2018, my boss, Lukas Müller, the head of IT,

had the onerous task of introducing change in the project. At first, the team's

agile transformation proved to be an uphill fight …



Keep it simple and stupid: our

development and testing method was

"born" on a paper napkin

Fast feedback: we iterated each week

in the beginning, then moved to two

weeks

Encourage self-organisation: we

quickly got rid of processes and

RASCI-matrices

Visualise: visualise your product,

current state of development, your

progress, your process, everything!


Sprint: 1 Week

Wednesday

Backlog Refinement &

Testing Planning

08:15 – 09:45

Break with Coffee and

Gipfeli 09:45 – 10:00

Project Mgmt Varia

10:00 – 10:30

On-prem Working

Session

from 10:30

Thursday

User Review

09:00 – 10:30

Break with Coffee and

Gipfeli 10:30 – 10:45

User Workshops &

Trainings

10:45 – 12:30

On-prem Working

Session

from 12:30

Tuesday

Daily

07:45 – 08:00

Friday

Daily

07:45 – 08:00

Monday

Deadline deployment user

stories and creation of test

cases for current Sprint

Testing Goal


0

100

200

300

400

500

600

700

800

10 12 14 16 18 20 22 24 25 26 27 28 29 30 31 32 33 34

Work

Rem

ain

ing

[SP]

Sprint

Measured & Projected PA 4

Measured Projected PA 5 (excl. Mico)

Measured & Projected PA 5 (incl.Mico)

Measured & Projected PA 7 (incl.Mico)

Sprint 30:105 SP (65 SP)Sprint 31: 50 SP (22 SP)Sprint 32: 23 SP (22 SP)Sprint 33: 100 SP (72 SP)

AddressObjects

ELS Heli(Pre-)

Production System

Mico

Radio

F24

SAP

Data Import

Mission Platf.

ARS

MicoTest System

Make call

Open mission

Get mission details

Create protocol entry

Replicate master data

Contact Data

ELS HeliInt

ELS HeliTest

IMPORT IMPORT

Done

In Progress

Open

Descoped

Legend


2. Making sure the users have

their say

New Methodology

User Acceptance


Originally, scope was documented with business

and IT requirements

Business requirements were clustered into Epics

Epics contained a "high-level" User Story, were

linked to the original business requirements and

had acceptance criteria that sketched the broad

scope

Epics were, in turn, clustered into User Stories

that detailed the broad scope (based on several

business requirements at times)

For each iteration, a focus and testing goal were

defined based on the Epics and User Stories

implemented

Epic Alarming rescue helicopters ("resources")

As an emergency dispatcher,

I'd like to alarm "resources" over

various channels and see their

status, so that I can initiate the

handling of a mission and check

upon its status.

Alarming Cantonal police

forces in case of a traffic

accident

User Story


User acceptance of an application is the ultimate goal. But

writing User Stories is not enough for that.

Already at the end of Sprint 1, a group of 4 emergency

dispatchers started to test "out-of-the-box" functionalities and

how they were adapted to their needs. User feedback was

afterward consolidated, inserted as User Stories into the Product

Backlog and, if prioritised high enough, considered in a Sprint.


For each Sprint, we assigned a

so-called "testing goal", e.g. "we

are testing mission handover to

SAP"

The Product Increment that was

deployed at the end of each

Sprint should allow the users to

accomplish the testing goal in

the following Sprint

Feedback (new User Stories or

Defects) were considered as

part of "blocker" User Stories


3. Which new communication

system?!

New Methodology

User Acceptance

New Communicatio

n System


During the analysis phase of the project, it became clear that a very fundamental assumption did not hold. The

communication system, which is THE central system in the operations centre, also needed to be replaced. A mere

three months later, in July 2018, it was decided that a new communication system will be developed from scratch.

Roughly halfway through the project, this posed a major risk. We decided to embrace it.


We grabbed the opportunity and decided that

this could be a major chance for our

emergency dispatchers to get the system that

they both need and want

We started off by synchronising the other

project stream to our 1-week Sprint cadence

Based on 5 guiding Epics, we started

integrating from the 1st common Sprint and

conducted common reviews

After a mere 5 weeks, we could already say

that the integration risk was significantly

reduced by using user feedback and agile

methodologies

Geo-

objects

ELS Heli(Pre-)

Productive System

Mico

Funk

F24

SAP

SAP Import

MissionPlatform

ARS

MicoTest System

Make call

Open mission

Get caller details

Generate protocol entry

Replicate dataContact

Data

ELS HeliInt

ELS HeliTest

IMPORT IMPORT


4. Packing and leaving on-prem

New Methodology

User Acceptance

New Communicatio

n SystemNew

Datacentre


Public safety organisations in Switzerland

traditionally hosted most of their IT

infrastructure in their own server rooms or data

centres. Nowadays complexity and security

demands mandate that "IaaS services" be

operated by professional organisations.

Accordingly, Rega's IT strategy foresaw that by

the end of 2019, all servers should be moved to

external datacenters.

So why let's not be the first ones to be there?


In the original waterfall project plan, a so-

called "Pre-SAT" was scheduled for the 20th

of September 2019

As the new environment would allow for a

truly georedundant architecture, we decided

to conduct the Pre-SAT in the new

environment

Back then, we still needed to have core

switches installed and even had no physical

WAN connection (and only 2 months time)

Having overcome those obstacles, we had a

running (earmarked to be) production system

already 9 months before go-live


Datacenter Zurich

Rega Center

Test-System

(ZH)

Int-System (ZH)

Prod-System

(ZH)

EZ Prod/Int/Test Clients

Datacenter Bern

Int-System (BE)

Prod-System

(BE)

"Classic" three-tier architecture moved to fully

virtualised environment

Automatic failover between georedundant

sites in Zurich and Bern

Fully separated production, integration and

test environments

Integration 1:1 image of production

environment

Frequent system patching combined with

regression testing ensures system stability

and security

Arbitrarily short release cadence


5. Not everything is software

New Methodology

User Acceptance

New Communicatio

n SystemNew

DatacentreNew Client Infrastructure


Building usable software means not only to

focus on the usability of the software, but also to

consider the devices that it will be used on and

the context of usage. Always keep ergonomics

in mind.

Once again: early prototyping and feedback is

vital. Once you got a first direction you need to

test, test and test together with your users and

iterate fast.


6. Testing "the real deal"

New Methodology

User Acceptance

New Communicatio

n SystemNew

DatacentreNew Client InfrastructureIntegrationTest

Coverage


Complex infrastructures can best be

integrated and tested using iterative

approaches

In general: the earlier you integrate, the

better

It pays off to build up test capabilities that

enable true end-to-end testing

The integrated systems should be

operated in an environment that mimics

production 1:1

Keep in mind: The closer we are to reality,

the less risk we face in production


Usability testing should equally occur with

the future setup in the environment that

the users will work in

Early in the mission control system and

the communication system projects, we

conducted workshops in which the future

interplay of applications was defined

We chose the new workstation together

with the users, showing them two variants

Testing ergonomics "in the field" should be

done for each user group, as even the

"best" system may prove to be unusable

there


7. Going live and the aftermath

New Methodology

User Acceptance

New Communicatio

n SystemNew

DatacentreNew Client InfrastructureIntegrationTest

CoverageRoll-out


Guiding principle number 1: prepare what you

can prepare upfront

Guiding principle number 2: test everything that

you can test upfront, in the process and in the

aftermath

Guiding principle number 3: aim for a lazy go-live

– the less you need to do, the better

After go-live, the system continued to be in use

without any further interruption / maintenance

A mere 3 non-critical incidents occured in July

DevOps and continuous improvement of ITIL

processes

how we embraced risk to catalyse change

Documents