reliability and checklists - stanford universityweb.stanford.edu/~kunz/chalmers/reliability.pdf ·...
TRANSCRIPT
Center for Integrated Facility Engineering
Reliability and checklists
John Kunz
CIFE, Stanford University
Center for Integrated Facility Engineering
Pre-plan examples
• Sketch
• Build checklist
• Do
• Monitor and challenge
Problems Outcomes Resources
Problems
for session
to focus on
Desired outcomesIntended
Participants
Participant
discipline
Pre-session
assignments
Member of
pre-plan
team
(yes/no)
Role in ICE
session
Member of
post-
session
wrapup
team
(yes/no)
Agenda items
Outcome intent
met?
(Yes/Partial/No)
Meeting
space,
technologies,
models, tools
Specify
architectural
spaces first
floor
Spaces to model are
specified
Mary Architect Owner share
project goals and
objectives
Yes Discipline expert Yes Review project
goals and
objectivesPartial
Smart Boards
Size building
systems
Systems in first floor are
sized and specified
Joe MEP Engineer Define and share
team charter
No Discipline expert No Specify spaces,
systems,
components to
model
Yes
Excel
Identify building
construction
components
for first floor
Components (first floor)
that take > 1 hour to
install are listed and
added to spec of
components to model
Hamid PM PM assure
availability of BIM
authoring and
review tools for
team members
Yes Facilitator Yes assign modeling
tasks to individuals
No
Meeting space
with tables,
chairs for team
of 10
ICE pre-plan
Participants Agenda
Center for Integrated Facility Engineering
Product sketches
• How to clarify their meaning for project
Center for Integrated Facility Engineering
Submission-2
Comments and questions
Center for Integrated Facility Engineering
Agenda – Thursday
• Reflections, review of Tuesday
• Checklists for reliability
• Submission-1 Examples
• ICE session: project definition w/organization
analysis and optimization
• Initial student presentations
• SimVision charrette to analyze and optimize a
predefined baseline organization
Center for Integrated Facility Engineering
Reliability Overview
Session Objectives
[Lecture/
discussion;
Understand and experience:
• Reliability issues in AEC globally
• Invisibility of reliability and risk in projects
• Checklists to improve reliability
• Risk assessment to identify reliability
10
Center for Integrated Facility Engineering
Reliability
Data from Chile post-earthquake
– ~500K/~5M homes
damaged or destroyed: <2σ
– ~4 joint failures /~100 in
(collapsed) buildings: 2σ
– 4/~10,000 post-1985
buildings collapsed in major
damage area: >3σ
Conclusions: Chile and
developed economies
• We can design and
construct structures well
• For structures,
– Statistical performance
OK to good
– Society needs higher
reliability from AEC
– Reliability of (design-
construction-operation)
management is the issue
11
Center for Integrated Facility Engineering
Invisibility of reliability and risk in projects
Project success factors:
Cost
Schedule
Quality
Safety
• Reliability -
Indicates that projects predict, measure and manage
against these factors
12
Center for Integrated Facility Engineering
Risk assessment
• Do formal risk assessment periodically during project
– Small number of “experts”
• Consider factors that affect life-cycle risk/success
– Market and demographic changes
– Practice (methods) changes
– Natural and societal challenges
– Design/construction risks
• Create mitigation activities to address biggest risks
– Flexibility (buffer) in design
– Checklists in design/construction processes
13
Center for Integrated Facility Engineering
• Spectrum of task types:
– Simple: many people can quickly learn to do the
work, e.g., shovel gravel
– Specialized: large amounts of training need to learn
to do the (virtuoso) work, e.g., Master-builder
(pilot/surgeon); high strength welding
– Complex: High uncertainty and knowledge required
exceeds that of any individual, e.g., construction
(flight/surgical) team; design structural system
• Reliability management a requirement in situations with complexity:
– w/complexity, checklists required for success
Reliability issues
14
Center for Integrated Facility Engineering
Causes of low-reliability
• Knowledge missing: we do not know how to do something we
are trying to do
– The fact that virtually all buildings have actual
energy >> predicted suggests that we lack some
important knowledge
• Incompetence: we do not do something we know how to do
– The fact that most buildings in Chile withstood big
earthquakes suggests that we know how to design
and build for earthquake performance
15
Center for Integrated Facility Engineering
Checklists address the competence problem
Good Checklists foster teamwork and discipline. They
• Define
– Tasks (“stupid” things) to do
– Collaboration (Information to exchange) among multiple
responsible stakeholders
• Allow specialists freedom to manage unpredicted conditions
• Need to balance
– Freedom (judgment) vs. discipline (procedure)
– Craft flexibility vs. protocol structure
– Specialty action vs. group collaboration
• Enable practice to “monitor and challenge”
16
Center for Integrated Facility Engineering
Reliability in other industries
• Airline operations, surgery:
– Lots of performance data, e.g., on-time records; “evidence
based medicine”
– Checklists, e.g.,
• Flight checklist for entire aircraft crew; Pause time for
checks in surgery
• Carefully crafted by industry, company, local groups
• Apply to very small steps in processes that have
statistical reliability problems; not used for many steps
• Designed to provide structure and professional flexibility
• Enable extremely high process reliability when used
broadly
17
Center for Integrated Facility Engineering
Culture of effective Checklist use
• Power: for complex, non-routine problems, push to source
of greatest actionability: the periphery
– Away from center
• Senior management:
– Sets goals and objectives
– Measures progress
– Facilitates (and assures) communication
• Team members:
– Have shared responsibility to act, to check: monitor
and challenge
– Act professionally: selfless, skilled, trustworthy,
disciplined
18
Center for Integrated Facility Engineering
Culture of effective Checklist use
• Initiate activation phenomenon
– Share names (if not already known)
– Share concerns and observations at start of each
major activity, e.g., the day
more engagement in checklist use
• Of marginal help, at best
– Published books of standards
– Financial incentive for performance
19
Center for Integrated Facility Engineering
Effective Checklists
• Precise: Clear, relatively short (~5-9 major steps), uncluttered
– Include “Do Confirm” steps
• Often part of a family of checklists, e.g., checklist for each day, for
different circumstances
• Have strengths:
– Help experts remember important steps in complex procedures
– Clarify priorities and sequences
– Prompt teamwork
– Enable rapid dissemination of new knowledge
– Do work – often dramatically
• Have limits:
– Cannot do skilled work
– Cannot force conformance
– Require training
– Require careful development
20
Center for Integrated Facility Engineering
AEC Checklists that can help reliability
• Examples:
– Gantt charts, deliverable schedules, punch lists, w/
• Tasks (process steps)
• Responsibilities: who does task; who helps
• Options, e.g., resources, start/end times, next steps
(successors)
– Commitments:
• Formal: RFIs, formal meetings, punch lists
• Informal: emails, informal meetings
– VDC checklists: process to specify functional intent,
design content and behaviors of project models …
and projects
21
Center for Integrated Facility Engineering
AEC Reliability data
• Large amounts of data are needed to identify process steps that affect life
or company health and safety
– Some industries have accident cause data, e.g., OSHA, NTSB
– Some performance data exist in AEC, e.g.,
• RS Means cost
• OSHA Total Construction safety Incidence Rates per 100
Full-time Workers:
– 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007
– 8.8 8.6 8.3 7.9 7.1 6.8 6.4 6.3 5.9 5.4
– Process reliability data largely invisible, even within companies
– Process reliability must be exceptionally high (> 6σ) to assure
acceptable lifecycle product reliability
22
Center for Integrated Facility Engineering
Management
practices
Hospital
Operations
Management
Performance
and target
management
Talent and People
management
Example dimensions
evaluated
Performance dialogue
and review
Interconnection
of targets
Consequence
measurement
Target balance
Performance tracking
3.5) Performance is
continuously tracked and
communicated, formally
and informally, to all staff
using a range of visual
management tools
1.1) Measures tracked do not
indicate directly if overall
hospital objectives are
being met. Tracking is
ad hoc
2.3) Most key performance
indicators are tracked
formally. Tracking is
overseen by senior staff
Dimension
Scoring criteria
Management Quality Survey: 20 questions in
three broad areas [Bloom]
24
Center for Integrated Facility Engineering 25
Intervention aimed to improve 38 core textile
management practices in 6 areas – for example:
India project
25
Center for Integrated Facility Engineering 26
Intervention aimed to improve 38 core textile
management practices in 6 areas – for example:
Frequent monitoring
Management Context
India project
26
Center for Integrated Facility Engineering
Treatment plants
Control plants
Share
of
key t
extile
manag
em
ent
pra
ctices a
dopte
d
Excluded plants
(not treatment or control)
Experiment gave treatment plants free consulting for 5 months
- adoption of these 38 management practices rose .2
.3
.4
.5
.6
-10 -8 -6 -4 -2 0 2 4 6 8 10 12 Months after treatment
Shar
e o
f 3
8 m
anag
emen
t p
ract
ices
ad
op
ted
Intervention
Treated
Control
27
India project
27
Center for Integrated Facility Engineering
02
04
06
08
01
00
12
01
40
-20 -10 0 10 20 30 40weeks since diagnostic phase
2.5th percentile
Performance improved – e.g., quality defects down 50%
Control plants
Treatment plants
Weeks after the start of the diagnostic
Qu
alit
y d
efe
cts
in
de
x (
hig
he
r sco
re=
low
er
qu
alit
y)
Start of
Diagnostic
Start of
Implementation
Average (+ symbol)
97.5th percentile
Average (♦ symbol)
97.5th percentile
End of
Implementation
2.5th percentile
India project
28 28
Center for Integrated Facility Engineering
Example Checklist Step in detailed
schedule Checks to make
before start of step
29
Monitor and Challenge
Center for Integrated Facility Engineering
Example Checklist Step in detailed
schedule Checks to make
before start of step
30
Monitor and Challenge
Center for Integrated Facility Engineering
Impacts of Safe Childbirth Checklist
Single centre testing, Gokak, India
Preliminary findings
• 3 months of pre-
intervention data
• Data on ~400
childbirths collected
(admission to
discharge)
• 3 weeks of checklist
implementation data
Indicator (selection) Baseline Post-
Intervention
Washing hands and
wearing gloves when
doing vaginal examination
or delivery
<10% >90%
Checking baby’s
temperature after birth <5% >90%
Routinely administering
Oxytocin within 1 minute
after birth
<5% >90%
32
Center for Integrated Facility Engineering
Checklist templates and VDC
We can use checklists to do Project definition (v) and define the:
1. Functional objectives – what you want for your project;
2. Scope or form – what you will or did do;
3. Behaviors – predictions and measurements of what you did
We can use these (actionable) checklist templates to:
• Specify work we do, e.g., BIM (create checklist)
• Verify work (e.g., BIM content) wrt specification (w/checklist) –
monitor and challenge
• Store results of analysis (in checklists)
• Evaluate how well the designed/built scope responds to objectives
for the Product, Organization and Process, i.e., manageable
elements of the project (using a checklist)
33
Center for Integrated Facility Engineering Inspired by Science and Decisions: Advancing Risk
Assessment, National academies press, 2009.
Questions to consider in Project Definition …
actionability for team, for us
• What is the problem? … what actions can a manager take?
– Manager can adjust functions, scope and details of product,
organization and process
– Project definition defines them
• For whom is it a problem?
– Project definition describes the organization and identifies the
stakeholders in the decision(s) that may/will face the problem
• Who is the decision maker(s)
– Project definition identifies stakeholders who can act
• Why do the modeling and analysis?
– Project definition identifies impact of performance on assessed
goodness|objectives, which allows team to identify impacts of choices
• So what?
– Adjust modeling and analysis level of detail to fit the impact of the
decision on form or scope, schedule, quality and risk
34
Center for Integrated Facility Engineering
Product breakdown structure: Design scope or form of Product
Content of Commitments – BIM content template
35
Building
Components Spaces Systems
HVAC Patient
rooms x 400
Services
e.g., CT
iRoom work
space
Public areas
Foundation
Above-
ground steel
Wall
systems
Skin
Windows
Roof
Legend
Type of product
Product sub-type
Note: functional intent of
product element is implicit
A LOD
B LOD
Specification of BIM
contentBIM elements
Level of detail (LOD) -
AIALevel of detail LOD) - comments
Due date
of next
BIM
version
BIM
content
conforms
to spec
[Yes/No]
Components Foundation 100 (Conceptual) Minimum dimension > 4"
Components Above-ground steel 100 (Conceptual) Minimum dimension > 4"
Components walls - Exterior, basic200 (Approximate
geometry)Minimum dimension > 4"
Components walls - Interior - Partition 100 (Conceptual) Minimum dimension > 4"
Components Roof 300 (Precise geometry) Architectural, not construction detail
Components Windows 300 (Precise geometry) Architectural, not construction detail
Components Doors200 (Approximate
geometry)
Spaces Rooms 100 (Conceptual) at least 2 rooms; size of each 150 - 250
ft2
BIM Content Specification
Center for Integrated Facility Engineering
Work breakdown structure: Design scope or form of Work
Content of Commitments – production plan template
36
Tasks
Design Build Commission Operate
Design: building
elements
[Architect]
Assess:
Behaviors
[Owner]
Design: Building
systems
[HVAC/MEP designers]
Design: Building
systems [Architect]
Legend
Type of process task
Task sub-type
Note: functional intent of
task element is implicit
A LOD
B LOD
Task Priority Short descriptionResponsible
team
Budget (FTE-
hours)
Coordination
dependent
team(s)
Approval
teamDue date
Done on-
timeComments/ model image(s)
1
A (Contract
requirement) Add door in BIM PM 20 Architect PM 1/1/2010 No
2
Commitments - example
Center for Integrated Facility Engineering
Metrics Implementation template
(C) 2013
37
P: Process; O: Outcome
Prediction
Name CommentTarget
valueTolerance: +-∆
How to use in
managementSource of data
Type [P,
O]
Stakeholder
s who saw
data last
week
Collection
frequency Objective Weight
Predicted/
measured
value (how
you are
doing)
Assesed
value
MQuality: POE satisfaction wrt
program (%)100 5
Guide
commissioning,
next job
Client
assessmentO Owner only
Turnover time
+ 6-24 months40 86 1
ECost conformance to plan (item
actual - predicted/predicted)100 5 Plan next job
Client
assessmentO PM only Turnover time 25 98 3
T Schedule conformance to plan (%) 100 10 Plan next jobClient
assessmentO Alll on team Turnover time 35 ? NA
RPredicted Cost conformance to plan
(item actual - predicted/predicted)100 5
Attention
management
Periodic project
progress reportP Subteam only Weekly 15 99 3
IProduction schedule conformance
to plan (%)90 10
Attention
management
Periodic project
progress reportP Alll on team Weekly 10 75 2
CAssessed Quality conformance to
plan (% of items with rating >=4 on
scale 1:5)
100 10Attention
management
Periodic project
progress reportP Subteam only Weekly 15 95 3
SStakeholder participation that is
timely and meaningful (%) 90 10
Adjust plan by
stakeholder review
and assessments
Periodic project
stakeholder
survey
P Alll on team Weekly 10 90 3
Metrics
Intent Evaluation