INSTRUCTIONAL  DESIGN  IN  ACTION:  observing  the  judgments  of  ID  practitioners      elizabeth  boling,  colin  m.  gray,  and  verily  tan  

collaborators  

-  Cesur  Dagli  

- Muruvvet  Demiral-­‐Uzan  

-  Funda  Ergulec  

- Abdullah  Altuwaijri  

-  Khendum  Gyabak  

- Megan  Hilligoss  

-  Remzi  Kizilboga  

-  Kei  Tomita  

background,  purpose,  &  previous  studies  

-  Views  on  design  have  been  shifting  in  instructional  design  -  From  prescriptive  enactment  of  theory  to  disciplined  practice  

(Bichelmeyer,  Boling  and  Gibbons,  2006;  Smith  &  Boling,  2009)  

-  From  practice  defined  by  research  to  practice  as  definitive  of  design  (Gibbons,  Boling,  &  Smith,  2014;  Schwier,  Campbell,  &  Kenny,  2007)  

-  Research  on  ID  practice  has  been  limited  - On  its  own  terms  (Rowland,  1992)  

-  Comparing  practice  to  existing  ID  models  or  frameworks  (e.g.,  ADDIE)  (Wedman  &  Tessmer,  1993;  Visscher-­‐Voerman  and  Gustafson,  2004)  

- Attempts  to  approach  the  complexity  of  practice  by  translating  it  into  explicit,  teachable  terms  (Ertmer,  York,  &  Gedik,  2009;  Ertmer  et  al.,  2009)  

design  practice  &  judgment  

- Moving  beyond  a  scientific  or  scientised  framing  of  design  practice  (Cross,  2011;  Stolterman,  2008)  

-  Reframing  our  understanding  of  practical  knowledge  (phronesis)  as  specialized  knowledge,  not  just  common  sense  that  accompanies  scientific  action  (Dunne,  1997)  

-  Related  constructs:  -  Tacit  knowledge  (Holt,  1997;  Polanyi,  1966;  Vickers,  1984)  -  Conceptual  design  sense  and  critical  flexibility  (Yanchar  

&  Gabbitas,  2011)  

- Design  character  and  judgment  (Nelson  &  Stolterman,  2012;  Korkmaz  &  Boling,  2014)  

Type Operationalized  Definition

framing Creating  a  working  area  for  design  activities  to  occur,  often  by  introducing  constraints  (client  or  tool)  or  ways  of  assessing  outcomes.  This  occurs  dynamically  across  multiple  levels.

deliberated  off-­‐hand Recalling  to  consciousness  previous  judgments  that  have  led  to  successful  practices  and  opening  them  to  the  possibility  of  adaptation  or  use.

appreciative Placing  high  value  and  emphases  on  certain  aspect/s  of  a  design  situation  while  backgrounding  others.

quality Making  design  decisions  about  the  effectiveness  of  visual  and  other  forms  of  style,  or  to  demonstrate  due  diligence,  often  in  accordance  with  company  standards,  in  relation  to  a  concrete  design  artifact.

appearance Assessment  of  overall  quality,  relating  to  an  entire  product  or  experience,  rather  than  just  a  portion.  This  often  includes  part/whole  relations  within  a  frame  of  aesthetic  experience  or  measurement  against  heuristic(s).

connective Making  connections,  or  bridging  various  design  objects  that  are  central  to  the  design  process  and  activity.  The  connections  made  in  this  context  are  not  generalized  but  specific  to  the  design  situation.

design  judgment  types  

Type Operationalized  Definition

compositional Making  connections  or  bringing  various  design  objects  together  that  are  central  to  the  design  process  and  activity.  The  connections  made  in  this  context  are  generalized  and  not  specific  to  a  particular  design  situation  but  to  the  overall  process.

instrumental The  selecting,  utilization,  or  influence  of  a  tool,  concept,  or  method  in  reaching  an  established  design  goal.

navigational Considering  a  path,  plan,  or  certain  manner  (of  individual,  disciplined  preference)  in  approaching  a  task  or  a  challenge  to  get  to  a  desired  state.

default Giving  an  automatic  response  to  a  situation  without  deliberation.

core Statement  about  one’s  value  or  thinking,  usually  revealed  when  pushed  by  “why”  questions  concerning  one’s  judgment.

design  judgment  types  

research  questions  

- What  do  IDs  do  in  practice  consistent  with  design  judgment?  

- What  design  judgments  take  place  in  ID  activities?  

method  

-  Two  sites  of  ID  practice  

-  8  practicing  IDs  

- Data  Collection  -  Field  observations  (20  hours  total)  with  handwritten  fieldnotes  

-  Follow-­‐up  interviews  with  notes  and  audio  recording  

- Analysis  - Unitized  coding  of  judgments  - Holistic  case  summaries  

findings  &    discussion  

!!

Fram

ing!

Off*han

d!

App

reciative!

Qua

lity!

App

earanc

e!

Conn

ectiv

e!

Compo

sitio

nal!

Instru

men

tal!

Nav

igationa

l!

Defau

lt!

Core!

TOTAL!

Gabriel! 9! 8! 10! 4! 5! 4! 2! 8! 5! 0! 1! 56!

Emily! 5! 6! 8! 2! 0! 3! 4! 8! 6! 10! 0! 52!

Julia! 0! 4! 3! 5! 4! 2! 4! 2! 2! 4! 0! 30!

Heather! 5! 1! 7! 0! 0! 0! 0! 0! 3! 0! 0! 16!

Ethan! 6! 0! 3! 1! 1! 0! 0! 1! 5! 2! 1! 20!

Ethan! 3! 0! 2! 1! 1! 1! 0! 1! 3! 0! 0! 12!

Claire! 3! 3! 6! 3! 1! 2! 3! 2! 3! 0! 0! 31!

Adam! 3! 3! 0! 1! 1! 1! 2! 1! 1! 0! 0! 13!

Sally! 13! 4! 4! 8! 14! 7! 5! 13! 8! 15! 1! 92!

TOTAL! 47! 29! 43! 25! 27! 20! 20! 36! 36! 31! 3! 322!!

Frequency  of  Judgments  based  on  Nelson  and  Stolterman  (2012)  Types  

judgments  are  happening    all  the  time  - Average  of  35  design  judgments  per  observation  

- Over  16  judgments  per  hour,  averaged  across  all  participants  

judgments  create  and  are  shaped  by    situational  factors  in  design  - Design  environment/office  culture  

-  Role  or  position  of  the  designer  

-  Project,  client,  and  external  factors  

 

judgments  are  clustered    and  layered  

- Multiple  design  judgment  types  are  clustered  and  layered  “in  the  wild”  

-  Stolterman:  “[judgments]  as  pearls  and  they  are  connected  with  strings…If  you  take  one  and  hold  it  up,  then  the  others  just  hang,  as  a  cluster…”  

-  Pure  philosophical  types  versus  complex,  contextually-­‐bound  expressions  in  reality  

Name   Company   Role  Years  of  

Experience   Background  

Gabriel  Campus-­‐wide  Consultancy   Media  Consultant   6  

Degree  in  Computer  Science;  Masters  in  Comm.  &  Tech.  

Emily   Established  ID  Firm   ID   0.17   ID  in  non-­‐profit  fields  

Julia   Established  ID  Firm  Senior  ID/Project  Leader   7  

Degree  in  Fine  Arts;    

Masters  in  IST  

Heather   Established  ID  Firm   Project  Manager   11  

Degree  in  English;  

Masters  in  IST  

Ethan   Established  ID  Firm   ID   2.5  

Degree  in  IT;    

Masters  in  Instructional  Tech.  

Claire   Established  ID  Firm   ID   6.5  

Degree  in  Ed.  Counseling;    

Masters  in  EdTech  

Adam   Established  ID  Firm   Course  Director   10  

Degree  in  Journalism;    

EdS  in  IST  

Sally   Established  ID  Firm   ID   3   Masters  in  Screenwriting  

example  

She  started  the  meeting  [with  a  client  via  teleconferencing]  with  questions.  She  needs  clarification  on  the  document  to    understand  the  content  better  and  said  she  wanted  to  ask  questions  to  the  person  who  created  high  level  outline.  [...]  She  continued  making  clarification  on  the  understanding  of  the  content  and  she  asked  “what  communication  skills  and  active  listening  skills  mean.  She  wanted  to  sure  whether  what  she  understood  is  same  what  they  mean  with  these  terms.  Then,  she  stated  that  she  would  like  to  have  the  definition  of  “active  listening  skills.  (Claire,  123-­‐124;  164-­‐166)  

limitations  

- Data  limited  to  one  ID  at  a  time,  although  others  appear  in  the  frame  

- No  continuity  across  projects  or  project  teams  -  Convenient  sample,  with  most  IDs  originally  educated  in  a  single  ID  program  

- Over-­‐explanation  of  practice  in  some  instances  - Operationalization  of  a  philosophical  model  of  judgment  is  incomplete  

implications  &  future  research  

-  Judgments  are  made  on  a  continuous  basis  throughout  projects,  and  not  just  as  mental  “adjustments”  to  models;  these  judgments  cannot  be  discovered  through  the  application  of  a  priori  scientific  models  

- Move  toward  understanding  practice  on  its  own  terms,  using  this  as  a  base  for  scholarship  that  can  then  improve  practice  

———————————  - We  know  almost  nothing  about  this  space  -  Knowledge  of  professional  practice  is  highly  important  for  ID  education,  including  engagement  in  authentic  situations  

references  Bichelmeyer,  B.,  Boling,  E.,  &  Gibbons,  A.  (2006).  Instructional  design  and  technology  models:  Their  impact  on  research,  practice  and  

teaching  in  IDT.  In  M.  Orey,  J.  McLendon  &  R.  Branch  (Eds.).  Educational  Media  and  Technology  Yearbook  2006.  Westport,  CT:  Libraries  Unlimited.  

Cross,  N.  (2011).  Design  thinking:  Understanding  how  designers  think  and  work.  New  York,  NY:  Berg.  Dunne,  J.  (1997).  Back  to  the  rough  ground:  Practical  judgment  and  the  lure  of  technique.  Notre  Dame,  IN:  University  of  Notre  Dame  

Press.  Ertmer,  P.  A.,  York,  C.  S.,  &  Gedik,  N.  (2009).  Learning  from  the  pros:  How  experienced  designers  translate  instructional  design  models  

into  practice.  Educational  Technology,  49(1),  19-­‐27.  Ertmer,  P.  A.,  Stepich,  D.  A.,  Flanagan,  S.,  Kocaman-­‐Karoglu,  A.,  Reiner,  C.,  Reyes,  L.,  ...  &  Ushigusa,  S.  (2009).  Impact  of  guidance  on  the  

problem-­‐solving  efforts  of  instructional  design  novices.  Performance  Improvement  Quarterly,21(4),  117-­‐132.  Holt,  J.  E.  (1997).  The  designer's  judgement.  Design  Studies,  18(1),  113-­‐123.  Korkmaz,  N.,  &  Boling,  E.  (2014).  Development  of  design  judgment  in  instructional  design:  Perspectives  from  instructors,  students,  and  

instructional  designers.  In  Design  in  educational  technology  (pp.  161-­‐184).  Switzerland:  Springer.  Nelson,  H.  G.,  &  Stolterman,  E.  (2012).  The  design  way:  Intentional  change  in  an  unpredictable  world  (2nd  ed.).  Cambridge,  MA:  MIT  Press.  Polanyi,  M.  (1966).  The  tacit  dimension.  Garden  City,  NY:  Anchor  Books.  Rowland,  G.  (1992).  What  do  instructional  designers  actually  do?  An  initial  investigation  of  expert  practice.  Performance  Improvement  

Quarterly,  5(2),  65-­‐86.  Gibbons,  A.  S.,  Boling,  E.,  &  Smith,  K.  M.  (2014).  Instructional  design  models.  In  Handbook  of  research  on  educational  communications  

and  technology  (pp.  607-­‐615).  New  York,  NY:  Springer.  doi:10.1007/978-­‐1-­‐4614-­‐3185-­‐5_48  Schwier,  R.  A.,  Campbell,  K.,  &  Kenny,  R.  F.  (2007).  Instructional  designers'  perceptions  of  their  agency:  Tales  of  change  and  community.  

In  M.  J.  Keppell  (Ed.),  Instructional  design:  Case  studies  in  communities  of  practice  (pp.  1-­‐18).  Hershey,  PA:  Information  Science  Publishing.  

Smith,  K.  M.,  &  Boling,  E.  (2009).  What  Do  We  Make  of  Design?  Design  as  a  Concept  in  Educational  Technology.  Educational  Technology,  49(4),  3-­‐17.  

Stolterman,  E.  (2008).  The  nature  of  design  practice  and  implications  for  interaction  design  research.  International  Journal  of  Design,  2(1),  55-­‐65.  

Vickers,  S.  G.  (1984).  Judgment.  In  The  Vickers  papers  (pp.  230-­‐245).  London,  UK:  Harper  &  Row.  Visscher-­‐Voerman,  I.,  &  Gustafson,  K.  L.  (2004).  Paradigms  in  the  theory  and  practice  of  education  and  training  design.  Educational  

Technology  Research  and  Development,  52(2),  69-­‐89.  Yanchar,  S.  C.,  &  Gabbitas,  B.  W.  (2011).  Between  eclecticism  and  orthodoxy  in  instructional  design.  Educational  Technology  Research  and  

Development,  59(3),  383-­‐398.  doi:10.1007/s11423-­‐010-­‐9180-­‐3  Wedman,  J.  and  Tessmer,  M.  (1993),  Instructional  Designers  Decisions  and  Priorities:  A  Survey  of  Design  Practice.  Performance  

Improvement  Quarterly,  6(2):  43–57.