why and how you value your plumber: establishing and conveying value in the outcomes of a data...
TRANSCRIPT
Why and how you value your plumber:
establishing and conveying value inthe outcomes of a data virtual
organization.
Why and how you value your plumber:
establishing and conveying value inthe outcomes of a data virtual
organization.
Peter Fox (TWC/RPI)ESIP, Jan 4. 2011
Peter Fox (TWC/RPI)ESIP, Jan 4. 2011
What you will hearWhat you will hear
A bit of history – plumbers (PG13) Why – VOs: Outcomes and value How
Evaluating – really, really? Provenance for a virtual organization?
Opportunities for ESIP and Type Is A rough attempt to pull it all
together
A bit of history – plumbers (PG13) Why – VOs: Outcomes and value How
Evaluating – really, really? Provenance for a virtual organization?
Opportunities for ESIP and Type Is A rough attempt to pull it all
together
ESIP historyESIP history 11th + year – the honeymoon is well over,
past the 7-year itch, now it’s time for ESIP and those that fund it to get comfortable
Plumbers – I promise not to over-do this analogy but it arose (surprise) from the early years of DODS, nka OPeNDAP
The way to make data matter is to get it from those that have to those that want it Inlets and outlets
Infrastructure is required Computers and people Fittings, pipes and yes plumbers
11th + year – the honeymoon is well over, past the 7-year itch, now it’s time for ESIP and those that fund it to get comfortable
Plumbers – I promise not to over-do this analogy but it arose (surprise) from the early years of DODS, nka OPeNDAP
The way to make data matter is to get it from those that have to those that want it Inlets and outlets
Infrastructure is required Computers and people Fittings, pipes and yes plumbers
Virtual Organizations, i.e. ESIP
Virtual Organizations, i.e. ESIP
‘A VO is a group of individuals whose members and resources may be dispersed geographically and institutionally, yet who function as a coherent unit with a common goal.’ (Beyond Being There 2008) or a long-term common interest (Ahuja)
ESIP has VOs within VOs, i.e. some organizational structure; executive committee, committees, working groups, clusters
Many goals, different levels of coherency
‘A VO is a group of individuals whose members and resources may be dispersed geographically and institutionally, yet who function as a coherent unit with a common goal.’ (Beyond Being There 2008) or a long-term common interest (Ahuja)
ESIP has VOs within VOs, i.e. some organizational structure; executive committee, committees, working groups, clusters
Many goals, different levels of coherency
Reminder: roles and relationships
Reminder: roles and relationships
‘The virtual research organization: in which members of various corporate and academic research units voluntarily come together to advance a technology on an ongoing basis.
These members assume well defined roles and status relationships within the context of the virtual group that may be independent of their role and status in the organization employing them’ (Ahuja et al., 1998).
I.e. do your goals match ESIPs?
‘The virtual research organization: in which members of various corporate and academic research units voluntarily come together to advance a technology on an ongoing basis.
These members assume well defined roles and status relationships within the context of the virtual group that may be independent of their role and status in the organization employing them’ (Ahuja et al., 1998).
I.e. do your goals match ESIPs?
Contrasting a Virtual Information Organization with a
Traditional one
Contrasting a Virtual Information Organization with a
Traditional one
Successes focus on valueSuccesses focus on value
Evaluation of a VO (1)?Evaluation of a VO (1)? An assessment of the overall
effectiveness of an effort, ideally yielding a numeric measure by which informed cost-benefit analysis of investment decisions can be made.
An assessment of the degree to which the effort fulfills its specification in terms of whatever measures were pre-specified.
An assessment of the overall effectiveness of an effort, ideally yielding a numeric measure by which informed cost-benefit analysis of investment decisions can be made.
An assessment of the degree to which the effort fulfills its specification in terms of whatever measures were pre-specified.
Evaluation of a VO (2)?Evaluation of a VO (2)? An assessment of whether the effort fulfills the
purpose for which it was intended An assessment of whether the artifacts or
processes embodied in the effort have been proved to be superior to an alternative, where that alternative is frequently the traditional solution
An assessment of whether funds allocated to an effort have been productively used, yielding useful generalizable results
An assessment of whether the effort fulfills the purpose for which it was intended
An assessment of whether the artifacts or processes embodied in the effort have been proved to be superior to an alternative, where that alternative is frequently the traditional solution
An assessment of whether funds allocated to an effort have been productively used, yielding useful generalizable results
Evaluation of a VO (3)?Evaluation of a VO (3)? An assessment of whether the effort proves
acceptable to the intended end-users An assessment of whether end-users continue
to use it in their normal work An assessment of where the effort fails to
perform as desired or as is now seen to be desirable
An assessment of the relative importance of the inadequacies of the effort
An assessment of whether the effort proves acceptable to the intended end-users
An assessment of whether end-users continue to use it in their normal work
An assessment of where the effort fails to perform as desired or as is now seen to be desirable
An assessment of the relative importance of the inadequacies of the effort
(Orthogonal) Dimensions of evaluations
(Orthogonal) Dimensions of evaluations
Structured Less structured
Quantitative Qualitative
Summative Formative
Controlled experiments Ethnographic observations
Formal and rigorous Informal and opportunistic
http://janus.ucc.nau.edu/edtech/etc667/proposal/evaluation/summative_vs._formative.htm
Formative and SummativeFormative and Summative Evaluation carried out for two reasons:
giving feedback = formative evaluation grading the outcome = summative evaluation
“When the cook tastes the soup, that’s formative; when the guests taste the soup, that’s summative." (Stakes)
What you use to evaluate (formative) is often different from what others use (summative)
Evaluation carried out for two reasons: giving feedback = formative evaluation grading the outcome = summative evaluation
“When the cook tastes the soup, that’s formative; when the guests taste the soup, that’s summative." (Stakes)
What you use to evaluate (formative) is often different from what others use (summative)
Summative == MetricsSummative == Metrics Things you can measure (numerical) Things that are categorical
Could not do before Faster, more complete, less mistakes, etc. Wider range of users What others for ESIP?
Measure or estimate the baseline before you start
Things you can measure (numerical) Things that are categorical
Could not do before Faster, more complete, less mistakes, etc. Wider range of users What others for ESIP?
Measure or estimate the baseline before you start
Evaluation questions: To what extent does ESIP
Interviews/ Focus Group
Surveys Document Analysis
Observation
Activities enhance end-user access to and use of data to advance science and education needs?
X X X X
Activities enable higher levels of semantic capability and interoperability such as explanation, reasoning on rules, and semantic query?
X X X X
Contribute to the development and support of community resources, data systems and provision of results from diverse observing systems using suitable technologies?
X X X X
15
Evaluation questions and associated data collection
methods
Evaluation questions: Interviews/ Focus Group
Surveys Document Analysis
Observation
What factors impede or facilitate progress toward ESIP goals?
X X
What progress has been made toward sustaining and ‘scaling up’ ESIP activities?
X X X
Does ESIP contribute to the reports on modern data frameworks, user interfaces, and science progress achieved?
X X X X
Does ESIP incorporate user experiences in the evolution of its organization?
X X X X
16
Evaluation questions and associated data collection
methods
VOs - artifacts or processes?
VOs - artifacts or processes?
Both – i.e. what is created, documented, saved as well as the things that were done along the way and by whom, and how, etc.
E.g. community building, inter-agency working activities, tutorials, …
VOs tend to deal better with artifacts than processes
Both – i.e. what is created, documented, saved as well as the things that were done along the way and by whom, and how, etc.
E.g. community building, inter-agency working activities, tutorials, …
VOs tend to deal better with artifacts than processes
Recall VO- coupled elements
Recall VO- coupled elements
Technology
Communication Patterns
OrganizationalStructure
Some of this is, er…Some of this is, er… Provenance - Origin or source from
which something comes, intention for use, who/what generated for, manner of manufacture, history of subsequent owners, sense of place and time of manufacture, production or discovery, documented in detail sufficient to allow reproducibility (or assess value!)
Yes, apply this to ESIP – VO provenance
Provenance - Origin or source from which something comes, intention for use, who/what generated for, manner of manufacture, history of subsequent owners, sense of place and time of manufacture, production or discovery, documented in detail sufficient to allow reproducibility (or assess value!)
Yes, apply this to ESIP – VO provenance
And the reality?And the reality?It’s about the questions that are being asked, e.g.What exactly happened during that technical workshop last summer and how has it helped our agency?Exactly what value do you (or we) get by you being the chair of a Committee (Working Group, Cluster)?
It’s about the questions that are being asked, e.g.What exactly happened during that technical workshop last summer and how has it helped our agency?Exactly what value do you (or we) get by you being the chair of a Committee (Working Group, Cluster)?
The ecosystem?The ecosystem? These are what enable anyone to explore/
confirm/ deny their value perceptions or get quality answers to direct questions…
These are what enable anyone to explore/ confirm/ deny their value perceptions or get quality answers to direct questions…
Accountability
ProofExplanation Justification Verifiability
‘Transparency’ (the illusion of it)
Trust
Provenance - Internal/ External
Identity
Why an illusion?Why an illusion? It’s not that the word transparency is
wrong, it is what it is being used for – “If I let you see everything, you can get
answers to your questions”
Nope, not unless you are very lucky… It depends on
Who is asking the question (and why) What the answer will be used for CONTEXT and ROLE (poorly represented)
It’s not that the word transparency is wrong, it is what it is being used for – “If I let you see everything, you can get
answers to your questions”
Nope, not unless you are very lucky… It depends on
Who is asking the question (and why) What the answer will be used for CONTEXT and ROLE (poorly represented)
So … translucencySo … translucency
See just what is necessary and sufficient Practical definition
As close to the relevant data, information and knowledge artifacts presented in an appropriate form
Yes, I mean curation for evaluations and value assessments
Methodological means - many
See just what is necessary and sufficient Practical definition
As close to the relevant data, information and knowledge artifacts presented in an appropriate form
Yes, I mean curation for evaluations and value assessments
Methodological means - many
And some …And some … Identity
YOUR identity Friends, organizations Communities Peer and non-peer relations
Accountability By whom, to whom When and how often
Documentation – are you happy Ted?
Identity YOUR identity Friends, organizations Communities Peer and non-peer relations
Accountability By whom, to whom When and how often
Documentation – are you happy Ted?
Just rememberJust remember
Plumbers fix things, they show up, the do the work, they leave and then you get the bill… they work in an analog world
Thus the value proposition is different (you all know that) in a digital world
So, we need a …
Plumbers fix things, they show up, the do the work, they leave and then you get the bill… they work in an analog world
Thus the value proposition is different (you all know that) in a digital world
So, we need a …
We need a Knowledge BaseWe need a Knowledge Base
26
Populating the KB: Communication
Populating the KB: Communication
A key feature of virtual organizations is a high degree of informal communication
Because of a lack of formal rules, procedures, clear reporting relationships, and norms, more extensive informal communication is required
-> as a result a lot of VO provenance is either Not collected Collected but fragmented, disconnected Not identified as provenance (when it is created)
A key feature of virtual organizations is a high degree of informal communication
Because of a lack of formal rules, procedures, clear reporting relationships, and norms, more extensive informal communication is required
-> as a result a lot of VO provenance is either Not collected Collected but fragmented, disconnected Not identified as provenance (when it is created)
OpportunitiesOpportunities To undertake both formative (ESIP) and
summative (Type Is) evaluations Give some level of formalization to the meaning/
terms and relations – progress An incentive to identify provenance of the VO
that is ESIP ‘Rules’ to define the combinations of evidence
required – value – ESIP and Is Type Is - some use cases (state+value) A few test cases, I/O metrics
To undertake both formative (ESIP) and summative (Type Is) evaluations
Give some level of formalization to the meaning/ terms and relations – progress
An incentive to identify provenance of the VO that is ESIP
‘Rules’ to define the combinations of evidence required – value – ESIP and Is
Type Is - some use cases (state+value) A few test cases, I/O metrics
Comment: Translucent VOs
Comment: Translucent VOs
Along the spectrum from opaque (bad) to transparent (unsustainable) is a balance
Applies to leadership, communication, feedback Posit: balance arises out of flowing
requirements consistently to the working level of the VO and percolating direct experience and value of outcomes up
Usually depends on the right people – not everyone can evaluate (or be evaluated)
Along the spectrum from opaque (bad) to transparent (unsustainable) is a balance
Applies to leadership, communication, feedback Posit: balance arises out of flowing
requirements consistently to the working level of the VO and percolating direct experience and value of outcomes up
Usually depends on the right people – not everyone can evaluate (or be evaluated)
Bringing this togetherBringing this together A persistent presence and record of artifacts, activities
and processes is needed: state and metrics/ value No, this is currently not being done (or feasible)…
Facilitate sustained activities - support the notion of nested VOs by establishing their value (may change funding model)
But really - take ESIP from an organization to a ‘brand’ (like AMA) – then you know who they are gonna call (w/ apologies to Ghostbusters – that was the other story I could have told)
A persistent presence and record of artifacts, activities and processes is needed: state and metrics/ value No, this is currently not being done (or feasible)…
Facilitate sustained activities - support the notion of nested VOs by establishing their value (may change funding model)
But really - take ESIP from an organization to a ‘brand’ (like AMA) – then you know who they are gonna call (w/ apologies to Ghostbusters – that was the other story I could have told)
