wp4 presentation, tencompetence kick-off, dec 8 th – 9 th 2005
TRANSCRIPT
WP4 Presentation,TenCompetence Kick-off, Dec 8th – 9th 2005
WP4 Presentation,TenCompetence Kick-off, Dec 8th – 9th 2005
WP4:Pilots with the Integrated System
Validation of the Project UPF 8, ALTRAN 7, LogicaCMG 3, OUNL 3, GIUNTI 5, CERTH 4, SU 4, SYN 2
Dai Griffiths, Josep Blat, UPF
WP4 Presentation,TenCompetence Kick-off, Dec 8th – 9th 2005
Aims of the (integration) activity
• To provide realistic scenarios and corresponding pilots which constitute an effective environment for evaluation of the project, whereby the systems resulting from the research and development can be challenged and validated
• To provide the overall project evaluation strategy
WP4 Presentation,TenCompetence Kick-off, Dec 8th – 9th 2005
Tasks, deliverables, timetable (I)
• T1. Define initial pilot scenarios (m6)• T2. Define the pilot evaluation plan (m6)• T5. Develop an overall evaluation plan (m6)• D4.1: Report (m 6) on:
– scenarios and pilots definition– user requirements and validation criteria for the
overall project– evaluation strategies and initial evaluation plan
implementation
WP4 Presentation,TenCompetence Kick-off, Dec 8th – 9th 2005
Tasks, deliverables, timetable (II)
• T3. Set up the initial pilots (m6-12)• T4. Validate the initial system through the evaluation
of the Digital Cinema pilot (m12-18)• T6. Generate user requirements for cycle-2 pilots
(m12-18)• D4.2: Report on results of first evaluation of pilots
(month 18)– Suggested interim report on m12 to submit for the review
WP4 Presentation,TenCompetence Kick-off, Dec 8th – 9th 2005
Pilots envisaged
• Pilot 1 Training for digital movie production (UPF)
• Pilot 2 Training for continuous education of health workers (UPF)
• Pilot 3 Unesco-IHE – Institute for Water Education (?)
• Pilot 4 The Antwerp Lifelong Learning City (?)– Request an initial description
WP4 Presentation,TenCompetence Kick-off, Dec 8th – 9th 2005
Pilots (I)
• T1. Define initial pilot scenarios (up to m6)– Pilot 1: Digital movie production
• Partners (?)– Visual effects and other UPF postgraduate courses– Others from possible partners: Mediapro, Brainstorm, Thomson, Barco,
XDC, Filmlight, DVS, Pandora (MediaSalles, ...)– Commitments definition: elaborate on association document
• User requirements, validation criteria, evaluation• Target groups definition: learning providers, learning designers,
content, tutors, learners ... availability: STRESS THE IMPORTANCE OF A COMMUNITY OF USERS
• Plan for the implementation: define competencies based syllabi, ... INSERT HERE MATERIAL FROM IP-RACINE – DISCUSS TRANSFORMATION INTO COMPETENCE ORIENTED – DISCUSS THE CHALLENGES POSED BY THE PILOT
– Establish a model for the other pilots
WP4 Presentation,TenCompetence Kick-off, Dec 8th – 9th 2005
Pilots (II)
• T1. Define initial pilot scenarios (up to m6)– Pilot 2: Training for continuous education of
health workers• Partners (?) Hospital Clínic, Doyma (others)• Gastroenterology/hepatology, Pediatrics, Colo-rectal cancer
screening, ...• User requirements, validation criteria, evaluation• Target groups definition: learning providers, learning
designers, content, tutors, learners ... availability
WP4 Presentation,TenCompetence Kick-off, Dec 8th – 9th 2005
Pilots (III)
• T1. Define initial pilot scenarios (up to m6)– Pilot 3: Unesco-IHE – Institute for Water
Education• ?
WP4 Presentation,TenCompetence Kick-off, Dec 8th – 9th 2005
Pilots (IV)
• T1. Define initial pilot scenarios (up to m6)– Pilot 4 The Antwerp Lifelong Learning City
• ?
WP4 Presentation,TenCompetence Kick-off, Dec 8th – 9th 2005
Evaluation planning and initial implementation
• T2. Define the pilot evaluation plan, specifying the validation criteria and the strategies for evaluation (up to m6)
• T5. Develop an overall evaluation plan (up to m6)• Validation focus along cycles
– 'proof of concept' during the first cycle through the Digital Cinema pilot
– 'usability' during the second cycle through Health Workers pilot, the UNESCO-IHE pilot, the Antwerp Lifelong Learning City pilot, and an extension of the Digital Cinema pilot.
– ' wider applicability and sustainability' during the final cycle.
WP4 Presentation,TenCompetence Kick-off, Dec 8th – 9th 2005
Evaluation planning (I)
• T5. Develop an overall evaluation plan (up to m6)– objectives addressed (including gender issues),– criteria and indicators:
• quality criteria• effectiveness in context• technical criteria
• components and functions to be assessed• methodologies in context of the project• the different actors and “reference users” in the process and
their tasks• the coordination and procedures, defining a timeline for
evaluation
WP4 Presentation,TenCompetence Kick-off, Dec 8th – 9th 2005
Sample evaluation grid from TA
Activities Aspect RTD activities
Technical Design & Implementation activities
Pilots & Validation activities
Pilots & Validation activities
experiments Experiments pilots demonstrators
Objective Technical verification and validation of the functional and usability aspects of the tools
Verification of the integration and the interoperability between the components
Validation of the environment integrated and of the individual components. Focus:- functionality- operationally- usability
Validation of the environment integrated and of the individual components. Focus:- sustainability
Feeds into The concerning activities
Technical Design & Implementation + Aspect RTD activities
Technical Design & Implementation + Aspect RTD activities
Requirements & Analysis activities
Period ‘continuously’ after completion of each integration loop: 3 times
after the conclusion of the experiments of the first two integration loops: 2 times.Round 1: small scaleRound 2: large scale
after the conclusion of the experiment of the final integration loop: 1 time
WP4 Presentation,TenCompetence Kick-off, Dec 8th – 9th 2005
On users/actors and methodologies
• Users / actors– learning providers, learning designers, tutors, learners ... (different
from different areas?)
• Methodologies to be used:– Ethnographic study, Interviews, Focus groups, Questionnaires,
Observations, Experiment, Peer review, Think-alouds, User diaries, System log data, Textual data, Cost analysis, Reviews, Usability inspection, ...
• Components / areas (to be defined for a good plan)
WP4 Presentation,TenCompetence Kick-off, Dec 8th – 9th 2005
Evaluation planning in detail
• T5. Develop an overall evaluation plan (continued)– more detail needed for planning and initial implementation– evaluation contact in each organisation – not only this WP
partners– define components to be evaluated; provide initial evaluation
sheet for each component (model provided)– Initial scenarios for pilots probably needed to plan for
the whole period– UPF in charge of coordination and assistance, every partner
responsible of carrying out their own evaluation, evaluation committee to supervise overall evaluation and validation
– Everybody to send a list of components, and an evaluation sheet for each component
WP4 Presentation,TenCompetence Kick-off, Dec 8th – 9th 2005
Evaluation sheet modelComponent to be evaluated Areas to be assessed
Parameters to be assessed
Actors / partners
Role of the actors
Evaluation procedure / material
Starting date
Immediate action (and date)
Intermediate action (and date)
End date
Conclusion (general / for each area)
Overall comments
Evaluation committee comments
WP4 Presentation,TenCompetence Kick-off, Dec 8th – 9th 2005
Pilot evaluation plan and implementation
• T2. Define the pilot evaluation plan, specifying the validation criteria and the strategies for evaluation (up to m6)– Define an implementation plan for the first pilot – Model for the coming plans (which have to be
initially defined)
WP4 Presentation,TenCompetence Kick-off, Dec 8th – 9th 2005
Deliverable reminder; contributions
• D4.1: Report (m 6) on:– scenarios and pilots definition– user requirements (?) and validation criteria for the overall project– evaluation strategies and initial evaluation plan implementation
• Contributions of the different partners (to be defined now for the first 6 months of the project)– OUNL ?– ALTRAN ?– LogicaCMG ?– GIUNTI ?– CERTH ?– SU ?– SYN ?– Anybody missing? Any contribution missing?
WP4 Presentation,TenCompetence Kick-off, Dec 8th – 9th 2005
Evaluation representatives (all partners)
• OUNL ?• ALTRAN ?• LogicaCMG ?• GIUNTI ?• CERTH ?• SU ?• SYN ?• L3S ?• INSEAD ?• UB ?• UvA ?• SURF ?• Evaluation committee?
WP4 Presentation,TenCompetence Kick-off, Dec 8th – 9th 2005
Reminder of coming tasks
• T3. Set up the initial pilots (m6-12)– Initial pilot on Digital Cinema; start earlier if possible to be able to
validate; otherwise, make it a phased pilot, where validation can start early
• T4. Validate the initial system through the evaluation of the Digital Cinema pilot (m12-18)– Phased work or earlier start
• T6. Generate user requirements for cycle-2 pilots (m12-18)– On the basis of early work on pilots definition
• D4.2: Report on results of first evaluation of pilots (month 18)– Suggested interim report on m12 to submit for the review
WP4 Presentation,TenCompetence Kick-off, Dec 8th – 9th 2005
Ourselves
• Dai Griffiths (WP manager) [email protected]
• Josep Blat (in charge of DC pilot) [email protected]
• Sergio Sayago (evaluation specialist) [email protected]
• and others!
WP4 Presentation,TenCompetence Kick-off, Dec 8th – 9th 2005
By way of summary
• Evaluation, validation, user requirement activities are probably everybody’s task
• Components and overall evaluation/validation strategies to be defined: contributions expected from everybody; not an easy task
• Pilot 1 (Digital Cinema) to be planned and carried out during first period
• Pilots 2, 3, 4 to be planned during m12-18; but initial scenarios important for the whole user reqs, validation, evaluation planning due by month 6. A second updated deliverable by month 12?
• And don’t forget the periphery of the project! Possible extra pilots, associates feedback ...