p erformance m easure r eporting : tpp and preis g rantees oah/acyf and rti international webinar...

1

PERFORMANCE MEASURE REPORTING:TPP AND PREIS GRANTEES

OAH/ACYF and RTI InternationalWebinarAugust 29, 2011

2

Reporting System Development Today: Overview of the system

Data to be collected Options for ways to enter or upload data

Next Steps Pilot testing and refinement of the system Training on how to use the system

3

Today’s Topics

Review of data collection and reporting Roles and responsibilities of RTI and

Grantees Introduction to the Web Portal System Tracking data using spread sheets or web Uploading data Reporting on participant-level measures of

perceived impact  Reporting on participant-level measures of

actual impact

4

Data Collection and Reporting Schedule

Instrument Data Collection Time Point Reporting

Enrollment Baseline ImplementationPost Program

Later Follow up Semi-Yearly Yearly

Demographics - Student Reach ALL ALLParticipant Level Impact Questionnaire CD TIER 2 CD TIER 2 CD TIER 2 ALL

Perceived Program ImpactQuestionnaire ALL ALL

Student Attendance-Dosage ALL ALL

Other Attendance-Reach and Dosage ALL ALL

Adherence ALL ALL

Fidelity Observation ALL ALL

Partners ALL

Training ALL

Dissemination ALL

Fidelity Process ALL

5

Roles and Responsibilities

Developing and maintaining website (e.g. assigning IDs, troubleshooting)

Training and TA Analyzing data

provided Reporting performance

measures to OAH

Collecting data Setting standards and

tracking implementation Organizing and labeling

data following guidance from RTI

Ensuring quality and security of data

Reporting accurate data on time

RTI Grantees/Evaluators

6

Web Portal Overview

A secure website hosted by RTI for reporting performance measures

Each grantee will have its own page Grantees will receive credentials to access the website Website allows for 3 different modes of data entry

Raw data entry (performance measures computed by the system)

Uploading of data files Direct entry of computed performance measures – special

conditions. Provides dynamic reports tailored to your

specifications

7

Security and Confidentiality of Data

Data stored in a SQL Server database

Database accessible only to analysts assigned to the project

Electronic communications occur via a secure Internet connection

128-bit encryption of transmissions through secure socket layers (SSL)

Reports will present data in aggregate form

RTI will assign credentials to grantees

Individuals granted access to the website will have IDs and passwords

IDs and passwords are not to be shared

No personal identifiers may be sent to RTI for participant level data – youth or parents Numeric IDs No social security numbers

RTI Grantees

8

Web Portal System Home Page

9

Data Reporting Overview

All grantees need to be sure that the program(s) (e.g., BART, Reducing the Risk) you are implementing are the correct ones listed on the system

All grantees need to identify the classes or sections in which the program is being implemented – can upload

Grantees that will use the system for tracking dosage will need to identify the participants on the system

Grantees can track dosage, fidelity, and reach using spreadsheets or other systems

10

Web Portal System Home Page Menu

11

Grantee Data: Adding New Programs

12

Adding Program Sections/Classes

13

Adding Participants

14

Demographic Data

Demographic information needs to be collected when you enroll participants in case they Are not scheduled to take a baseline

measure Miss taking the scheduled baseline

measure Drop out before taking the post-test

Use the web system or use forms that you develop

If you have questions specific to your unique situation, please contact your project officer

15

Entering Demographics on the Web

Participant ID: --------------------------------Gender: o Male

Femaleo Other

Age: o <= 10 yearso 11-12 13-14 o 15-16 o 17-18 o >= 19

Grade: o <= 6 grade 7-8o 9-10o 11-12o GEDo Technical/vocationalo Collegeo Not currently in school

Ethnicity: o Hispanic/Latino Not Hispanic/Latinoo Unknown/not reported

16

Enrolling Participants in Class

17

Adding Class Sessions

18

Uploading Data

19

Adding Demographics by Excel

Participant Gender Age Grade Ethnicity Race

Language

Special Population

1010234 2 4 3 2 3 1 6

ID only 1=male 1 - <= 10 years

1 - <= 6 grade 1 - Hispanic/Latino

1 - American Indian/Alaskan Native 1 - English

1 - Pregnant or parenting teens

2=female 2 - 11-12 2 - 7-8 2 - Not Hispanic/Latino 2 - Asian 2 - Spanish

2 - Youth in Foster Care

3=other 3 - 13-14 3 - 9-10 3 - Unknown/not reported

3 - Black/ African American 3 - Chinese 3 - Homeless youth

4 - 15-16 4 - 11-12

4 - Native Hawaiian/other Pacific Islander 4 - other

4 - Youth in Juvenile Justice system

5 - 17-18 5 - GED 5 - White 5 - other

6 - >= 19

6 - Technical/ vocational 6 - Other 6- none

7 - College 7 - More than one race

8 - Not currently in school

8 - Unknown/unreported

20

Adding a Class/Section using Excel

SectionName Facilitator Location TimeDay

Section01 John Smith Wilson High Mondays @ 2pm

21

Enrolling a Participant in a Class/Section using Excel

Participant

1010234

22

Deriving Performance Measures After basic data are in the system it will

be possible to calculate performance measures Reach Dosage Fidelity

Dosage and Fidelity will require additional data that can be included either using web entry or excel

23

Reach

# of youth served in program Defined as participating in at least one activity Data will be reported by demographic

characteristics obtained at enrollment # of other clients served – parents, sibs

Defined as participating in at least one activity Data will be reported as numbers

Will need attendance data for RTI to compute

24

Dosage

Data obtained from Attendance Logs Youth/other client attendance clients every session

Tracking options RTI will have a web entry system for entering

individual or summary data Grantees can tally data on a spreadsheet and upload

Measures to be computed Median percent of total intended program services

received by youth/other participants % of participants who received at least 75% of the

program

25

Recording Attendance on the Web

Add a SessionIf you have your own system for maintaining session information, please click here to upload data. Otherwise, please use this form to add session information.

Session name/number: ______________Session date: ____________Session time: ______________Number of Activities Scheduled: ______________Number of Activities Completed: ______________

Session participants (check all who attended) Participant 1Participant 2Participant 3Participant 4Participant 5Participant 6NOTE: Prepopulated from enrollees to classSave and add anotherSave and finish   

26

Recording Attendance in Excel

SessionName Participant Attended

Session01 1010234 1

1010235 0

1010236 1

Session02 1010234 1

1010235 1

1010236 1

1= Attended 0 = Didn’t Attend

27

Fidelity – Facilitator Measured Adherence to Program Core Components

Defined as the percentage of prescribed activities that are completed

Completed for all sessions with youth and other clients

Activities tracked on Fidelity Monitoring Logs Grantees can enter directly on web Grantees can tally data on a spreadsheet and upload

Measures to be Computed Median % of activities completed of those scheduled Median % of sessions implemented Median % of sessions with a fidelity log

28

Recording Adherence on the Web

Note: Need to assess whether activities were implemented with minor or major changes prior to entering

29

Recording Adherence using Excel

SessionName SessionDate ActivitiesScheduled ActivitiesCompletedSession01 10/1/21 8 8

Session02 10/8/21 10 9

30

Other Information Needed for Fidelity

Total number of prescribed sessions for each program – grantee to determine

For programs where a variable number of sessions can be offered, the prescribed sessions for each section – grantee to determine

The total number of sessions delivered – grantee to determine

31

Recording Sessions Implemented

SectionName

Session01

Session02

Session03

Session04

Session05

Section01 1 1 1 1 1

Section02 1 1 1 0 0

Section03 1 1 0 1 1

Section04 1 0 1 0 1

Section05 1 0 1 1 1

Grantee responsible for setting a target for the total number of prescribed sessions to be delivered and knowing which ones are delivered 1=Implemented 0 = Not Implemented

32

Recording Sessions Implemented with Logs

SectionName

Session01

Session02

Session03

Session04

Session05

Section01 1 1 1 1 1

Section02 1 1 1 0 0

Section03 1 1 0 1 1

Section04 1 0 1 0 1

Section05 1 0 1 1 1

Grantee responsible for knowing which sessions are delivered1= Log Received 0 = Session Delivered but No Log

33

Fidelity Observation

For Adherence (% activities completed) Use the Fidelity Monitoring Logs (i.e., same

fidelity monitoring tool as the facilitator) Enter on web or on excel spread sheets just

as you will enter for facilitator measured fidelity

For Program Quality Use the Observation Form for TPP Grantees Enter on web or excel spread sheets and

upload to web

34

Fidelity Observation Measures Adherence Performance Measures

Total # sessions implemented Total # sessions observed by an

independent observer Median percentage of activities completed

across sessions that were observed Minimum and maximum percentages of

activities completed across sessions observed

All can be calculated from data entered

35

Recording Quality in Excel

Percentage of observed sessions rated either 4 or higher for overall quality

SectionName

SessionName

Qual1

Qual2

Qual3

Qual4

Qual5

Qual6a

Qual6b

Qual6c

Qual6d

Qual6e

Qual7

Section01

Session04

4 5 3 3 4 5 4 4 5 3 4

36

Grantee Summary Measures – All Grantees

Dissemination# manuscripts accepted/published total # titles of manuscripts# manuscripts submitted total # titles of manuscripts# Presentations - National Total # titles and venues national presentations# Presentations - Regional Total # titles and venues regional presentations# Presentations - State Total # titles and venues state presentations

# Partners# organizations/schools with formal agreements Total ## organizations/schools assist with implementation Total ## organizations/schools actively involved but no formal agreement Total #

Partner Retention# organizations/schools with a formal agreement at start of program year still engaged Total #

Training of Facilitators# new facilitators/teachers Trained Total ## facilitators/teachers receive follow up training Total #

37

Grantee Summary Measures – Tier 2/PREIS

Completed and approved pieces of program necessary to package for replication:

Program Component ApprovalDate

LogicModel

CoreComponents

FideltyTools

CurriculumManual

Facilitator Manual

TrainingMaterials

AdaptationGuidance

38

Grantee Summary Measures (cont.)

Fidelity Process Checklist Reported once a year via web based data entry or

uploading excel spread sheet

Process1

Process2

Process3

Process4

Process5

Process6

Process7

Process8

Process9

Process10

Process11

Process13

2 2 2 1 2 1 1 2 0 1 2 1

0= Not at all 1 = Somewhat 2 = Completely

39

Perceived Impact Measures

Data can be entered in 2 ways: Enter data directly into the web-based reporting

system Create subsets of larger datasets that contain only the

variables required for performance measures, and upload (Excel, CSV)

RTI will provide standard variable labels and formats

40

Entering Perceived Impact on the Web

41

Actual Impact Measures

For grantees with rigorous evaluations, evaluators will collect additional participant-level data (actual impact measures) as part of their scheduled data collections

Evaluators will upload data only for those variables specified in the performance measure documentation

Data reported annually—any data collected that year

42

Uploading Participant-Level Data

43

Important Dates

All data must be uploaded no later than November 30 and May 31 of each year

Same dates as your progress reports All grantees who enter summarized data,

must send their detailed data to RTI for verification and must provide documentation of how variable track to RTI variables

Those who enter summarized data must send their progress officer a detailed description of the system for approval

44

Next Steps for RTI

Wrap up programming – target completion date 9/2 Pilot test web reporting system – target completion

date 9/9 Send email to grantees to request information

regarding users of the system (name, role, email) and after receipt to create accounts – target completion date 9/9

Virtual tour via a WebEx for the users who will be on the system and distribute the credentials - 9/12 -9/14

Launch reporting system!!NOTE: ALL DATES ARE APPROXIMATE

45

Questions

Data and reporting questions – RTI Special circumstances regarding

collection – your project officer