Outcome-Based Management Program

Data Collection Requirements

- 2 -- 2 -

Data Collection Background

The Outcome-Based Management (OBM) program will require improved and consistent data collection requirements to ensure that quality data is entered on a timely basis

Why are we updating the data collection requirements for clubs?

The OBM program will require more numerical data to understand the clubs’ impact on our members

Better data collection may streamline work associated with program reporting (i.e., TEEN Reach, monthly reports) and grant writing

Which data collection processes have updated requirements?

Club Attendance Tracking

Selected Program Attendance Tracking

Outcome Measurement Toolkit Administration

Selected Survey and Test Administration

Who is responsible for meeting these updated requirements?

All staff will be held accountable for meeting these updated requirements through regular service center oversight and the annual review process

You may be meeting some of these requirements, but we need to meet them all to be successful!

- 3 -- 3 -

Data Collection Requirements

The updated data collection requirements encourage consistent data acquisition across clubs

Data Collection Process

Process Description Usage Updated Requirements

Club Attendance Tracking

Tracking member attendance at clubs, including entry and exit times

Used with other metrics to see how club attendance impacts members

Every member must check in and out at front desk to participate in club activities

Club staff must scan members into KidTrax on a daily basis Club staff must ensure that member entry and exit times

are properly inputted into KidTrax

– Batch scanning only should be used in limited circumstances

Selected Program Attendance Tracking

Tracking member attendance at selected programs

Used with other metrics to see how program attendance impacts members

Every child must check in and out with the Program Instructor to participate in selected programs and activities

Club staff must enter program attendance within two days of program session

Batch scanning is acceptable for data entry

Outcome Measurement Toolkit Administration

Administering survey tool developed by BGCA on a regular basis

Used with club and program attendance to analyze changes in attitudes and behaviors

Toolkit must be completed by the same members 3x per year during time period in October, January, and May

A set number of members must use toolkit over this period All answers must be inputted into KidTrax

Selected Survey and Test Administration

Administering pre / post surveys and tests developed by BGCA / clubs on a regular basis

Used with program attendance to analyze changes in attitudes, behaviors, and knowledge

Selected pre / post surveys and tests must be administered to all members 2x per program at the beginning and end of program

Surveys / tests must be administered each time the program is run

All answers must be inputted into KidTraxThese requirements are important because operations will only review data entered in KidTrax

- 4 -- 4 -

Enforcement and Staff Impact

To make sure we meet these new requirements, the service center will monitor clubs’ data on a regular basis and take appropriate actions to ensure data is entered accurately and timely

• Reviewed by service center on an on-going basis

• Review will ensure data entered within two days of program session

• Reviewed by service center after toolkit administration period

• Review will ensure that clubs are capturing appropriate % of members

Selected Program Attendance Tracking

Outcome Measurement Toolkit Administration

Club Attendance Tracking

• Reviewed by service center on an on-going basis

• Review will ensure:– Data entered on a daily basis– Clubs inputting proper entry and exit

times (i.e., not batching)

• These requirements are expected to be met by all staff members and will be a part of the performance review process

• Clubs that are not in compliance will have a letter sent to the Club Director and Board President

• Staff disciplinary action will be taken for non-compliance in accordance to existing policies and practices

Selected Survey and Test Administration

• Reviewed by service center on a quarterly basis

• Review will ensure that survey and tests are being administered to all program participants

Enforcement Staff ImpactData Collection Process

- 5 -- 5 -

Attendance Tracking Process and Details

- 6 -- 6 -

Club Attendance Tracking (Stand-Alone Clubs) – Suggested Process

Enter /

Exit

Club

Member has ID Card

Member does not have ID

Card

Scan ID card

Option 1: Retrieve backup ID card, scan in

Scanner Available

Scanner Not Available

Scanner Available

Scanner Not Available

Option 2: Manual KidTrax scan

Manual KidTrax scan

Responsibility: Front Door Volunteer / Assigned Club Member / Staff Member

Frequency: Daily

Additional Data Entry Time (Staff): N/A

- 7 -- 7 -

Club Attendance Tracking (School-Based Clubs) – Suggested Process

Enter /

Exit

Club

Member has ID Card

Member does not have ID

Card

Scan ID card

Option 1: Retrieve backup ID card, scan in

Scanner Available

Scanner Not Available

Scanner Available

Scanner Not Available

Option 3: Front door resource enters name and entry / exit time on sign-in

sheet

Front door resource enters name and entry /

exit time on sign-in sheet

Front door resource enters data into

KidTrax w/in one day

Front door resource enters data into

KidTrax w/in one day

Responsibility: Front Door Volunteer / Assigned Club Member / Staff Member

Frequency: Daily

Additional Data Entry Time (Staff): ~10-20 min per day (depends on # of members that attend)

Option 2: Manual KidTrax scan

- 8 -- 8 -

Club Attendance Tracking – Additional Details

Topic Details

Backup / Manual Scans

If a member forgets his or her card and there is immediate access to Trax Attendance Manager, the front door resource should either:

– Scan the member in using a backup ID card

– Perform a manual scan (see Manual Scan instruction guide for further details)

Sign-in Sheets If 1) a member forgets his or her card and there is no immediate access to Trax Attendance Manager, 2) a club does not have an ID scanner, or 3) a large number of members enter the club at one time, the front door resource must record the member’s name and entry / exit times on a sign-in sheet then input that information into KidTrax within a day

All clubs should continue to use sign-in sheets for backup purposes (i.e. KidTrax down)

Data Entry Batching of sign-in sheet information is not permitted at stand-alone clubs, unless emergency situation

– Clubs need to record exact entry and exit times by using ID scan or manual scan Batching of sign-in sheet information is permitted at school-based clubs for members who remain at the site

the entire time the club is open

– Members who enter 30 minutes after the club opens or leave 30 minutes before the club closes must have their times recorded on the sign-in sheet and be entered into the system separately

If KidTrax is down, all clubs are allowed to batch scan entry and exit information if they have followed the existing protocol (e.g., notified the general service center)

Best Practices A volunteer, club member, or staff member should be assigned attendance data entry responsibilities

– Please identify a person who is trained / familiar w/ KidTrax and able to assist others in entering data

Many clubs have created various binders to have backup scanning information for their members

– Clubs can print all member names and barcodes, then keep them at the front desk for scanning

– Clubs can use clear business card pages to store all backup ID cards Need to be diligent about tracking exit times (e.g., one person should be responsible for monitoring when

members exit)

Expectations and Enforcement

Clubs must enter data within a day of attendance and include entry and exit times Data from selected days will be reviewed by service center on an on-going basis This review will look at when and how the data was entered (i.e., were the majority of members were batched)

- 9 -- 9 -

Selected Program Attendance Tracking – Suggested Process

Enter

Program

Member puts check next to name on attendance sheet

Program Instructor collects sheets and

enters data into KidTrax

Responsibility: Program Instructor*

Frequency: Up to 2 days after program session

Additional Data Entry Time (Staff): ~5-15 min per program session

Exit Program

Program Execution

*This responsibility will be included in Program Instructor goals / performance reviews

Program Instructor provides pre-populated

program attendance sheet to members

- 10 -- 10 -

Selected Program Attendance Tracking – Additional Details

Topic Details

Attendance Sheets Attendance sheets (part of the program planning kit) should be used for each program session and include every member’s name who has attended the program and rows for write-in members

– These attendance sheets should be used for multiple sessions and stored by the Program Director Program Instructor is responsible for ensuring all members are accounted for on the attendance sheets These attendance sheets should be stored for the entire school year

Scanners Handheld scanners will not be available for program attendance tracking at this time

Data Entry The Program Instructor is responsible for data entry within two days of the program session

– Directors / Managers are responsible for providing appropriate KidTrax training to Program Instructors

Batching of sign-in sheet information is permitted for attendance information

– See Batch Attendance instruction guide for further details

Best Practices Program Instructors should allot an extra 5-15 minutes per program session to input data Having a dedicated computer terminal makes inputting data quicker and easier Work w/ the Club Director / Program Director / Club Manager / Membership Clerks to ensure data is batched

and entered correctly

Expectations and Enforcement

Coordinators must enter data within two days of the program session Data from selected program dates will be reviewed by service center on an on-going basis This review will see if the majority of members were entered within the proper timeframe

Participating Programs

On-Going Programs SMART Programs Power Hour College Bound Keystone Club Torch Club

Quarterly Programs SMART Moves SMART Girls Street SMART Passport to Manhood Money Matters

Career Launch Youth for Unity Triple Play Skill Tech / Digital Suite

- 11 -- 11 -

Survey and Test Administration Guidelines

- 12 -- 12 -

Outcome Measurement Toolkit – Guidelines

Responsibility: Club Director / Program Director / Club Manager / TEEN Reach Coordinator

Frequency: 3 times per year (allot ~2 week; must be completed by October 24, January 18, and May 16)

Additional Data Entry Time (Staff): ~5 mins per survey when toolkit completed on paper

Topic Details

Toolkit Administration

The Outcome Measurement Toolkit is now be available in KidTrax and needs to be completed regularly In order to measure the impact on our members, clubs are required to test the same members during at least 2

testing periods per year Members must complete the appropriate survey toolkit for their grade level (i.e., elementary school, middle

school, or high school) Staff should administer the toolkit online for every level except Cadets, who should complete it by paper

– Clubs may administer the toolkit by paper for other groups if there are resource constraints or youth are not able to use computers effectively

Toolkits administered on paper must be entered by staff members into KidTrax within the designated time period

Timeline The toolkit must be administered 3 times per year

– Clubs may allot extra time to input the toolkits, but must be completed by October 24, January 18, and May 16

– Suggested administration dates: October 13-24, January 7-18, and May 5-16 Scheduling of the computer lab, staff assignments and members is the Director’s / Manager’s responsibility Regular programming should continue during this time for all other areas

TEEN Reach Participants

TEEN Reach participants are required to use the toolkit once per year, but do not need to participate during every administration period

– It is recommended that coordinators administer the toolkit during the January session

– All participants must use the toolkit found in KidTrax and not the BGCA website A separate section of the toolkit containing sensitive questions must be administered by TEEN Reach

coordinators to program participants

– This section will be completely anonymous, but must still be entered into KidTrax

- 13 -- 13 -

Outcome Measurement Toolkit – Guidelines, Cont’d

Topic Details

Best Practices One staff member should be in charge of all computer room scheduling and works in conjunction with computer room manager to administer testing

Clubs should provide open programming to members who complete toolkit early Incentives should be given for member participation (e.g., pizza party, field trip, Power Points) and clubs are

also encouraged to sponsor raffle prizes drawn from participants (i.e., items attractive to members and their families)

See Survey Best Practices slide for additional suggestions

Expectations and Enforcement

Clubs are expected to survey 70% of their projected ADA goals for that year (i.e., unduplicated members)

– See OMTK Goals worksheet for target goals Service center will review the participation rates after the testing period and report back to the clubs

Other Notes Toolkit administration will be included in the goals and performance reviews of the Club Director / Program Director / Club Manager

Aggregate results will be available in real-time for program improvement and staff discussions

– Plan to spend at least two staff meetings to review results of the toolkit

Responsibility: Club Director / Program Director / Club Manager / TEEN Reach Coordinator

Frequency: 3 times per year (allot ~2 week; must be completed by October 24, January 18, and May 16)

Additional Data Entry Time (Staff): ~5 mins per survey when toolkit completed on paper

- 14 -- 14 -

Selected Survey and Test Administration – Guidelines

Topic Details

Survey / Test Administration

Pre / post surveys and tests for selected programs are available on KidTrax and must be completed by all participating members

Program Instructors may administer tests either electronically or by paper, but they are responsible for putting the information into KidTrax

– Surveys and tests must be linked back to an individual member for tracking purposes

Timeline Surveys and tests must be given 2 times per program (during first and last day of instruction) Allot 1-2 days to complete testing of members and data entry by staff

Best Practices Incentives may be given for member participation (e.g., pizza party, field trip, Power Points) See Survey Best Practices slide for additional suggestions

Expectations and Enforcement

Clubs are expected to survey all program attendees on the first and last day of instruction Service center will review the participation rates on a quarterly basis

Participating Programs

SMART Moves SMART Girls Street SMART

Other Notes Survey administration will be included in goals and performance reviews of the Program Instructor Results will be available in real-time for program improvement and staff discussions These requirements do not change any guidelines for other programs (e.g., 21st Century)

Responsibility: Program Instructor

Frequency: 2 times per program (during first and last day of instruction)

Additional Data Entry Time (Staff): ~5 mins per survey when survey or tests completed on paper

- 15 -- 15 -

Survey Best Practices

There are a number of best practices you can use for administering surveys and tests to members

Prepare all materials prior to the scheduled survey. If using hard copies, keep in mind kids often respond better to surveys printed on colored paper

Leave enough time to set up and arrange the room appropriately prior to the members’ arrival

Introduce yourself to the members as they arrive and give a brief description of your purpose for being there and administering the survey. Sample script:

– I am here to ask you to help make our clubs better. The survey your are about to take is being used to get your ideas about our club. It is not a test and there are no right or wrong answers — we want to know what your experiences and opinions are. Take your time and be sure to answer each question based on what you really think. If you don’t understand a question, please raise your hand or come find me and I can help you. The survey should take you only 8 to 10 minutes. (May differ depending on survey length).”

Request that members attempt to answer every question so their opinions and experiences can be included

Let members know you are here to help them if they have any questions

While members are completing the survey, quietly walk around the room (do not hover), answer questions and make sure things are running smoothly

When youth finish the survey, thank them for their time and provide incentives, if appropriate

Source: The Colorado Trust, ” Toolkit for Evaluating Positive Youth Development”