1

Regents ScanningCFN Training Sessions

Updated April 28, 2011

22

ObjectivesThe purpose of today’s session is to EDUCATE you on the new Regents scanning procedures, and EMPOWER you to turn-key the process at the school level.

After the training, you will understand:

> Why the DOE is implementing the new Regents scanning process and the benefits of the changes;

> The overall scanning process to be performed by schools, and step-by-step procedures for each stage;

> How schools should plan for Regents exam week, and special situations to prepare for;

> What documents and resources are available for training and support, including how to use the demo IT environment to train schools; and

> Best practices for turn-keying the scanning process at schools.

33

Agenda> New SED Regents requirement & DOE scanning solution> Regents scanning A–Z

Process overview and timeline Step-by-step review Live demonstration

> Turn-keying to schools Lessons learned from the pilot program Demo testing environment Group activity: case study and discussion

> NY SED Special Study – June 2011 Regents> Ongoing support and resources> Q & A

44

SED Regents Requirement & DOE SolutionSED Requirement: Beginning in June 2011, the SED will require all districts to collect and submit item-level data for the following Regents exams:*

DOE Solution: Leverage existing technology and systems currently used by schools to capture and report item-level data. New scanning process highlights include:

> Answer documents: student-specific documents will now be generated and printed at school level

> Scoring: teachers mark Constructive Response items only> Data capture: scan answer docs with attendance scanners> Reporting results: real-time results available in ATS immediately

after scanning; automated data transfer from ATS to STARS*Other Regents exams & RCTs will use the traditional administration and scoring process.

55

SED Regents Requirement & DOE Solution

> Time savings through automation of manual processes Less overall time required for scoring No further data entry requirements for aggregate scores

> Increased accuracy in reporting test results Elimination of manual scoring of MC questions Section scores totaled automatically Automatic application of conversion tables

> Improved instruction Item-level data available for use in informing instruction at

student, teacher, school, and district levels> Audit trail

Answer document image-capture and line-item results will aid in special inquiries

Benefits to schools under the new process:

Pre-Planning

STARSATS &

Scanner

ATSSTARS

(& Beyond)

1. Schools receive training and begin preparations for Regents Scanning process.

2. Schedule students for Regents and produce exam invitations in STARS.

3. STARS data flows to ATS. Schools test scanner and generate and print pre-slugged answer docs in advance.

6. Schools scan answer docs for upload to ATS.

7. ATS instantly applies scoring key & conversion chart for scale score. Results available immediately.

8. ATS passes data nightly to STARS for report cards and transcripts, and for reporting to NYSED.

Exam Week & Scoring

5. Schools administer exams. Answer documents for walk-ins are printed on-demand. Scoring committee marks Constructive Response items and bubbles in scores on answer documents.

Attendance Scanner

Re-Scanning (As Needed)

Regents Scanning A-Z: Process Overview

6

Logistics Planning

Manual Score Verification

4. Schools conduct Regents week planning activities for logistics, including assignment of roles, and performing checklist items in the week before exams are administered.

7A. Score verification6A. Re-scan docs.

Overnight Data Transfer

Overnight Data Transfer

77

Timeline of Key StepsDates Key Step

ASAPALs begin scheduling sessions to train schools on the new process

May 1 – 31Relevant personnel at schools assign roles, conduct planning, review materials, test scanner, etc.

May 1 – 31Schools clean and test scanner and ensure adequate supplies (toner, paper, etc.)

By May 27 Schools register majority of students for Regents in STARS

June 1–10 Schools print answer documents in ATS for most students

June 8–14 Regents week logistics planning & final preparation

June 15–24 Regents administration

June 15–24 Scanning and data validation

88

Step 1: STARS> Main changes from the traditional process:

No changes – the process is exactly the same.> Main tasks:

Register students for Regents exam, and generate exam invitations.> Task owner: typically school Program Chair> Best practices:

Complete registration process for majority of student by May 27. Middle Schools: use proper exam codes in STARS for 8th graders.

If appropriate, notify the high school where students will take exam. Only register students expected to take exams (though when in

doubt, err in favor of registration). Use “Master Schedule” function for creating a single section, and

“Master Schedule Builder” for multiple sections.> Supporting materials: Regents Scanning Handbook, School Progress

Reports available to Assessment Liaisons> Support: CFN Data & IT Liaison

99

Steps 2–3: Printing Answer Docs in ATS> Main changes from the traditional process:

Answer documents are no longer included in the Regents exam booklet to be torn out.

Pre-slugged student-specific answer documents are printed in advance, or generated on-demand at school level.

No generic answer documents may be used. Answer docs are 1 page for Algebra & English; 2 for all others.

> Main tasks: Test scanner to ensure readiness, adequate supplies of toner

and paper. Print answer docs (for pre-registered students).

> Task owner: typically Pupil Accounting Secretary> Supporting materials: Regents Scanning Handbook, School

Progress Reports> Support: DIIT Help Desk, 718-935-5100

1010

Traditional & New Answer DocumentsTraditional Process Regents Scanning

Individualized bar code

Student MC answers

CR teachers’ grades

Alt. language & accommodations

Pre-populated student info

Scorers’info

1111

Steps 2–3: Printing Answer Docs in ATS> There are five options for printing answer docs:

1. “Not previously printed” (Batch printing for registered students)

Prints batch of answer documents that have not been printed

2. “Print by student ID” (For DOE walk-in student)Prints an answer sheet for any student one at a time. This option

may be used for another DOE student taking the exam at your school. (Use ATS SBIO screen to look up student ID.)

3. “Print by name (non-pub)” (For non-DOE walk-in student)Prints an answer sheet for a non-DOE student who is taking the

exam at your school.

4. “Charter school option” (For schools not using STARS)Batch printing option for schools who are unable to register students in STARS.

5. “Reprint answer documents”Reprint previously printed docs by student ID or answer document page number.

1212

Steps 2–3: Printing Answer Docs in ATS

Sorting Options

> Use RGPR screen for printing options 1, 2, and 3.

1313

Steps 2–3: Printing Answer Docs in ATS> Best practices for printing answer documents

Answer documents must be printed on white paper only. Use the same stock paper as is normally used for attendance forms. Using colored paper or heavy stock may result in scanning errors.

Print answer docs for pre-registered students by June 10. After printing batches of answer documents, a “test page”

will print. Scan the test page to confirm ink quality for batch. When printing, you may sort print jobs by teacher, section, or

room number. Ensure that adequate supplies of paper and toner are on-

hand.

1414

Step 4: Logistics Planning for Regents Week> Main changes from the traditional process:

Educate relevant staff members on new procedures and responsibilities for Regents scanning, and organize staff roles.

> Main tasks: Distribute test rosters and page one of answer docs to appropriate staff in

advance. Train exam proctors on new procedures and policies. Ensure adequate supply of toner, paper, pens/pencils. Establish process and task owners for dealing with walk-ins. (See slide

“Instructions for Walk-Ins”) Review Implementation Checklist

> Best practices: Roles and responsibilities are clear and communicated to all relevant staff

before Regents Week. Perform all checklist items.

> Supporting materials: Regents Scanning – Implementation Checklist> Support: CFN Assessment Liaison

1515

Step 5: Exam Week & Scoring> Main changes from the traditional process:

Answer docs for “walk-ins” printed on-demand on test day. Students use pre-slugged answer documents; pencil for MC,

pen for essays and DBQ questions. Teachers mark Constructive Response items only using pencil.

> Main tasks: Generate answer documents for “walk-in” test-takers. (See next

slide.) Proctors administer exams according to best practices and

emphasize new procedures (e.g. completely fill in bubbles, no stray marks, sign student declaration when finished).

> Best practices: Good preparation will lead to smooth exam administration

> Support: CFN Assessment Liaison

1616

Step 5: Instructions for “Walk-Ins”Schools should make preparations and assign roles to handle walk-in students during Regents week.

Step DOE Student Non-DOE Student

Confirm student’s ID Check photo ID Check photo ID

School notification Check student record against notification sent from her home

school

Must present signed letter on school letterhead with exam

specified

Print answer doc in ATS “Print by student ID”* so that data flows to STARS

“Print by name” for non-pubwalk-in student

Enter exam code Use appropriate exam code Use appropriate exam code

*If Student ID is unknown, use ATS SBIO menu to look it up.

> Inform appropriate staff how to direct walk-ins upon arrival, and appoint staff to print answer docs using ATS.

> Print answer documents by following the below instructions:

> Detailed instructions on handling walk-ins is included in the Handbook and FAQ.

1717

Step 6 & 6A: Scanning Answer Documents> Main changes from the traditional process:

Scanning will capture item-level data and images of answer documents (and any ancillary re-scans) for audit trail.

Immediate results available in ATS; conversion chart automatically applied for instant score.

> Main tasks: After notification by DOE, scan docs using attendance scanner. Docs with errors in reading bar codes will be reprinted with an error

message. Docs with bubble read errors will be indicated on DFS confirmation sheet. Re-scan affected documents.

Review ATS reports for “Warnings” observed in Teacher section.> Task owner: staff members who process attendance scanning> Best practices:

Scanning tips available in Regents Scanning Handbook Ensure that scanner has been cleaned in advance.

> Support: DIIT Help Desk, 718-935-5100

1818

Step 6 & 6A: Scanning Tips> Ensure the scanner is clean; a dirty scanner may result in inaccurate

results. See Tips for Scanning appendix for instructions on cleaning.> When scanning 2-page documents, it is recommended to batch all

page 1 documents and scan separately from page 2 documents. > Schools may scan the first page of two-page documents immediately,

before scoring is complete for page two.> If only one page of a 2-page document is scanned, it will be shown in

ATS as an “Incomplete Scan” until the second page is processed.> Use the ATS Status Report to review warnings.> After scanning, a confirmation page will print indicating pages with

scanning errors. The scanner will reprint pages for which the bar code could not be read.

> File the scanned batches together with the DFS confirmation page. This will assist in retrieving papers as needed in the future.

> Daily cutoff for overnight data transfer to STARS is 9pm on weekdays and 4pm on weekends.

1919

Step 7: ATS Reports & Results> ATS reports available immediately after scanning:

Use Document Status Report to ensure that all answer docs have been properly scanned, and to flag warnings.

Use Item Report to see students’ scores and inform promotion status of individual students.• Review report for “warnings/multiples/omits” by using “Include only

W/M/O” option. Use Item Distribution Report to inform decisions on present

and future instruction.> Detailed instructions on how to access the ATS reports are

available in the Regents Scanning Handbook.> Given the length and format of the ATS reports, they are best

viewed when printed. You may not see all the data in the ATS screen.

> Schools may download the ATS reports into Excel for further analysis.

2020

Step 7: ATS Reports (REDS - Status Report)> Report uses:

Displays score, accommodations, warnings, for each student. Shows summary of documents scanned, outstanding, and warnings. Use report to track outstanding unscanned and incompletely scanned

document.

Accom.Scores Scan-specific info

Document statuses

2121

Step 7: ATS Reports (REDS – Item Report)> Report uses:

Displays line-item data for each student. Shows specific questions to review (warnings). Use to flag Teacher section questions to investigate (e.g. omits, multiples).

TEACH SECSTUDENT SECTION

Correct answers

Multiple answersIncorrect answers CR raw scores

2222

Step 7: ATS Reports (RADI – Item Dist. Report)> Report uses:

Shows school-level aggregated data for each question. Use data to flag specific subject matter, question types, etc. for instruction.

‘*’ denotes correct answer

2323

Step 7A: Re-Scoring & Score Verification> Main tasks:

AS OF MAY 24, 2011 – THERE IS NO RESCORING OF ANY REGENTS EXAM INCLUDING ALL MATH AND SCIENCE EXAMS with scaled scores of 60-64.

Verification of scanned results for approximately 5% of answer docs.

Complete Manual Scoring Data Validation Form. Consult the FAQ as Regents week approaches for updates on

scoring procedures.> Best practices:

Two people at school level perform data verification> Supporting materials: Regents Scanning Handbook> Support: CFN Assessment Liaison

2424

Step 8: Accessing Results in STARS> Main changes from the traditional process:

Data flows to STARS automatically; no manual data entry required.

Results will be automatically applied to transcripts and report cards from STARS.

> Main tasks: Provided scanning is completed by 9pm – on day after

answer documents are scanned, schools may generate a STARS reports for all registered students who sat for Regents, regardless of test location.

> Best practices: Any future score changes (through re-scanning, DOE answer

key alteration, etc.) will flow through to STARS automatically.> Support: CFN Data & IT Liaison

Live Demonstration

2626

Turn-Keying: Demo Environment

> The demo environment is available from May 2-31. After May 31, the demo period will end and all demo data will be removed.

> Detailed instructions on how to access and use the demo environment are available through a Demo Script. You will use your existing ATS username and password.

> Demo scope:

> We highly encourage schools to take advantage of the demo environment to gain familiarity with the new processes and to help ensure smooth Regents administration.

A demo environment is available for training purposes. Schools are encouraged to use this to prepare for Regents Week and gain an understanding of the required processes.

DEMO

STARS(Scheduling)

ATS(Printing)

ATS(Scanning & Reporting)

STARS(Report Card &

Transcripts)

2727

Turn-Keying: Lessons Learned from Pilot

> Schools with advanced preparation performed well during January Regents Preparations include: communication of new processes and

policies, defined staff roles, on-time scheduling/printing of answer documents, prep for “walk-in” students, etc.

> Many schools needed additional support and training around the technical systems. Many schools did not utilize ATS reports (Document Status Report,

Item Report, Item Distribution Report) for scanning validation and instructional benefits. The demo environment is intended to assist with this for the June implementation.

> Scanning technology was validated through the pilot.

DOE conducted a pilot program in January to administer the new process at select schools. The most valuable lessons learned from the pilot are:

2828

Turn-Keying: Case Study & Discussion> Review of Regents Scanning Implementation Checklist.> Case Study exercise.> Discussion: in groups of 3-5, discuss the following questions:

How do you plan to effectively turn-key this new initiative to schools?

Who will you include in the school-level training? What challenges do you anticipate? What items need to be emphasized or modified for different

size schools within your network? How will you support your schools in using the new item-

level data that is produced from this process? What questions do you have about the process itself or its

implementation? (Record for Q&A)

2929

Turn-Keying: Roles & ResponsibilitiesAssessment Liaisons Data IT Liaisons

Overall point of contact for schools for Regents Scanning initiative

Ensure schools and network staff have appropriate access levels in ATS and STARS

Responsible for coordinating and delivering training to relevant school staff within network

Support schools with questions on printing answer documents. Accompany AL during school questions.

Ensure schools are properly equipped for Regents scanning (e.g. paper, toner, printing capability, STARS scheduling)

Support schools in scheduling students for Regents in STARS and with all technical questions relating to ATS

Verify schools have performed data verification of students exams

Provide any additional training to schools on ATS or STARS, as requested

Be on-hand during Regents week to provide support as needed

Monitor STARS and ATS before & during test administration for data integrity issues and general activity. Be on hand for support during Regents week.

Escalate any unanswered policy or process questions/issues

Escalate any technical policy or process questions/issues

3030

Turn-Keying: Network & School ChecklistMain Action Items Timeline

CFNs coordinate trainings with schools May 1-31

Schools notify and train relevant staff, and make preliminary preparations (check toner, paper supplies)

May 1-31

Schools utilize demo environment to run mock exam process May 2-31

Schools schedule students for Regents in STARS By May 28

Schools print answer documents for majority of students By June 10

Schools establish & communicate Regents week plan (roles, logistics) June 1-14

Schools administer exams, scan documents, conduct data validation June 15-24

*Refer to Implementation Checklist for a complete list of tasks to be performed by schools.

3131

SED Special Study – June 2011 Regents

> Affected Regents exams: Comprehensive English Algebra 2/Trigonometry

> Affected schools: See NY SED Special Study for June 2011 Regents – Affected

Schools handout> Process highlights:

The special study process will supersede any NYC DOE administrative process – including Regents Scanning for the Comprehensive English exam.

CFNs will receive instructions on how affected schools must administer these exams.

The NYS Education Department is conducting a state-wide special study which will affect select schools for two June 2011 Regents.

3232

On-Going Support & Resources

> Documentation and updates: ARIS: https://www.arisnyc.org/connect/node/395478/community

> Technical support (STARS & ATS): CFN Data & IT Liaison

> Scanner and printer: DIIT Help Desk, 718-935-5100

> Overall process support: Borough Assessment Implementation Directors (BAID)

> Regents week emergency inquiries: Patrick Purcell (ppurcell@schools.nyc.gov; 212-374-5553) Marc Bush (mbush3@schools.nyc.gov; 201-755-2519) Kirby Fields (kfields6@schools.nyc.gov; 646-379-8422)

The following resources are available to support implementation and execution of Regents scanning.

*Please note that guidance from DOE on unresolved items will be issued on a rolling basis.