NCTE/NCATE REVIEWER NOTES:Using the PRS System and Other Issues in the Review Process

Part I – The PRS SystemThe online system that NCATE has in place for reviewing all programs requires specific

procedures from the time that reviews are assigned. The following information should help withthe technical aspects of program review.

1. NCATE staff assigns the Login ID and password necessary to enter the PRS systemfrom the following URL: http://prs.ncate.org/. The ID is the email address that youhave supplied to NCATE through the NCTE staff liaison. The password is a uniquecombination of letters and numbers randomly selected and assigned. Only NCATEstaff can change these entry codes.

2. The PRS system is set up to run on any computer that can utilize Adobe Reader 7.0(or higher). Certain MAC users must follow directions on the initial PRS site in orderto utilize the Reader. Configuration for users of Firefox is found on the site and in theAppendix that follows these notes. Users of Safari will need to contact NCATE forspecific directions if they are unable to load Firefox onto their MAC’s.

3. See the following frames copied from the NCATE/PRS site showing the login pageand the reviewer page.

LOGIN PAGE

E-mail address sent byNCATE

Random letters and numberssent by NCATE

MAC user information

Adobe Reader download

REVIEWER PAGE

Containsteam memberand SPAcontactinformation

Copy of priorsubmission

Template for allTemplate for all SPAsSPAs

Lead reviewer submitsseparate report

1 3

Consensus report byLead Reviewer.Clone either review(1, 3) or write anew.

Copy ofprior auditreport

4. To open the program report, click on the .pdf icon in the column labeled “Current.”When you are ready to write your reviewer report click in the appropriate reviewerreport column (yellowed in). Lead reviewers place their original report in the “LeadReviewer” column and their consensus or final report in the “Team Report” column.Lead reviewers will find numbers in their Team Report column ( or .). Thesenumbers refer to the other reports and can be used to “clone” either report as the teamreport. The clone can be revised prior to submission, but only one of the initialreports can be cloned for the Team Report. Clicking on the “people group” under thecolumn heading “Team” will bring up information on the team, Program Coordinatorand NCATE liaison. Note: Until a review report has been “submitted,” a torn papericon will appear in the column. After the review report has been submitted, a pdficon will appear only after the reviewer has logged out and then logged in again.

5. See the following frames for information on how the program report initially willappear on your monitor as a pdf. file. Note that there should be tabs on the left handside of the screen. The attachment tab will open a window with all the attachments tothe program report. The body of the report does not contain the assessmentinformation, the program of study, and other graphs or charts. There is a limit to thenumber of attachments a program may submit. At present that number is 20.

PROGRAM REPORT PAGE

Open this tab to bring up theattachments.

6. As you initially read through the pages of the report, you may wish to leave the attachmentsunopened in order to make the print more readable. Some reviewers prefer to print theprogram report for ease of reading; others print the attachments for the same reason. If youare comfortable reading from the screen, it is not necessary to print any part of the report. Itis highly recommended, though, that you take notes as you read both the report and theattachments to simplify writing your review.

REPORT FORM WITH ATTACHMENTS OPEN

Click here to alphabetize the list ofassessments

7. Looking at the frame above, note that these attachments are in WORD. Some programsattach EXCEL documents or pdf files. You should have no problem opening theseattachments if you have these programs on your computer. If you have a problemopening any attachment, contact the NCTE/NCATE Program Coordinator who will beable to assist you either by phone or by e-mail.

8. Note also that the column labeled “Description” shows the attachments in alphabeticalorder. If your attachments are not in alphabetical order, you can click on the title columnand they will rearrange themselves accordingly.

Part II – Issues for ReviewersA. All Reviewers

1. There is a standard Program Recognition Report Form for all reviewers to use. Ablank copy of this form can be found on the PRS reviewer page (third column).

All information comesfrom the Program ReportCover Page.

2. All recognition decisions and any conditions from a decision of “Recognized withConditions” are at the front section of the report (Part A). Even though you may fillin later Parts first, the report is not finished until all parts/sections of the report arecompleted. This includes a decision for each Program Standard in Part B, and allsections of Parts C, D, E. Part F is optional for the reviewer.

3. The following frame contains explanations for the testing information required byNCATE and the section on meeting the Standards. You will note that a third choicehas been added to meeting or not meeting the Standard. This choice should only beused if the program recognition decision is “Recognized with Conditions.” For allother decisions, each Standard is either “met” or “not met.” A Standard may be metat either the “acceptable” or “target” level (See Handbook for Reviewers).

RECOGNITION REPORT FORM (PAGE 2)

This section should only be filled in for“conditional recognition” decisions

If the program has less than 10 completers,the test results are not applicable.

This section must be completed evenif the program is not recognized.

This status only should be used if theprogram is Recognized with Conditions

4. If the status of meeting any Standard is “not met” or “met with conditions,” then thereviewer needs to comment on the reason for the decision. This comment need not belong, but it should give a reason. The following are some of the comments thatreviewers have used for this section:

“Insufficient evidence in cited assessments to show that candidates have the opportunityto meet the standard.” “No evidence…” (Not met)“No assessments cited for this standard.” (Not met)“The standard requires engagement of the candidate with students. The assessments citeddo not meet that requirement.” (Not met)“The assessment(s) cited is/are too generic to provide evidence for this standard.” (Notmet).“Although there is evidence in the cited assessments, the required amount of data has notbeen collected yet.” (Met with conditions)“The assessments cited contained only limited evidence. If an overlay or addendum isimplemented, more substantive evidence should be available.” (Met with conditions)“No (or insufficient) evidence found in the assessments cited. (An)Other assessment(s),not cited for this standard, did contain evidence. See Part C” (Met)

Although it is possible to just reference Parts C and E where the assessments may becommented on specifically and at length, most reviewers comment within the Standardssection. There is no problem with repeating the same comment throughout the report ifthe decision reflects the same reason.5. The next frame illustrates Parts C, D, and E of the Recognition Report where more

substantive criteria should be given to assist the program in understanding the finaldecision.

Comments should focus onAssessments 1, 2, and othercontent based assessments.

Comments should focus onAssessments 3, 4, and otherprofessional content orpedagogical assessments.

Comments should focus onAssessments 5, and othereffects based assessments.

Comments should focus onSection V of the submission..

Must be completed for “Notrecognized”; may becompleted for any otherdecisions..Part F is optional

B. Lead Reviewers1. Even though you will be writing the “Team Report,” it is essential that you work with

your team to come to consensus prior to submitting this final report. You may workvia e-mail or conference call (See Cavalier instructions in the Appendix to thisdocument), but you need to be in touch with your team member(s).

2. Until the final Team Report is posted by hitting the “submit” button and then exitingPRS, the audit team cannot process your report. Since we are all on deadlines, alwayslet the SPA Coordinator know if you will not be able to submit the final team reportby the NCATE deadline. The Coordinator has some leeway based on the number ofreports processed in each round, but it is always best to “be on time” with that finalreport.

3. Some report decisions may be changed during audit. The NCTE auditors are themost experienced reviewers in the program and they have a keen eye for the match of

assessments to standards with the time to search for evidence that a less experiencedreviewer may overlook. They also are focused on the specific English language artsrequirements of the standards; generic assessments provide little evidence in theseareas. If a decision is changed, the new report is available on the reviewer’s webpage under “audit” after the report has been sent to the institution. The Coordinator isalways available to discuss any change with the lead reviewer or team member.

Part III – Revised Reports.A. Reports revised from a “Not Recognized,” “Further Development Required,” or

“Recognized with Probation” Decision1. Programs may submit revised reports that contain changes or revisions that are

not substantive in terms of their overall program. There might be revisions totheir existing assessments or they may design and implement an additionalassessment. These reports are considered “revisions.” Even though they arerevisions, the programs must adhere to the NCATE data requirement, presentlyone year (two years beginning in Fall 08). The original program submission andthe original final audit report can be found along with the submission of therevision on the reviewer’s PRS web page.

2. Programs may decide to substantively change their entire program with majorrevisions to existing assessments and/or the addition of assessments within theNCATE required five types. These reports are also considered revisions althoughthe program may label them “new” reports because of the changes from the initialsubmission. Even though they are substantively new, the programs must adhereto the NCATE data requirement, presently one year (See above).

3. Revised report review teams should have a member who was on the original team.Scheduling difficulties may preclude this happening. The NCTE/ NCATEProgram Coordinator will work closely with any “new” team that reviews arevised report.

B. Reports revised from a “Recognized with Conditions (R-C)” Decision1. When a report receives an R-C decision, the program has approximately 18

months to meet the conditions stated in the report. Dates are provided by NCATEwhen the report is returned to the program.

2. Reviewers must focus ONLY on the conditions stated on the prior submission.No “new” problems should be articulated. If, in trying to meet the conditions, theprogram does create other problems, of course these must be considered in thereview decision. Also to be considered is a program that does not focus itsrevision on the conditions given it in the original submission.

3. Reviewers may decide that a program has not had sufficient time to collect therequired amount of data and, therefore, extend the R-C decision for an additional18 months. If the program has not met the conditions of the prior decision thenthe reviewers may decided to lower the decision to “Further DevelopmentRequired” or “Recognized with Probation.”

APPENDIX

Instructions for MAC users of Firefox

Select "Firefox" from your menu bar and choose "Preferences"

Under Preferences, click "Content" to make sure "Enable Javascript" is checked and then click"Advanced" button next to "Enable Javascript"

Select "Raise or lower windows" and click "OK"

MAC sometimes opens the report by using its "PREVIEW" application instead of using AdobeReader(See Menu Bar).To quickly find if the program report was opened by Adobe Reader:

If there is no "Adobe" next to your "Apple" icon on top left of the screen, that means youare opening the report by "Preview".

If you cannot see an application tab (far left side) when you are on the Program Report page,your system has not opened the Reader correctly. You either need to contact your IT person ore-mail NCATE. See below

Contact Cora at NCATE for further assistance: cora@ncate.org

Conference Call Information

(This information may change periodically; contact NCATE for the most recent codes if thisinformation does not work)

Default Chairperson Name: SHERRIE KOOBCompany: NATL COUNCIL FOR ACCREDITATIONAccess Phone Number: 800-563-4076International Access Phone Number: +1 30324891307-Digit Access Code: 4166144Passcode (may be needed by Lead Reviewer): 2010Web URL: www.cavtel.com/conferenceLines available per conference: 24

Setting up a conference is easy! Just follow these simple steps:1. Inform all participants of the date and time of the conference and give them the Access

Phone Number and 7-Digit Access Code2. When it is time for your conference call to begin, dial the Access Phone Numbe, then

follow the prompts to start the call.3. to join the conference, participants should dial the Access Phone Number at the

designated time, then enter the 7-Digit Access Code.

Lead Reviewer Conference Controls* 1 Dial out to a participant (then follow prompts - see below)

* 1 Join new participant* 2 Join new participant to the conference & dial a new participant.* 3 Disconnect line & rejoin conference* 4 Disconnect line & dial another participant

* 2 Record conference (on/off) (if activated)* 4 Lock conference* 5 Unlock conference* 6 Mute line* 7 Unmute line* 8 Conference continuation* 9 Roll call (if activated)* # Count participants# # Mute all lines9 9 Unmute all lines0 0 To ask for an operator* * List available commands

Team Member Conference Controls* 6 Mute line* 7 Unmute line* 9 Roll call (if activated)* # Count participants0 0 To ask for an operator* * List available commands

Please call Cavalier Customer Care at 800-930-6794 or 303-446-4637 if you need assistance.

When call has been made, Chairperson should send an email to sherie@ncate.org with a cc tomargie@ncate.org indicating (a) date & time of call, and (b) the name of the SPA. Doing thiswill assist NCATE in tracking and authorizing the calls when they appear on our bill fromCavalier.