PAT I E N T- C E N T E R E D O U TC O ME S R E SE A RC H I N ST I T U T E

Patient-Centered Outcomes Research Institute (PCORI)

PCORI Online Instructional Guide for Reviewers

About PCORI Online

What is PCORI Online?

• PCORI Online is the tool used to manage all aspects of PCORI’s grants process

• PCORI Online is used not only by applicants, but by Reviewers and SROs

• From an applicant’s letter of intent (LOI) to scoring for Reviewers - all of this is done via PCORI Online

• PCORI Online is accessible anywhere there is Internet access with a web browser

• PCORI Online will be the tool you use to download applications for review, score applications, and submit scores and comments

All scoring is to be done using the PCORI Online tool. Do not submit scores via email or other means

Accessing PCORI Online

• Open your browser and go to the following URL: contracts.pcori.org

• Enter your login credentials - this is your username and password combination - and click the ‘Submit’ button

• If you are a Reviewer, you should have received your login credentials. If you did not, please contact reviewers@pcori.org. If you forgot your password, click on ‘Forgot Your Password’

• If you continue to have issues logging in, please send an email to reviewers@pcori.org

The time out is set to 1 hour. If there is no activity on the page (e.g. user has not saved the page), then there will be a pop-up 10 min before the timeout.

How do I Access PCORI Online?

Enter your email address

My Tasks

• After login, PCORI Online takes you to the ‘Tasks’ tab. Here you can see tasks that are assigned to you.

• Notice that the task to review applications is listed under ‘My Tasks’

• Under task, click on ‘Complete Phase 1 Reviews’

• On the left is the navigation menu. The green bar indicates which section you are in.

If you are idle for 10 minutes in PCORI Online, the system will log you out

Home Page

Confidentiality

• After opening the task for the first time, you are taken to the confidentiality page. Please read the confidentiality agreement and the conflicts of interest agreement in the documents section.

• You must agree to the confidentiality agreement and confirm that you viewed/attended the training sessions by checking the ‘I Agree’ check box

• If you do not select "I agree", you will not be allowed to retrieve your assignments.  If you do not agree with the agreement, contact reviewers@pcori.org

• Click ‘Save and Continue’

Read Agreements

Select Checkbox

Read the Confidentiality and COI Agreements

Assigned Applications

• After agreeing to the confidentiality agreement, you are taken to the ‘Review Assignments’ page where you will see all the applications that your SRO has assigned to you

List of assigned applications

Instructions

Conflicts section

Review Applications Assigned to You

Assigned Applications: Columns

Under the ‘Assigned Applications’ section you will see a list of columns. They are defined as follows:

EGID- This is a unique ID assigned to the application in the system. No two EGIDs are the same

Primary Contact/PI- The main contact for the application. This may also be the PI

Organization- The organization from which the application is coming from

Role- Your role as the reviewer: Primary, Secondary or Tertiary (For Phase 1, all reviewers are the same, these roles are for internal management)

Overall Application Score- This is displayed after you have completed scoring the application

Conflict/Decline- This is used to indicate whether you have a conflict with the application or not. Also, if you choose to decline the review, you must use this dropdown

Conflict/Decline Type- This a dropdown menu used to indicate the conflict type if you have one and reasons for choosing to Decline.

PDF Icon- You may view the application in PDF format at anytime by clicking the PDF icon

Application Conflicts/Declining Applications

Before scoring an application, you must indicate whether or not you have a conflict. To do so:

No Conflict- If you have no conflict, select ‘No Conflict’ from the ‘Conflict/Decline’ menu

Conflict- If you have a conflict, select Conflict from the ‘Conflict/Decline’ menu

You must also indicate the type of conflict. Select the type of conflict from the ‘Conflict/Decline Type’ dropdown menu.

Decline- If for some reason (apart from a conflict) you do not wish to review the application, e.g. insufficient expertise or time, please select ‘Decline’ from the ‘Conflict/Decline’ dropdown menu and then select the appropriate reason from the ‘Conflict/Decline Type’ dropdown menu

Click on the PDF icon to view the application and check out sections of the application such as ‘Personnel’ and ‘Organization’ for potential conflicts

Assigned Applications

After indicating conflicts or no conflicts, click on the ‘Save’ button

Those applications for which you have a conflict or decline to review no longer show up

You will then need to click on the ‘Critique Form’ Icon tobegin scoring the application

Note: If you accidentally decline an application or indicate as having a conflict when there is none, please contact your SRO

Critique FormConflicts/Decline Applications No longer appear

Accessing the Critique Form

Scoring

About the application scoring screen:

From your assigned applications listing, after you have indicated whether you have conflicts or not, the ‘Critique Form’ icon appears. Click on this icon.

Upon clicking the icon, you are taken to the scoring page for that particular application

PCORI’s eight criteria are listed, along with a section on Human Subjects

The eight criteria are each broken out separately with the criterion described, a dropdown for the 1-9 scoring scale, and a comment box Enter bulleted list of

three strengths and three weaknesses

Criteria description

scoring chart below provides a descriptive guide of how strengths and weaknesses are considered in a rating:

Impact Score Descriptor Additional Guidance on Strengths/ Weaknesses

1 Exceptional Exceptionally strong with essentially no weaknesses

2 Outstanding Extremely strong with negligible weaknesses

3 Excellent Very strong with only some minor weaknesses

4 Very Good Strong but with numerous minor weaknesses

5 Good Strong but with at least one moderate weakness

6 Satisfactory Some strengths but also some moderate weakness

7 Fair Some strengths but with at least one major weakness

8 Marginal A few strengths and a few major weaknesses

9 Poor Very few strengths and numerous major weaknesses

High

Medium

Low

For the application score and individual criterion scores, the far right column in the

Scoring Cont’d

Scoring Cont’d

From the dropdown menu select the 1-9 score listed for the specific criterion

A narrative is required for each criterion in addition to the 3 strengths and 3 weaknesses

Enter your narrative in the comments field. Use a bulleted list describing three strengths and three weaknesses of the application.

There is a 5000 character limit in all of the comments fields

If you find a conflict of interest at any point during your review of the application, notify your SRO immediately

Scoring Scale

Entering Scores and Comments

Human Subjects

You must fill out the section on Protections for Human Subjects

From the dropdown, indicate ‘Yes’ or ‘No’ whether an application has acceptable risks and/or adequate protections in place for human subjects

If ‘No’ in the comments is filed, please provide comments on relevant Human Subjects concerns/flaws in the application

If ‘Yes’, in the comments field, please describe the safeguards in place

Indicate if acceptable or not

Enter Human Subjects Comments

Overall Application Score

Finally, you must enter an overall application score for the application

Enter the score from 1-9 using the ‘Overall Application Score’ dropdown menu

In the comments section, enter your narrative and indicate three strengths and three weaknesses with regards to the overall impact of the application. There is a 5000 character limit for the comments field.

Enter an Overall Application Score and Comments

Submitting your Scored Application

You must complete all assigned applications before you can submit. The system will not allow you to submit individual applications one at a time.

After completing the scoring of each application, click on ‘Save’

Once you have completed scoring all applications, the ‘Submit’ button now appears on the Review and Submit page

Click on ‘Submit’ to submit all your applications

Once you submit your scored applications, you cannot go back and revise your scores. Make sure your scores are correct before submitting

‘Submit’ appears when all applications are completed

Will indicate complete/incomplete items

Review and Submit

PCORI Online Tips

The time out is set to 1 hour. If there is no activity on the page (i.e., user has not saved the page), there will be a pop-up 10 minutes before the timeout

You must complete scoring all of your assigned applications before you can submit. Example: If you are assigned 3 applications, you must complete scoring for all 3 in order to submit.

If you need help or have a question, please submit your question to reviewers@pcori.org. Your request will be responded to within 2 business days.