NCHRP 20-65 (47) - MTAP Survey Tool Used to Assess FTA Contractor Performance of State DOT Triennial and Other FTA Reviews - An UpdateBiennial FTA State Programs Meeting / State Public Transit Partnerships Conference August 7, 2103

KFH Group, Inc.Beth HambySue Knapp

Background•To assist FTA with improving State Review

processes, MTAP collects input from State DOTs on their experience following each FTA review via an online survey tool.

•Survey was created in Vovici (used by AASHTO for other surveys).

•Value of survey has been somewhat limited by: ▫open-ended answer formats▫challenges in using Vovici

Research Objective:•Update MTAP’s online survey to:

▫improve efficiency / ease of completion▫increase the value added nature of the

process (improve results for ease of use, benefits)

▫incorporate technology improvements▫provide for more detailed responses to 2-3

specific questions of interest to MTAP, SCOPT, FTA - not just a "comments" field

Project Tasks•Task 1 – Assess Survey Tool Improvement

Needs•Task 2 – Revise Survey Content & Structure•Task 3 – Assess Available Survey Technology

Options•Task 5 – Set Up & Test Revised Survey

Using Selected Technology•Task 6 – Prepare Final Report

Project Status

•Task 1 – Assess Survey Tool Improvement Needs – completed

▫Reviewed current survey tool - http://vovici.com/wsb.dll/s/54c9g50982

▫Reviewed reports generated from current tool

▫Gathered feedback/suggestions from States/MTAP Committee FTA Office of Oversight and Program

Guidance MTAP Coordinator

Project Status•Task 2 – Revise Survey Content &

Structure – nearing completion▫Modified existing questions –

standardized responses where possible to make analysis easier

▫Added new questions▫Prepared revised survey for review

by MTAP/ Steering Committee; sent late July

Draft Revised Survey• Numerous new questions seeking specific feedback

▫Basic experience of reviewer▫Consultant performance on specific compliance

topics▫For each scaled rating (Excellent/ Good/ Fair/ Poor),

asks why (open-ended)▫Extent of Regional participation▫Effectiveness of entrance/exit meetings▫SMR workshop effectiveness▫Interest in follow-up discussion with MTAP or FTA

• Standardized responses; limited open-ended questions to explanation and “other”

• Reworded some questions to align with responses

Draft Revised Survey1. Contact and Background Info

▫ Added: Basic experience of respondent2. Consultant Performance

▫ Added: Fields for explanation3. General Experience with Review

▫ Added: Regional office participation▫ Added: Entrance/Exit meetings▫ Added: Overall organization and clarity

4. SMR-Specific Feedback▫ Added: Respondent’s participation▫ Workshop/workbook effectiveness▫ Additional training/tech assistance needs

5. Additional Feedback

Project Status•Task 3 – Assess Available Survey

Technology Options – under way ▫Criteria include:

User-friendliness to create & administer survey

User-friendliness to complete & submit surveys

Functionality/customization options Reporting capabilities Cost

Survey ApproachesGeneral options for getting feedback from

states:

▫Online survey tools – basic to advanced Focus of our assessment

▫Pen & paper Too labor intensive

▫Telephone interviews May be appropriate for follow-up

Basic Online Survey Tools•Examples include:

▫SurveyMonkey Basic / Select / Gold ▫SurveyMoz Free / Premium▫SurveyGizmo Basic▫QuestionPro Professional

Basic Online Survey Tools•Advantages:

▫Easy to learn and use▫Provides some skip logic functionality▫Provides some basic cross-tab analysis▫Can export results to Excel for additional

analysis▫Inexpensive ($0 - $25/month)

Basic Online Survey Tools• Disadvantages:

▫Limited collaboration capabilities▫Limited survey customization options▫Limited reporting / analysis functions – need to

export to third party application

• Basic tools are becoming increasingly sophisticated (e.g., SurveyMoz)

• Some offer higher levels of customization for a higher cost (e.g., SurveyMonkey Platinum - $65/month)

Advanced Online Survey Tools•What MTAP is currently using (Vovici)

•Known as Enterprise Feedback Management (EFM) Systems

•Examples include:▫Key Survey▫Qualtrics▫QuestionPro Corporate▫SurveyGizmo Enterprise▫Vovici

Advanced Online Survey Tools•Advantages:

▫Allows for greater collaboration in survey administration (behind the scenes) Multiple user accounts possible Can share surveys with other survey

administrators Private results through secure dashboard accounts

▫Greater flexibility in branding and customizing ▫Can build sophisticated logic into survey▫Allows for complex analysis▫Custom reports can be developed within the tool

Advanced Online Survey Tools•Disadvantages:

▫Complex interface▫Requires training / steeper learning curve▫May require frequent use to maintain skills▫May require experience in Cascading Style

Sheets (CSS) to customize look and feel of survey

▫Expensive ($100 or more/month – few disclose pricing online) However, no out-of-pocket cost to MTAP for

Vovici; AASHTO already pays for it for other uses

Next Steps•Fine-tune survey questions with input

from MTAP/ Committee•Finish technology assessment•Set up & test revised survey using

selected technology•Prepare final report

For more information or feedbackBeth HambyKFH Groupbhamby@kfhgroup.com206-448-6749