district data literacy survey analysis -...
TRANSCRIPT
District Data Literacy Survey AnalysisPrepared for the Arkansas Department of Education
September 2018
www.hanoverresearch.com
Table of Contents
2
Introduction ……………….. P 3
Recommendations ……………….. P 4
Key Findings ……………….. P 5
Section I: Respondent Overview ……………….. P 9
Section II: Data Use ……………….. P 12
Section III: Data Access, Analysis, and Guidance ……………….. P 19
Section IV: Training Needs ……………….. P 24
Section V: Additional Feedback ……………….. P 29
Introduction
3
This report presents the results of surveys administered by the Arkansas Department of Education (ADE) to school district test coordinators and similar district employees about the use of various data types and systems in their districts.
In total, 185 respondents completed at least part of the survey. Only one response per district was kept, leaving 143 respondents in the analysis.
Recommendations
Based on the results from this survey, Hanover recommends that the ADE:
Offer targeted trainings and guidance to support districts to use student growthdata, as well access and analyze data from sources with which district leadersare less commonly comfortable (e.g., Arkansas Student Intervention System,Cognos, SIS reports/Data Center, and studentGPS);
Support districts to use effectively the types of student achievement data thatare most predictive of student secondary and post-secondary success;
Provide trainings primarily through webinars or in-person conferences andworkshops, rather than self-paced courses; and
Consider how the needs and concerns of district leaders may vary by districtsize. While respondents report that district leaders often use studentperformance data similarly regardless of the size of their district, respondentsindicate that larger districts are more likely to use ELPA21 data and data fromunspecified sources, as well as use data to measure equity.
4
Key FindingsDistrict Data Use
▪ Respondents indicate that districts most frequently use studentperformance data as measured by ACT Aspire and K-12 interimassessments. Nearly all surveyed districts use ACT Aspire (99%) and K-2interim assessments (96%), while over 80 percent of districts also frequentlyuse student class performance, periodic assessment data, ACT/SAT, studentattendance, and growth data.
– Correspondingly, respondents indicate that these seven types of studentperformance data are also the most useful in their district's efforts towardsuccess. Likewise, respondents indicate that the least commonly used types ofstudent performance data (i.e., PSAT and NAEP results) are also the least useful insupporting student success.
▪ Nearly all (98%) of districts use student data to measure student progress,while approximately nine out of every 10 districts use student data tomeasure the effectiveness of instruction, plan professional and curriculumdevelopment, meet state and federal reporting requirements, and measureschool and/or program efficacy.
– Respondents report that districts are less likely to use student data to report tocommunities and stakeholders (79%), support schools and/or districts (69%), andmeasure equity (46%).
5
Key FindingsDistrict Data Use, By District Size
▪ Respondents report that, for the most part, districts frequently usestudent performance data from similar sources, regardless of district size.However, respondents at larger districts are significantly more likely toreport that their district frequently uses ELPA21 data and performance datafrom unspecified sources not listed in the survey.
– For example, 78 percent of districts with six or more schools and 73 percent ofthose with three to five schools use ELPA21 data, compared to only 40 percentof those with one to two schools that use this data.
▪ Accordingly, smaller districts are significantly less likely to considerELPA21 and alternate data sources (e.g., AAPA, MSAA, DLM) as useful.
– Only 30 percent of districts with one to two schools find ELPA21 a useful source,while only 25 percent find alternate assessments a useful source. Incomparison, 77 and 69 percent of districts with 6 or more schools that findELPA21 and alternate assessments useful, respectively.
▪ Larger districts are also more likely to focus on measuring equity. Indeed,74 percent of districts with six or more schools use data to measure equity,compared to 40 percent of districts with three to five schools and 38percent of districts with one to two schools that use data for this purpose.
6
Key FindingsDistrict Data Access, Analysis, and Guidance
Districts are typically more comfortable accessing and analyzing data fromsources for which the ADE provides guidance on their use:
▪ Respondents indicate that district leaders are more commonlycomfortable accessing and analyzing data from My School Info,Arkansased.gov, and various assessment portals. Over 60 percent ofrespondents indicate that their district's leaders are comfortable accessingand analyzing data from these sources. Similar percentages of respondentsalso indicate that the ADE provides clear guidance on how to effectivelyutilize these sources.
▪ Conversely, respondents indicate that district leaders are less commonlycomfortable accessing and analyzing data from sources including ArkansasStudent Intervention System, Cognos, SIS reports/Data Center, andstudentGPS. Forty-five percent of respondents or fewer indicate that theirdistrict's leaders are comfortable accessing and analyzing data from thesesources. Respondents correspondingly are less likely to report that the ADEprovides clear guidance related to utilizing these sources in comparisonwith other sources.
7
Key FindingsDistrict Training Capacity and Needs
▪ Respondents broadly report that their districts are equipped to supportschool and district leaders in the analysis and interpretation of studentdata. Indeed, three-quarters of respondents “agree” or “strongly agree”that their district is equipped to provide these supports.
– However, approximately half (48%) of respondents indicate that their districtwould like additional training related to the use of student growth data, whilebetween approximately a quarter and a third of respondents indicate that theirdistrict would like additional training related to topics including ACT Aspire, K-12interim and periodic assessments, various alternative assessments, the ACT/SAT,and student attendance.
▪ When seeking data literacy support, districts most often turn to co-ops(84%), colleagues (80%), and the ADE (78%). Relatively fewer respondentsindicated that their districts rely on professional organizations (24%) orvendors/consultants (23%) to provide this support.
– With regard to district leaders’ preferred delivery method for training arounddata literacy, respondents indicate that most leaders prefer to receive theirtraining through webinars (73%) or in-person conferences/workshops (61%).Conversely, only one-third of respondents indicate that leaders in their districtwould prefer self-paced online courses.
8
9
SECTION I: RESPONDENT OVERVIEW
Response Rate by District Size
10
District Size # of RespondentsTotal Districts
ContactedResponse Rate
1-2 Schools 55 113 49%
3-5 Schools 61 109 56%
6+ Schools 27 42 64%
Response Rate by District Size
Of the 264 districts that were invited to participate in this survey, 143 (54%)responded. As the figure below indicates, a higher percentage of districts withthree or more schools responded to the survey, in comparison with districtswith only one or two schools.
District and Respondent Breakdown
11
10%
90%
0% 20% 40% 60% 80% 100%
Other
District Test Coordinator
Which of the following best describes your position? (n=143)
19%
43%
38%
0% 20% 40% 60% 80% 100%
6+ Schools
3-5 Schools
1-2 Schools
District Size (n=143)
The majority (81%) of districts that responded to the survey have betweenone and five schools. Across these districts, district test coordinatorsresponded to this survey in nine out of 10 districts.
12
SECTION II: DATA USE
Performance Data Type
13
11%
20%
57%
59%
61%
67%
68%
81%
84%
84%
86%
88%
96%
99%
0% 20% 40% 60% 80% 100%
NAEP results
PSAT
Advanced Placement (AP)
Student discipline
ELPA21
Alternate assessments (e.g., AAPA, MSAA, DLM)
Graduation rates
Growth data
Student attendance
The ACT / SAT
Grades 3-10 periodic assessments (e.g., ACT Aspire Interim and Classroom)
Student class performance (e.g., grades, credits, GPA, transcripts)
K-2 interim assessments (i.e., Istation, Renaissance STAR, NWEA MAP)
ACT Aspire (summative assessments)
What student performance data does your district frequently use? (n=142)
Note: Percentages sum to more than 100 because respondents could select multiple options.
37%
11%
26%
59%
67%
78%
74%
59%
85%
85%
93%
81%
85%
96%
100%
20%
10%
22%
62%
55%
73%
68%
73%
88%
87%
83%
88%
88%
98%
97%
11%
11%
16%
51%
60%
40%
62%
65%
71%
80%
80%
85%
89%
93%
100%
0% 20% 40% 60% 80% 100%
Other*
NAEP results
PSAT
Advanced Placement (AP)
Student discipline
ELPA21*
Alternate assessments (e.g., AAPA, MSAA, DLM)
Graduation rates
Growth data*
Student attendance
The ACT / SAT
Grades 3-10 periodic assessments (e.g., ACT Aspire Interim and Classroom)
Student class performance (e.g., grades, credits, GPA, transcripts)
K-2 interim assessments (i.e., Istation, Renaissance STAR, NWEA MAP)
ACT Aspire (summative assessments)
What student performance data does your district frequently use?
1-2 Schools (n=55) 3-5 Schools (n=60) 6+ Schools (n=27)
Performance Data Type by District Size
14Note: An asterisk (*) indicates the presence of at least one statistically significant difference between district sizes.
Data Purposes
15
46%
69%
79%
89%
90%
91%
92%
92%
98%
0% 20% 40% 60% 80% 100%
To measure equity
To support schools and/or districts
To report to communities and stakeholders
To measure effectiveness of school and/or district programs
To meet state and federal reporting requirements
To guide curriculum development
To plan professional development
To measure effectiveness of instruction
To measure student progress by school and/or district
For what purpose(s) does your district use student data? (n=144)
Note: Percentages sum to more than 100 because respondents could select multiple options.
Data Purposes by District Size
16
74%
85%
93%
89%
100%
89%
100%
96%
100%
40%
67%
85%
95%
90%
92%
93%
97%
98%
38%
64%
65%
84%
85%
91%
85%
85%
96%
0% 20% 40% 60% 80% 100%
To measure equity*
To support schools and/or districts
To report to communities and stakeholders*
To measure effectiveness of school and/or district programs
To meet state and federal reporting requirements
To guide curriculum development
To plan professional development
To measure effectiveness of instruction
To measure student progress by school and/or district
For what purpose(s) does your district use student data?
1=2 Schools (n=55) 3-5 Schools (n=60) 6+ Schools (n=27)
Note: An asterisk (*) indicates the presence of at least one statistically significant difference between district sizes.
Data Type Usefulness
17
35%
37%
5%
12%
32%
27%
18%
12%
10%
11%
24%
26%
31%
33%
27%
31%
24%
18%
18%
15%
13%
10%
9%
7%
7%
9%
30%
29%
31%
39%
42%
37%
42%
41%
39%
34%
37%
35%
17%
21%
20%
19%
29%
37%
38%
41%
46%
54%
53%
57%
0% 20% 40% 60% 80% 100%
NAEP results (n=114)
PSAT (n=115)
Alternate assessments (e.g., AAPA, MSAA, DLM) (n=131)
Advanced Placement (AP) (n=129)
ELPA21 (n=126)
Student discipline (n=133)
Graduation rates (n=130)
Grades 3-10 periodic assessments (e.g., ACT Aspire Interim and Classroom)(n=130)
Student class performance (e.g., grades, credits, GPA, transcripts) (n=132)
The ACT / SAT (n=130)
Student attendance (n=136)
K-2 interim assessments (i.e., Istation, Renaissance STAR, NWEA MAP) (n=135)
Growth data (n=133)
ACT Aspire (summative assessments) (n=135)
How useful do you consider the following in your district's efforts towards student success?
Not Useful at All Slightly Useful Somewhat Useful Quite Useful Very Useful
Data Type Usefulness by District Size
18
8%
17%
69%
58%
77%
62%
69%
65%
65%
85%
88%
92%
96%
100%
10%
10%
46%
51%
54%
54%
73%
78%
87%
82%
79%
93%
95%
95%
8%
7%
35%
46%
30%
61%
71%
74%
81%
80%
91%
81%
82%
85%
0% 20% 40% 60% 80% 100%
NAEP results
PSAT
Alternate assessments (e.g., AAPA, MSAA, DLM)
Advanced Placement (AP)
ELPA21
Student discipline
Graduation rates
Grades 3-10 periodic assessments (e.g., ACT Aspire Interim and Classroom)
Student class performance (e.g., grades, credits, GPA, transcripts)
The ACT / SAT
Student attendance
K-2 interim assessments (i.e., Istation, Renaissance STAR, NWEA MAP)
Growth data
ACT Aspire (summative assessments)
How useful do you consider the following in your district's efforts towards student success?(% Very Useful + % Quite Useful)
1-2 Schools (n=39-53) 3-5 Schools (n=50-57) 6+ Schools (n=24-26)
Note: An asterisk (*) indicates the presence of at least one statistically significant difference between district sizes.
19
SECTION III: DATA ACCESS, ANALYSIS, AND GUIDANCE
District Leader Comfort Accessing Data
20
36%
41%
41%
45%
52%
62%
62%
64%
66%
72%
0% 20% 40% 60% 80% 100%
Arkansas Student Intervention System (ASIS)
SIS reports/Data Center
studentGPS
Cognos
Accountability reports center
Assessment portal(s)
Arkansased.gov (e.g., ADE Assessment Summary Reports)
eSchool
Triand
My School Info (e.g., Report Card, ESSA Index, Act 1015 Compliance)
I feel that district leaders are comfortable accessing data from... (n=135)
Note: Percentages sum to more than 100 because respondents could select multiple options.
District Leader Comfort Analyzing Data
21
35%
37%
37%
41%
52%
52%
53%
63%
68%
72%
0% 20% 40% 60% 80% 100%
studentGPS
Arkansas Student Intervention System (ASIS)
SIS reports/Data Center
Cognos
Triand
Accountability reports center
eSchool
Arkansased.gov (e.g., ADE Assessment Summary Reports)
Assessment portal(s)
My School Info (e.g., Report Card, ESSA Index, Act 1015 Compliance)
I feel that district leaders are comfortable analyzing data from... (n=133)
Note: Percentages sum to more than 100 because respondents could select multiple options.
ADE Guidance for Districts
22
44%
48%
50%
53%
56%
58%
59%
61%
74%
77%
0% 20% 40% 60% 80% 100%
Accountability reports center
Arkansas Student Intervention System (ASIS)
SIS reports/Data Center
Cognos
studentGPS
Triand
Arkansased.gov (e.g., ADE Assessment Summary Reports)
eSchool
Assessment portal(s)
My School Info (e.g., Report Card, ESSA Index, Act 1015 Compliance)
The ADE provides clear guidance on how to effectively utilize... (n=133)
Note: Percentages sum to more than 100 because respondents could select multiple options.
Guidance and Capabilities
23
66%
41%
41%
72%
64%
45%
62%
62%
36%
52%
52%
35%
37%
72%
53%
41%
68%
63%
37%
52%
58%
56%
50%
77%
61%
53%
74%
59%
48%
44%
0% 20% 40% 60% 80% 100%
Triand
studentGPS
SIS reports/Data Center
My School Info (e.g., Report Card, ESSA Index, Act 1015 Compliance)
eSchool
Cognos
Assessment portal(s)
Arkansased.gov (e.g., ADE Assessment Summary Reports)
Arkansas Student Intervention System (ASIS)
Accountability reports center
District Leader Capabilities and ADE Guidance
ADE Guidance (n=133) Analysis Comfort (n=133) Access Comfort (n=135)
Note: Percentages sum to more than 100 because respondents could select multiple options. No significance testing was run between questions.Bars denoted by “ADE Guidance” indicate answers to the question: The ADE provides clear guidance on how to effectively utilize...Bars denoted by “Analysis Comfort” indicate answers to the question: I feel that district leaders are comfortable analyzing data from...Bars denoted by “Access Comfort” indicate answers to the question: I feel that district leaders are comfortable accessing data from...
24
SECTION IV: TRAINING NEEDS
Additional Training Topics
25
5%
7%
8%
10%
11%
11%
14%
24%
24%
25%
26%
29%
32%
48%
0% 20% 40% 60% 80% 100%
PSAT
NAEP results
Advanced Placement (AP)
ELPA21
Graduation rates
Student class performance (e.g., grades, credits, GPA, transcripts)
Student discipline
Student attendance
The ACT / SAT
Alternate assessments (e.g., AAPA, MSAA, DLM)
Grades 3-10 periodic assessments (e.g., ACT Aspire Interim and Classroom)
K-2 interim assessments (i.e., Istation, Renaissance STAR, NWEA MAP)
ACT Aspire (summative assessments)
Growth data
For which of the following student performance data topics would your district like additional training? (n=130)
Note: Percentages sum to more than 100 because respondents could select multiple options.
Training Methods
26
33%
61%
73%
0% 20% 40% 60% 80% 100%
Self-paced online course
In-person Conference/Workshop
Webinar
How would leaders in your district prefer to receive training? (n=132)
Note: Percentages sum to more than 100 because respondents could select multiple options.
District Capabilities in Training
27
14% 8% 57% 18%
0% 20% 40% 60% 80% 100%
My district is equipped to provide support to school and district administration for analyzing and interpreting student data. (n=132)
Strongly Disagree Disagree Neither Agree nor Disagree Agree Strongly Agree
75%
Data Literacy Support Resources
28
23%
24%
78%
80%
84%
0% 20% 40% 60% 80% 100%
Vendors/consultants
Professional organizations (e.g., AAEA)
ADE
Colleagues
Co-op
When seeking data literacy support, which resource(s) does your district utilize? (n=132)
Note: Percentages sum to more than 100 because respondents could select multiple options.
29
SECTION V: ADDITIONAL FEEDBACK
Open-Ended Responses
30
Please use the space provided to share any other comments you have about student performance/assessment data.
Changing K-2 assessments was helpful for our district. We are not able to look at useful data and make decisions about instruction, curriculum, and student needs. Please look at the timeline for school improvement plans and how it coincides with the release of student performance data. District and school instructional teams need time to dig deep into the data to provide realistic and meaningful strategies for improvement. Until this happens, the improvement plans will be more of a document for compliance and not a document for actual use and improvement.
Student performance and school performance is becoming a wide spectrum. Each district should be mandated to employ a person thatsolely concentrates on dissecting data for the schools and staff. I would encourage the Arkansas Department of Education to make this a priority if data is to be analyzed properly in the years to come.
We need more training to use these effectively. I attended one of the ambassador workshops last year, and I was surprised at all of the data/tools available through programs like My School Info and Triand. There is a lot there, but most educators don't even know about it. I brought what I learned back to our district and shared a few key things. If we had some recorded webinars, we could keep going back to them to share with our teachers. It is really difficult to schedule PD time throughout the year, because there are so many things that need to be covered.
I am beginning my third year as the DTC of my district, and I still feel I am inadequate in assessing some of the data resources. I am a hands-on learner. Webinars are great introduction tools for me, but I cannot always apply what I've learned after the webinar is over. Workshops are similar; however, I do get a chance to ask questions and speak with other DTCs which helps a great deal. I would like classes, so that I can immerse myself into the tasks and really learn them.
Sample of Representative Verbatim Open-Ended Responses
Garrett BrannContent Director
202-350-4754
www.hanoverresearch.com
Michael HammondRelationship Director
202-793-8848
www.hanoverresearch.com