anywhere, anytime, over time: longitudinal data to the teacher desktop

1

Anywhere, Anytime, Over Time:

Longitudinal Data to the Teacher Desktop

February 14, 2013NCES MIS Conference

Bethann CanadaVirginia Department of Education

Dave UhligCharlottesville City Public Schools

2

Program Goals

• Distribute $2 million in grants to Virginia school

divisions (districts)

• Encourage and finance data projects in divisions

• Advance the goals of the SLDS program

• Encourage collaboration among divisions

• Support the 4 ARRA education reform areas

3

25% ARRA reform areas

25% Feasibility and technical merit

15% Collaboration and/or reusability

15% Rationality of Budget

10% Stakeholder Involvement/Agreement

10% Evaluation Plan

Score Guidance

0 No information/non compliance

1 Weak

2 Below Average

3 Average

4 Above Average

5 Excellent

OVERALL SCORE =

(ARRA Score) *.25 + (Feasibility/Technical Score) * .25 + (Collaboration/Reusability Score) * .15 + (Budget Score) *.15 + (Evaluation Score) * .10 + (Stakeholder Score) * .10

Eligibility Criteria: developed by division stakeholders

4

Technical Assistance and Outreach

• Directed e-mails

• Superintendent e-mails

• Weekly newsletter

• Webinars (4)

• Competition website

5

Proposals

• 47 proposals, representing 80 divisions (61%)

• Requests totaled $4,326,847

• Seven projects were partnerships

• Geographically disbursed

6

Awards

• 27 awards made to 37 divisions

• Awards totaled $1,974,144

• 3 consortiums

Alexandria City

Richmond City

Hampton City

Norfolk

Williamsburg

Charlottesville

Buena Vista City

Roanoke City

Bristol City Colonial Heights City

Petersburg City

Chesapeake City

Awards

8

Type 1: Hire StaffType 2: Upgrade or ConsolidateType 3: New System/Different ProductType 4: Convert from ManualType 5: Division Designed Type 6: Other

Types of Projects

9

To create a longitudinal data system specific to special education through the purchase of a web-based software solution and training. Replace current paper/ Excel-based system with electronic system that will organize IEPs and allow the tracking of special education students. Project will create an automated data management and control system to track longitudinal growth and achievement and manage students’ learning paths K-12.

Division Name: Clarke County Public Schools

Project Type: Convert from manual system to electronic system

Period of Performance: 4/2011 – 12/2011Number of Divisions 1Amount Requested $63,670

SECTION SCORE

ARRA Reform Areas (25%) 4.5

Feasibility/Technical Merit (25%) 4

Collaboration/Re-usability (15%) 3.5

Rationality of Budget (15%) 4

Evaluation Plan (10%) 4.5

Stakeholder Involvement/ Agreement (10%)

5

OVERALL CALCULATED SCORE 4.20

Strength/Weakness Summary: Clear focus on 3 ARRA goals for special ed. students. Good use of technology to move from a cumbersome laborious process and good job in outlining the areas of evaluation. Overall value of program in terms of improved flow of data, increased data accuracy, better decision-making and student support clearly outlined. Strong and detailed explanation of value to individual stakeholder groups. There is a concern that the project timeline is insufficient to accomplish the project. RECOMMENDATIONS FOR SUCCESS: (1) Proposal says this system was not already added due to budget constraints, but ongoing costs will be covered by local budget – Determine how much it will cost in out-years, how it will be covered, etc. (2) Need more clarity about vendor commitments, measures of successful implementation, accountability, etc.

DIVISION INFORMATION

TIER TWO SCORES TIER THREE FEEDBACK

PROJECT SUMMARY

10

To build a data warehouse through the purchase of a Web-based software solution. System will provide educators a single access point for individual or integrated local/state/national assessments.• Educators will have greater access to data to inform

classroom/school/division decisions and will become expert in the use of data to improve teaching

• Student achievement will increase due to educator use of assessment data to differentiate instruction.

SECTION SCORE




Rationality of Budget (15%) 3.5



4.5


Strength/Weakness Summary: Year-long training and mentoring for teachers is a strong component and description of stakeholder engagement is clear and concise. Good approach to data centralization, good set of basic evaluation practices and appears to have good training program. Strong support of 3 of the 4 ARRA reform areas. May be helpful to include a little more detail on the activities which will support stakeholder engagement in the future. Unclear regarding status of continued funding.RECOMMENDATIONS FOR SUCCESS: Ensure that the vendor and the division’s definitions of how to link student data with teachers are the same. Have a plan for how the information will be used. How will end users be engaged in the development of the access tools – start with their needs in mind. Establish that the privacy, security and confidentiality of the data will be maintained.


Division Names: Arlington Public Schools Wise County Public Schools

Project Type: Create new system / Convert to different product Period of Performance: 1 year beginning 4/11 Number of Divisions 2Amount Requested $89,770Amount per Division $44,885


PROJECT SUMMARY

11

TIER TWO SCORES

Create a platform where data from multiple sources can be merged and accessed through the purchase of software. Division will merge data sets to create a complete system that will generate reports, analyze data and monitor student achievement/intervention history. Upgrade to Pearson Inform will result in (1) data-driven adaptation of instructional plans to the changing needs of students and groups, (2) the ability to assess the impact of various programs, intervention strategies, and teachers, and (3) closer, more timely monitoring of student achievement and growth at the individual classroom, schools and district levels. System will impact instruction and other applications.

SECTION SCORE



Collaboration/Re-usability (15%) 2




4


Strength/Weakness Summary: Clear and detailed support for ARRA goals with well thought out impacts, comprehensive assessment of current problems and clear view of potential benefits to stakeholder group. Realistic view of funding – division will assume costs after project funding expended. Concern school will budget sufficiently to assume ongoing costs. No real details on incentives for different stakeholders to participate. RECOMMENDATIONS FOR SUCCESS: It appears the primary key to success in this project is the merging of "data silos" into one reposi-tory. This successful implementation requires a detailed analysis of the data currently collected, how it will be merged (data mapping) into the repository and, finally, a quality analysis plan to determine no data were lost. The user experience is reflected in the value of the reports developed. The Evaluation Plan, if followed, has the correct steps necessary to achieve success in this area.

Division Name: Charlottesville City Public Schools

Project Type: Update/upgrade current system/ Create central system/consolidate data

Period of Performance: 4/2011 – June 2013Number of Divisions 1 Amount Requested $ 40, 317


TIER THREE FEEDBACK

PROJECT SUMMARY

12

To consolidate existing disparate data systems into a central system through the purchase of a Web-based, application-neutral collection, aggregation and reporting portal (software purchase) to provide longitudinal student data. Will purchase hardware, configure data and provide professional development. System will enable access to understandable information about students and division will provide training on how to use the system. (PD will use a “train the trainer” approach.)

Division Name: Roanoke City Public Schools


Period of Performance: 7/2011 - 9/2011Number of Divisions 1Amount Requested $ 74,926

SECTION SCORE





Evaluation Plan (10%) 4


4



Strength/Weakness Summary: Very detailed explanation of how ARRA goals will be supported by the program, value and benefits to stakeholders and cost sharing details. Fully supports 3 ARRA areas and some support of the fourth. Addresses quantitative and qualitative measurement in the plan in many areas. Aggressive schedule may be difficult to achieve. Would have been beneficial to see evaluation of implementation included as well as more data on stakeholder engagement and participation. RECOMMENDATIONS FOR SUCCESS: A primary concern is knowing that the consolidation of disparate data is providing the desired information. A well defined data model with mappings between sources is one key to project success. Another is the development of several quality test cases (e.g. a sample report) that when constructed manually, can confirm that the report automation is accurate. These "golden test cases" can be used to confirm updates to the system have not created errors in reports. Finally, the success of the system is correlated closely with its usability. Continued interaction with users will allow the system interface to evolve to a desired tool versus an annoyance.


PROJECT SUMMARY

13

This project will fund the purchase of a data warehouse service and user-friendly software to create a system that assimilates data from disparate sources into a central repository and allows analysis of data. The goal is to have a simple-to-use solution that requires minimal training and provides the needed data without additional complexity. Award will pay for warehouse, design, implementation and training. Solution is a full turnkey solution.

Division Name: Bristol VA Public SchoolsProject Type: Update/upgrade current system/ Create central system/consolidate data

Period of 3-5 months, beginningPerformance: Summer 2011 Number of Divisions 1Amount Requested $75,000

SECTION SCORE







3.5


Strength/Weakness Summary: Sound approach and division is providing funding as well. Program plan contains comprehensive detail on how the solution advances each of the 4 ARRA reform areas. Contains clear statements on detailed benefits to different stakeholders. Not much detail on the assimilation (e.g., data mapping from all sources, frequency of update or whether it will be performed). Limited budget details with specific line item costs. Does the system improve anything that can be measured? RECOMMENDATIONS FOR SUCCESS: ( 1) Although described as a “turnkey” solution, needs more info on deliverables, representations of contractor regarding functionality, how successful implementation will be measured and assured, etc. (2) Realistic evaluation plan could help to articulate expectations and make sure the vendor shares the district’s goals and is ready to perform.



PROJECT SUMMARY

14

Division Name: Floyd County Public SchoolsProject Type: Update/upgrade current system/ Create central system/consolidate data

Period of Performance: Spring 2011-Spring 2013

Number of Divisions 1Amount Requested $ 75,000

To enhance the functionality of the current system by developing reports and tools. Four priorities are all directly related to enhancing the division's student information system in order to meet the data requirements outlined in ARRA. Priority 1: Develop reports and tools that analyze scores and performance for students, teachers and divisionPriority 2: Develop a tab to house teacher summative performance tool. Will be used to make teacher/student achievement correlationsPriority 3: Develop mapping for extended elements necessary to complete interface for eTranscriptsPriority 4: Develop SIF agent to implement horizontal interface and continue to progress for VDOE data exchange initiatives (linking students to teachers)

SECTION SCORE


Feasibility/Technical Merit (25%) 4.5





3.5


Strength/Weakness Summary: Good technical plan that works from an existing baseline. Strong details in the costing and functionality and also shows amounts agreed from vendor. Proposed solution can work across multiple school divisions. Budget includes consulting services, but does not include internal training, if necessary, for employees as well or additional hours for project staff. Why doesn't the vendor provide a necessary reporting upgrade as part of the ongoing maintenance contract? RECOMMENDATIONS FOR SUCCESS: Engage teachers and principals in developing the reports to ensure the types of information provided meet their needs.



PROJECT SUMMARY

15

Unify disparate sources of data through the creation of a data warehouse. Division’s goal is to build warehouse with open-source tools, which will permit customization and growth at minimal ongoing cost to the school division. Plan is to integrate data from SIS, assessments, IEP, educator assessments, MS office files and paper records. Project to be documented on a website.

Division Name: Smyth County Schools


Period of Performance: 4/2011 – 6/2013Number of Divisions 1Amount Requested $74,995.36

SECTION SCORE

ARRA Reform Areas (25%) 4






2



Strength/Weakness Summary: Solid description of current challenges and detailed description of solution attributes and how they will advance each of the 4 ARRA areas. Strong focus on support for e-transcript initiatives and provision of student data. Good set of measurements to determine the successful implementation and a detailed budget. Lacks specifics on how stakeholders will participate and how participation will be managed. No mention of how the system will improve the environment.RECOMMENDATIONS FOR SUCCESS: (1) Evaluation is tilted toward surfacing benefits of new technology, but not as assessing problems or challenges – questions should be added to surface these. (2) Days are budgeted for the development work – would want to know if contract puts risk of over-run on client or contractor – is a completed project assured within the budget?


PROJECT SUMMARY

16

Division Name: Washington CountyProject Type: Update/upgrade current system/ Create central system/consolidate data

Period of Performance: 1 YearNumber of Divisions 1 Amount Requested $53,463.77

PROJECT SUMMARY

Creation of a data warehouse to enable data management/ analysis from a central location through the purchase of hardware, software and services. The STEDfast system will allow administrators to access student and teacher data easily from a central location. Data will include student demographics, attendance, grades, graduation information, 9-week benchmark test scores, and SOL test results. Ad-hoc reporting also will be available. Future plans include adding other data sources as needed.

TIER THREE FEEDBACKSECTION SCORE







1.5


Strength/Weakness Summary Broad support for ARRA goals. Good, solid technical detail and future potential to share development findings with other divisions. Strongly written evaluation plan and budget demonstrates sound projections. Solution seems to be focused on needs of administrators - unclear to what extent other stakeholders will participate. No clear timeline to implementation. RECOMMENDATIONS FOR SUCCESS: Consider offering an out-of-the-box software solution that makes it possible to integrate information from at least the most popular student information systems on the market, benchmark testing reports and Virginia SOL test results.

TIER TWO SCORES


17

Division Name: Nelson County Public Schools

Project Type: Convert from manual system to electronic system

Period of Performance: May 2011-Dec 2011

Number of Divisions 1Amount Requested $ 75,000

Request for seed funding to purchase a web-based HR info system. System also will be used:• To connect HR info to SIS and financial system, link

student performance and for teacher and principal evaluations. Current system is paper-based and is a one-person personnel office.

• To share the single hosted instance of the standard database with CIT014, CIT021 and another division that did not submit to save costs.

SECTION SCORE







4


Strength/Weakness Summary: Sound technical approach. Specific examples of how all 4 ARRA goals will be met and clear, specific statements on impact on stakeholders. Funding is seed money, and it will be the imperative of the locality to fund the remainder Given the RFP constraint, the evaluation plan could change. Budget not detailed. Unclear if other collaborating divisions’ proposals must be successful for combined initiative to work. RECOMMENDATIONS FOR SUCCESS: Refine evaluation plan to include qualitative data on how it’s being used, satisfaction of teachers and principals, ways to improve, etc.



PROJECT SUMMARY

18

To create and implement a data system that consolidates data from multiple sources and includes comprehensive longitudinal reporting capabilities for student-level data. The data warehouse will act as a seamless extension to Interactive Achievement’s onTRAC software for districts that use that software and as a stand alone application for districts that do not. The system will enable teachers and principals to easily access and analyze student growth and achievement data over time, empowering educators to use this data for planning instruction and interventions, for measuring and improving student performance, and for assessing teacher effectiveness.

SECTION SCORE







2


Divisions Names: Consortium of Boutetort County Public Schools , Buena Vista City Schools, King George County Schools, Lancaster County Public Schools, Middlesex County Public Schools, Montgomery County Public Schools, Orange County Public Schools, Roanoke County Public Schools, Rockbridge County Schools, Surry County Public Schools, Westmoreland County Public Schools

Project Type: Update/upgrade current system/Create central system/consolidate data

Period of Performance: 07/01/11-03/01/12Number of Divisions 11Amount Requested $ 448,554Amount per Division $ 40,777

Strength/Weakness Summary : Current challenges on data fragmentation and access problems well documented. Comprehensive statements on project value to different stakeholder groups. Good development life cycle. Advisory team provides oversight and strong SDLC. No specific information on how stakeholders will be engaged, insufficient detail on how ARRA goals will be achieved and no data at the overall program level on how each goal will be met. Question on the long-term and ongoing funding of software licenses over the proposed 3-year period. RECOMMENDATIONS FOR SUCCESS: Establish system for sharing best practices in the beginning of the project. Figure out how districts not using SIF will be affected. Current focus is on the vendor to accomplish the work and ensure sustainability. What is the long term plan to build internal district capacity and/or to raise additional funds to ensure the work can continue after the 3-year contract ends?



PROJECT SUMMARY

19

Division Name: Rockingham County Public Schools


Period Of Performance: 04/2011 – 06/2012Number of Divisions 1Amount Requested $65,458.80

SECTION SCORE







3


Division already has piloted a division-written extension successfully. Will purchase server for storage and testing and software for the creation of custom SIF agents. In addition, will create an extension of the PS program to overcome its limitations and will purchase training for division technical staff to enable them to create new capabilities to extend PowerSchool. The nature of the solution will also make it adaptable to other web-based database systems, opening the door for customized extensions to otherwise closed systems.

Strength/Weakness Summary: Solid, minimal risk approach. The budget provides detailed line items costs associated and the cost for development training and education provides long-term sustainability, particularly for future enhancements. Project will meet the ARRA requirements, although the proposal team could have made how this is done clearer. Software evaluation plan is good, but the resulting system needs an evaluation product.RECOMMENDATIONS FOR SUCCESS: In substantial development projects, it is important to the continued success of the project that documentation for the effort not be treated lightly. This documentation should include a mapping between the data sources and the target repository. As mentioned in the evaluation plan, implementation of cyclical development model is good, but incorporating a more "rapid development" or agile-like (i.e. shorter cycles) development model can provide the potential for quicker stakeholder acceptance.



PROJECT SUMMARY

20

Purchase a web-based instructional system for standardized benchmark assessments, to gather reports on student achievement and data analysis. System and data will replace what’s lost when vendor went into receivership. Will gain tools to gather student data, make data driven decisions, and evaluate educator effectiveness. Product purchased is used by more than 80 VA school divisions.

Division Name: Goochland County Public Schools

Project Type: Other – Reestablish data and system after loss

Period Of Performance: Not statedNumber of Divisions: 1Amount Requested: $75,000

SECTION SCORE







2.5


Strength/Weakness Summary: Very clear understanding in this proposal of current concerns, issues, ARRA objectives and how all will be satisfied. Good detailed plan to accomplish goals with reasonable schedule planned and a detailed budget. There is a distinct lack of detail in how each stakeholder group will be engaged and no details on how to determine if the system is improving the environment. RECOMMENDATIONS FOR SUCCESS: Place greater emphasis on sharing the benchmarking results and project outcomes with other school divisions through VSUP. Consider stating why it is important to collaborate with other school divisions.



PROJECT SUMMARY

21

To enhance data systems and increase data warehousing capabilities and to provide a business intelligence tool for those who need one. Will benefit both larger and smaller divisions through the purchase of software, hardware and training.Warehouse: Development of the data warehouse and extract staging area data objects/scripts.Software: Oracle Business Intelligence Enterprise Edition software and server.Training and Collaboration: Training on OBIEE and subsequent collaboration on repository development.

SECTION SCORE







1.5


Strength/Weakness Summary: Strong sense of partnership. Good detail on problem and implementation and solid explanation of current challenges and the basis for teaming. Addresses effectiveness and efficiency to be measured and proposes ways to monitor. Specific users of the tool are not identified, nor is the value to them of using the tool. Individual stakeholder groups in each division are not identified and there is insufficient detail on the challenges of stakeholder management.RECOMMENDATIONS FOR SUCCESS: (1) Evaluation is tilted toward surfacing benefits of new technology, but not as assessing problems or challenges – questions should be added to surface these. (2) Days are budgeted for the development work – would want to know whether contract puts risk of over-run on client or contractor – is a completed project assured within the budget?

DIVISION INFORMATIONDivision Names: Alexandria City Public Schools, Augusta County Public Schools, Brunswick County Public Schools, Campbell County Public Schools, Charlotte County Public Schools, Chesapeake Public Schools, Fauquier County Public Schools, Colonial Heights City Public Schools, Greensville County Public Schools, Hampton City Public Schools, Louisa County Public Schools, Mecklenburg County Public Schools, Norfolk Public Schools, Petersburg City Public Schools, Powhatan County Public Schools, Prince George County Public Schools, Richmond City Public Schools, Williamsburg-James City County Public SchoolsProject Type: Update/upgrade current system/ Create central system/ consolidate data Period of Performance: 4/2011 - 6/2013 Number of Divisions 18Amount Requested $771,489Amount per Division $ 42,860


PROJECT SUMMARY

22

Charlottesville City SchoolsLDS Grant Proposal

Anywhere, Anytime, Over Time: Longitudinal Data to the Teacher Desktop

Award: $40,317

23

LDS Grant Criteria

• Linking student data with teachers responsible for their instruction*

• Reporting student-level transcript data (course enrollment and outcomes)

• Providing data on teacher and principal evaluation systems*

• Providing teachers and principals with information on student growth and achievement*

24

Problem Description

• Multiple SaaS assessment vendors: MAP/NWEA, IA, SOL, CogAT, PALS, AIMSweb, Achieve 3000, local assessments, etc.

• Stakeholders wanted a way to view assessment data using one interface

• FERPA regulations require a valid “educational interest”

• Limited IT staff to develop an in-house solution

25

Proposal for Pearson Inform

• Seamless Interface with PowerSchool, our SIS• Automatically synchronizes both demographics

and class assignments• Adheres to FERPA regulations • Common interface from anywhere• Assessment analysis by teacher at the time of

the test or by current teacher• Allows AIP and Remediation Groups

26

Demonstration

• Disclaimer: All student names have been randomized!

• Live DEMO of Pearson Inform was performed here

• The following slides are sample screen shots from Pearson Inform

29

Teacher 1 Teacher 2 Teacher 3

32

Questions?

Contact Information

Bethann Canada [email protected]

David [email protected]

mailto:[email protected]

mailto:[email protected]

anywhere, anytime, over time: longitudinal data to the teacher desktop

Documents

budget score

arra score

stakeholder score

evaluation score

feasibilitytechnical

collaborationreusability

arra education reform

manual system