Developing Data Manuals and Reports

Purpose

Proposed data manuals are explicit in their connection to Board mandates, which include existing Board expectations (program standards), federal rules, and / or state law as passed by the Legislature. Data Manuals enumerate the data elements that preparation programs collect each year and serve as the foundation for the Data Project

Data Manuals (DM) define statewide data collections, i.e. a data manual commits all PESB-approved educator preparation programs to the collection. Each Data Manual describes the universe, cardinality, elements, and valid values for the collection. The data is collected at the individual candidate level and held by programs in their data systems. Once the elements are defined, the Data Group, which includes the Data Stewards from each program, determines how the elements will be counted. All completed data manuals brought to the Board for adoption will have met the expectations (below) for creating a data manual for implementation.

Process for Development

Proposed data manuals are brought to the Board at its May meeting. Staff work with content experts, the Data Group, and the Metrics Group, when requested, to create a data manual that meets the implementation requirements through vetting with the Data Group. The developed data manuals are brought to the Board for adoption the following May and include the elements contained in the proposed version from the previous May, along with any supplemental elements proposed by the Metrics Group and that meet the data element requirements of the Data Group. All elements proposed by the Metrics Group will have been collectively agreed upon by the professional association’s process and have the commitment of all educator preparation programs to be collected by them.

The Data Manuals are intended to remain relatively static from year to year. Updates to the data manuals are driven by a Board policy or oversight need. Based on the type of change to be made, substantive changes revert to a content expert group, and technical changes are referred to the Data Group, or a subset, for resolution. Modifications to a Data Manual that are beyond technical fixes, such as adding or removing an element, are at the direction of the Board. The Board retains authority to direct staff to create or change data manuals based on the mandates or, in the case of collective action by the preparation programs, from recommendation by the Metrics Group to the Board for inclusion of data elements in a statewide data collection.

Educator preparation programs will have the opportunity to review developed data manuals prior to the Board’s decision on whether to adopt the data manual. Board deliberation on the approval of the collection and the data manual will be open to public comment. General public comment is available at all other Board meetings, and written comment can be submitted to the Board at any time through the regular commenting process. Programs are invited to individually or collectively offer input into the cycle as provided by the PESB policy feedback protocols published here.

Expectations

Data elements in each DM will comport with the following requirements:

Operational

    • Include a unique code, definition, example, and list of valid values (where appropriate).
    • Prevent creating silos of information within programs.
    • Be granular enough to aggregate into multiple reports.
    • Have potential data collisions identified and addressed.

Measurable

    • Captures what actually happens rather than what is intended to happen.
    • Includes a timestamp that identifies when the element occurred.
    • Reflects variance beyond binary values.
    • Identifies relative rather than categorical values.
The Data Group will determine whether a data manual meets these expectations.

The Annual Reports home page links to examples of Data Manuals.

Report Guidance

Each year a Report Guidance document is developed by PESB staff. This document identifies the data to be reported to the Board.  The content is limited to elements defined in an approved Data Manual, as authorized by the Board, or available from PESB with proper linking keys. While the Data Manual defines the information programs collect on their candidates, the Report Guidance defines the aggregations of that candidate level data that are to be submitted to the PESB. These aggregations may include joining elements from multiple Data Manuals as well as joining outside data (e.g. OSPI S-275) to the Data Manual elements. The Report Guidance defines the data universe, cardinality, elements, and valid values for the reports as well as the due dates for those submissions.

The Annual Reports home page links to examples of Report Guidance documents.

Process Benefits and Challenges

Programs report specific data annually to the PESB. While all of the report contents derive from the data manuals, not all of the report information is used only by PESB. Some of the data is batch uploaded for federal reports, such as the HEA Title II teacher preparation program report. This practice provides efficiency for programs by reporting similar data a single time to a single body and allows them to retain responsibility for holding the data of record in their files. Common definitions across reports, resulting from the work of the Data Group and content experts reduce confusion and create consistency across all reports, state and federal.

This data collection and reporting process has challenged program level data system capacity. All recognize the benefits of commonly defined data efficiently collected and securely held, but designing systems to achieve these benefits, including connecting between the program and larger university, or even within the programs themselves, has proven more challenging for some than others. As the regulating body that defines the requirements for program to collect and hold specific data for reporting purposes, PESB has adjusted its resources to provide staff technical assistance. In addition, resources have been made available, as well, to help programs build data systems capacity through identification and support of technical assistance provided by peers.