NYC Comptroller John C, Liu: High School Progress Reports Don't Measure Progress

Audit Finds that Department of Education’s Revisions to its High School

Grading System Leave Educators, Students Chasing a Moving Target

******************************************

New York City Comptroller John C. Liu announced that an audit of the Department of Education’s (DOE) High School Progress reports raised questions about the usefulness of the reports in comparing the yearly progress of schools.

“It’s troubling that a system that is used to decide school closings leaves teachers and students confused about what they need to do to improve,” Comptroller Liu said. “The Department of Education should not leave parents, educators or students in the dark when it’s deciding their fates.”

High School Progress Reports are a DOE accountability tool that assigns schools an annual grade of A through F. The Report grades play a significant role in the DOE’s decisions to reward high performing schools, perhaps with added funds, and restructure or close low-performing schools.

According to the audit, the DOE has revised the complex formula behind the grades every year. The frequent changes the agency has made to its grading and other formulas — without determining the impact of those changes — makes it difficult, if not impossible, to get a true picture of a school’s progress by comparing its grade from one year to the next. As a result, the High School Progress Reports paint an unreliable and confusing picture of a school’s progress or failure over time. Auditors recorded complaints from schools that the DOE’s lack of consistency made it difficult to set goals for students.

The audit focused on 10 high schools representing the five boroughs. It included three schools (Jamaica, Metropolitan Corporate Academy, and Norman Thomas high schools) that the DOE selected in January 2010 for closing.

CHIEF AMONG THE FINDINGS:

1. Inaccurate Picture of Year-to-Year Progress

The DOE’s changes to the formula behind the Progress Report grades make it difficult for parents and educators to measure a school’s performance from one year to the next.

The DOE says the Progress Report grades are meant as “a one-year snapshot” comparing one school against another in a given year, and not as a measure an individual school’s progress over time. Yet, the agency itself uses the formula to track achievement from one year to the next. For example, a school that receives a “C” three years in a row may be targeted for corrective action.

Example:

One school, Metropolitan Corporate Academy High School, which is set to close in 2014, improved its score every year from 2006 to 2010, but because of the DOE’s changing formula Metropolitan never rose above a “C.” In fact, the school fell to a “D” in the 2008-2009 school year even though its numeric score would have earned it a “B” under the 2006-2007 grading formula. It is impossible to tell to what extent Metropolitan’s scores reflect changes in its own performance or DOE’s changes to the grading formula. (See Table)

“The Comptroller’s audit of the High School Progress Reports demonstrates the difficulty of comparing a school’s letter grade over time when a school’s peer group composition and the cut scores for the grades change from one year to the next,” said Professor Aaron Pallas of Teachers College, Columbia University. “The recommendation that the DOE report high school progress report grades using both the old and new criteria would enable stakeholders both inside and outside of the schools to understand trends in school performance more completely.”

Action:

Since the audit, the DOE has posted an advisory on its website regarding year-to-year comparisons of High School Progress Report grades.

2. Lack of Communication

The audit determined that, while DOE met with school principals and others about changes, auditors found no evidence that it actually integrated feedback from them into the Progress Report. In fact, some educators told auditors that they felt as if they were chasing a moving target as they attempted to understand the changes that the DOE made to the grading formula each school year and to prepare students. The audit also found that the DOE did not do enough to inform schools what effect the changes to the grading system were expected to have on Progress Report grades.

“Taken alone, Progress Reports are an unreliable index for determining school closures or related high-stakes decisions,” said Professor David C. Bloomfield, chairman of the education department at the College of Staten Island. “Greater feedback by stakeholders, as recommended by the audit, might help to improve Progress Reports’ utility in this process.”

Action:

The DOE has since published materials summarizing and responding to feedback from educators and others involved in the 2010-2011 review process.

3. Data Reliability

The audit found that the data — student grades, Regents exam scores, and other information — that the DOE used to calculate each year’s Progress Report grades was representative of student data recorded in the DOE’s computer systems and verifiable. However, while the data in a given year was accurately recorded, it was not useful as a measure of an individual school’s progress over time.

RECOMMENDATIONS:

The DOE generally agreed with nine of the audit’s 10 recommendations and has begun to implement a number of them. However, the audit notes that “DOE inappropriately misinterpreted and even exaggerated, many of the audits ‘positive’ conclusions as an endorsement for the progress reports,” while simultaneously discounting its weaknesses.

SCOPE:

The audit was launched in March 2010 using data from the high schools’ 2008-2009 progress reports, the most recent data available at that time. The audit was expanded to include progress reports for the 2006-2007, 2007-2008, and 2009-2010 school years. In addition, auditors interviewed staff at the high schools in April and May 2010.

High School	Borough	2006-2007 Score/Grade	2007-2008 Score/Grade	2008-2009	2009-2010
ACORN Community HS	Brooklyn	63.3/B	64.4/A	65.4/B	71.7/A
Baruch College Campus HS	Manhattan	81.2/A	82.6/A	83.6/A	74.5/A
Curtis HS	Staten I.	64.2/B	59.4/B	68.8/B	69.8/B
DeWitt Clinton HS	Bronx	47.7/C	42.7/C	49.7/C	48.1/C
Flushing International HS	Queens	n/a	73.0/A	70.9/A	64.5/B
Jamaica HS	Queens	44/C	37.3/C	41.7/D	45/D
Metropolitan Corporate Academy HS	Brooklyn	35.9/C	39.1/C	43/1/D	48/C
New World HS	Bronx	n/a	97/A	92.1/A	85.6/A
Norman Thomas HS	Manhattan	33.4/D	29.7/D	36/D	36/F
Ralph R. McKee Career and Technical Education HS	Staten I.	63.8/B	67.6/A	79.5/A	76/A

BACKGROUND:

High School Progress Reports are a DOE accountability tool that assigns schools an annual grade (A through F) based on a variety of factors, including student performance, student progress, school environment, and comparisons between schools with similar populations. The letter grades were introduced in the 2006-2007 school year.

Comptroller Liu credited Deputy Comptroller for Audit H. Tina Kim and the Audit Bureau for presenting the findings. The full report is available at http://comptroller.nyc.gov/audits.