NYC Comptroller John Liu: High School Progress Reports Don’t Measure Progress

******************************************

Audit Finds that Department of Education’s Revisions to its High School

Grading System Leave Educators, Students Chasing a Moving Target

******************************************

New York City Comptroller John C. Liu announced that an audit of the Department of Education’s (DOE) High School Progress reports raised questions about the usefulness of the reports in comparing the yearly progress of schools.

“It’s troubling that a system that is used to decide school closings leaves teachers and students confused about what they need to do to improve,” Comptroller Liu said. “The Department of Education should not leave parents, educators or students in the dark when it’s deciding their fates.”

High School Progress Reports are a DOE accountability tool that assigns schools an annual grade of A through F. The Report grades play a significant role in the DOE’s decisions to reward high performing schools, perhaps with added funds, and restructure or close low-performing schools.

According to the audit, the DOE has revised the complex formula behind the grades every year. The frequent changes the agency has made to its grading and other formulas — without determining the impact of those changes — makes it difficult, if not impossible, to get a true picture of a school’s progress by comparing its grade from one year to the next. As a result, the High School Progress Reports paint an unreliable and confusing picture of a school’s progress or failure over time. Auditors recorded complaints from schools that the DOE’s lack of consistency made it difficult to set goals for students.

The audit focused on 10 high schools representing the five boroughs. It included three schools (Jamaica, Metropolitan Corporate Academy, and Norman Thomas high schools) that the DOE selected in January 2010 for closing.

CHIEF AMONG THE FINDINGS:

1. Inaccurate Picture of Year-to-Year Progress

The DOE’s changes to the formula behind the Progress Report grades make it difficult for parents and educators to measure a school’s performance from one year to the next.

The DOE says the Progress Report grades are meant as “a one-year snapshot” comparing one school against another in a given year, and not as a measure an individual school’s progress over time. Yet, the agency itself uses the formula to track achievement from one year to the next. For example, a school that receives a “C” three years in a row may be targeted for corrective action.

Example:

One school, Metropolitan Corporate Academy High School, which is set to close in 2014, improved its score every year from 2006 to 2010, but because of the DOE’s changing formula Metropolitan never rose above a “C.” In fact, the school fell to a “D” in the 2008-2009 school year even though its numeric score would have earned it a “B” under the 2006-2007 grading formula. It is impossible to tell to what extent Metropolitan’s scores reflect changes in its own performance or DOE’s changes to the grading formula. (See Table)

“The Comptroller’s audit of the High School Progress Reports demonstrates the difficulty of comparing a school’s letter grade over time when a school’s peer group composition and the cut scores for the grades change from one year to the next,” said Professor Aaron Pallas of Teachers College, Columbia University. “The recommendation that the DOE report high school progress report grades using both the old and new criteria would enable stakeholders both inside and outside of the schools to understand trends in school performance more completely.”

Action:

Since the audit, the DOE has posted an advisory on its website regarding year-to-year comparisons of High School Progress Report grades.

2.Lack of Communication

The audit determined that, while DOE met with school principals and others about changes, auditors found no evidence that it actually integrated feedback from them into the Progress Report. In fact, some educators told auditors that they felt as if they were chasing a moving target as they attempted to understand the changes that the DOE made to the grading formula each school year and to prepare students. The audit also found that the DOE did not do enough to inform schools what effect the changes to the grading system were expected to have on Progress Report grades.

“Taken alone, Progress Reports are an unreliable index for determining school closures or related high-stakes decisions,” said Professor David C. Bloomfield, chairman of the education department at the College of Staten Island. “Greater feedback by stakeholders, as recommended by the audit, might help to improve Progress Reports’ utility in this process.”

Action:

The DOE has since published materials summarizing and responding to feedback from educators and others involved in the 2010-2011 review process.

3. Data Reliability

The audit found that the data — student grades, Regents exam scores, and other information — that the DOE used to calculate each year’s Progress Report grades was representative of student data recorded in the DOE’s computer systems and verifiable. However, while the data in a given year was accurately recorded, it was not useful as a measure of an individual school’s progress over time.

RECOMMENDATIONS:

The DOE generally agreed with nine of the audit’s 10 recommendations and has begun to implement a number of them. However, the audit notes that “DOE inappropriately misinterpreted and even exaggerated, many of the audits ‘positive’ conclusions as an endorsement for the progress reports,” while simultaneously discounting its weaknesses.

SCOPE:

The audit was launched in March 2010 using data from the high schools’ 2008-2009 progress reports, the most recent data available at that time. The audit was expanded to include progress reports for the 2006-2007, 2007-2008, and 2009-2010 school years. In addition, auditors interviewed staff at the high schools in April and May 2010.

BACKGROUND:

High School Progress Reports are a DOE accountability tool that assigns schools an annual grade (A through F) based on a variety of factors, including student performance, student progress, school environment, and comparisons between schools with similar populations. The letter grades were introduced in the 2006-2007 school year.

Comptroller Liu credited Deputy Comptroller for Audit H. Tina Kim and the Audit Bureau for presenting the findings. The full report is available at http://comptroller.nyc.gov/audits.

NYS Comptroller Tom DiNapoli Audit Identifies Discrepancies in Dropout Rate Reported by NYC Department of Education

The dropout rate among New York City public school students is higher than claims made by the city Department of Education (DoE), according to an audit released by State Comptroller Thomas P. DiNapoli. DiNapoli’s auditors found that for the 2004 through 2008 school years, the dropout rate may have been as high as 16.5 percent, rather than the 13 percent cited by DoE. As a result, the graduation rate may have been as low as 62.9 percent, rather than the 65.5 percent reported by DoE.

“The city school system needs to sharpen its pencils when it comes to knowing which kids are dropping out and which kids are transferring to another school,” said DiNapoli. “DoE should be doing its homework and making sure the right papers are turned in to back up the reasons why students are leaving school.”

High school graduation and dropout rates are regarded as important indicators of a school’s effectiveness.


While the audit considered reported rates within 5 percent of audited rates to be generally accurate, the difference means that the graduation rate and discharge rate include thousands of students who actually dropped out.

DiNapoli’s auditors attribute the discrepancy to DoE’s erroneous classification of dropout students as having been “discharged” from high school. Discharged students should only be categorized as such when they transfer to another school or another educational program, leave the country, or are deceased.

DiNapoli’s auditors examined DoE’s discharge records for its 2004-08 general education cohort (the group of students who entered ninth grade in 2004 and were expected to graduate four years later), and found that in a random sample of 500 “discharged” students, 74 (14.8 percent) didn’t have the required documentation. As a result, all 74 should have been classified as dropouts.

Projecting the results of the sample to the entire cohort, DiNapoli’s auditors found that the correct graduation rate for the cohort was between 62.9 and 63.6 percent, rather than the 65.5 percent reported by DoE, and the correct dropout rate was between 15.5 and 16.5 percent, rather than the 13.0 percent reported by DoE. At some individual high schools, the correct graduation rates could be lower, and the correct dropout rates higher, than DoE reported.

According to DoE, the city’s 2004-08 general education cohort had a total of 88,612 students, of whom 46,896 graduated, 15,368 were still enrolled after four years, 17,025 were discharged, and 9,323 dropped out.

DiNapoli’s auditors also examined DoE’s discharge classifications for its 2004-08 special education cohort and identified similar errors. Auditors estimated that the correct graduation rate for this cohort was between 8.9 and 9.3 percent, rather than the 9.7 percent reported by DoE, and the correct dropout rate was between 20.6 and 23.8 percent, rather than the 17.2 percent reported by DoE. Even with the identified error rate, the NYC graduation rate is trending upwards as reported by DoE.

DiNapoli recommended that DoE officials:

• Ensure that DoE discharge guidelines fully align with New York State Education Department (SED) regulations;
• Instruct all schools to adhere to the SED regulations for discharge classifications, and provide training in the regulations for school staff who administer discharges; and
• Conduct periodic reviews of discharge classifications to determine whether they are being made and documented in accordance with SED regulations.

DoE officials generally agreed with DiNapoli’s recommendations and indicated they have taken action or will be taking action to implement them. Most notably, DoE’s guidelines were amended before the 2009-10 school year to better align with SED’s guidelines on required documentation to support a discharge classification. Click here for a copy of the report.

Progress Report: The MTA Capital Security Program - New York State Comptroller Tom DiNapoli

Improvements Made Despite Delays and Cost Overruns

Despite significant delays and unplanned costs, the MTA has greatly improved its security through capital improvements, operational initiatives and enhanced cooperation with other security agencies, according to a report released today by New York State Comptroller Thomas P. DiNapoli.

“The capital security program the MTA has implemented since 9-11 has made New Yorkers more secure,” said DiNapoli. “The MTA has made progress, particularly in the last two years. But the mass transit system is still inherently vulnerable. Individual projects in this program are months if not years behind schedule and well over budget, and additional capital improvements are needed. My office will continue to track MTA management of this program.”

The projects in Phase 1 of the MTA’s capital security program target the system’s most vulnerable and heavily used assets, including stations, transit hubs, bridges and tunnels. Each project involves one or more facilities and security improvements to elements such as electronic security and surveillance, fire, life and safety and evacuation enhancements, perimeter protection and structural hardening. This phase, originally scheduled for completion by September 2008, will not be completed until June 2012.

After more than nine years, the MTA has completed 11 of the original 16 security projects as well as elements of the five remaining projects. The MTA has hardened all 14 facilities planned for Phase 1; improved lighting, communication systems, and smoke and fire detection equipment in 15 facilities; installed perimeter protection around four facilities; and despite significant setbacks, the electronic security program.

As of December 2010, the MTA had completed 31 of 38 planned construction tasks and the remaining seven tasks were all in the process of construction, though more than 60 percent of the 38 tasks were behind their established schedules, including 11 that were behind by more than one year (five tasks were more than 30 months late). The cost of Phase 1 (including two facilities that were deferred from Phase 1 to Phase 2) has grown from $591 million to $851 million, an increase of 44 percent.

Electronic security projects in particular have encountered significant problems. The cost of this program has nearly doubled, growing from $265 million to $515 million. The MTA has continued work on the electronic security program by hiring other contractors, but full implementation is not scheduled to be completed until June 2011. A lawsuit between the MTA and Lockheed Martin, the original contractor responsible for the electronic security program, is still ongoing.

The MTA has also implemented a number of operational initiatives that have improved the system’s safety: security personnel have been increased, coordination with other security agencies has been enhanced and the MTA has implemented a public relations campaign to encourage the public to report suspicious activity.

The Comptroller noted that additional capital improvements are needed to improve overall security. Phase 2 will fund some of the remaining 33 security projects of the original 57 that were identified through a risk-based assessment. However, the MTA will not begin work on the remaining 16 projects until the scope is determined and funding becomes available.

Comptroller DiNapoli maintains a comprehensive approach to oversight of the MTA. Since 2007, DiNapoli has issued 27 reports on the MTA, including 14 audit reports and 13 reports examining MTA finances and operations. There are currently eight audits underway in various stages of completion, including a forensic audit of MTA use of overtime and one audit that is being performed jointly with the New York City Comptroller’s Office.

Click here for the report: http://www.osc.state.ny.us/press/releases/mar11/MTASecurityRpt12-2011.pdf

###

Follow NYS Comptroller on Twitter: @NYSComptroller

New York State Comptroller Tom DiNapoli: MTA Bus Fleet Maintenance Too Expensive

Nearly half of the Metropolitan Transportation Authority (MTA) bus fleet—including virtually all of the fleet’s hybrid-engine buses—have not been properly inspected according to an audit released today by New York State Comptroller Thomas P. DiNapoli. In addition, 62 percent of the MTA’s buses failed to meet reliability goals despite maintenance costs that topped $777.7 million in 2008. That amount was double the maintenance costs of other comparable transportation agencies around the nation.

“New Yorkers aren’t getting what they pay for when it comes to bus service,” said DiNapoli. “Other cities across the nation spend much less on maintenance and get better results. The MTA needs to step up bus maintenance performance and bring down maintenance costs.”

Three of the MTA’s seven constituent agencies provide bus service in New York City and Long Island. The MTA’s Regional Bus Operations oversees the authority’s 6,200-bus fleet and maintenance services at 29 depots and two overhaul facilities. DiNapoli’s audit examined records from the MTA’s Regional Bus Operations division between 2007 and 2009.

Auditors found that:

• Nearly two-thirds of the 29 bus depots did not meet their performance goals;
• Maintenance costs per mile of operation were much higher than other bus fleets around the nation;
• 584 of the MTA’s 1,255 required maintenance inspections were performed poorly, or not at all; and;
• Mechanical failures were more frequent than expected. One depot had a goal of 4,674 miles between failures, but its actual distance traveled between failures was 3,581 miles.

DiNapoli’s auditors recommended the MTA:

• Improve the reliability of its bus fleet;
• Identify reasons why maintenance costs are so high and develop a plan to reduce them, and;
• Prepare a comprehensive maintenance plan that includes information on maintenance program objectives and unscheduled maintenance operations.

MTA officials agreed with DiNapoli’s recommendations. Click here to read the full audit, or visit www.osc.state.ny.us.

New York State Comptroller Tom DiNapoli Releases Audits

New York State Comptroller Thomas P. DiNapoli announced today the following audits have been issued:

New York City Department of Education, Non-Competitively Awarded Contracts (Follow-Up) (2010-F-26)
The New York City Department of Education must comply with applicable procurement requirements when awarding non-competitively bid contracts. In audit report 2008-N-1, DiNapoli’s auditors examined whether the department complied with requirements and found that it didn’t. When auditors followed up, they found that department officials had made significant progress in correcting the problems identified in the initial report.

Department of Health, Inappropriate Medicaid Payments for Dental Services Provided to Patients With Dentures (Follow-Up) (2010-F-33)
The Department of Health administers the Medicaid program. In audit 2008-S-125, DiNapoli’s auditors reviewed claims for a five-year period and found that dentists may have been overpaid as much as $2.9 million for services provided to patients with dentures. Auditors determined that the Medicaid claims processing system, eMedNY, lacked the controls necessary to detect and prevent such overpayments. Auditors recommended that appropriate controls be developed and implemented. When they followed up, auditors determined that department officials made some progress in correcting the problems they identified but improvements were still needed.

City University of New York, Kingsborough Community College: Selected Financial Management Practices (Follow-Up) (2010-F-39)
The City University of New York consists of eleven senior colleges, six community colleges, and several other specialized and professional schools. In audit report 2008-N-9, DiNapoli’s auditors examined whether one of the community colleges (Kingsborough) complied with requirements relating to payroll, procurement and waiving of tuition, and found certain instances of non-compliance in all three areas. When auditors followed-up, they found that officials had made progress in correcting the problems they identified.

State Education Department, St. Francis de Sales School for the Deaf: Selected Financial Management Practices (Follow-Up) (2010-F-40)
St. Francis de Sales School for the Deaf is one of 11 private schools that receive operating aid directly from the state to provide educational services to disabled students. In audit report 2008-S-160, auditors identified a number of internal control weaknesses in the areas of procurement, cash disbursements and payroll. Auditors also found that the school’s board of trustees was not providing effective oversight of financial operations. When auditors followed up, they found that officials had made significant progress in correcting the problems identified, but improvements were still needed.

Office of Parks, Recreation and Historic Preservation, Natural Heritage Trust, Sources of Trust Revenues (2009-S-11)
The Natural Heritage Trust maintains over 200 custodial accounts for account holders, including the Office of Parks, Recreation and Historic Preservation. DiNapoli’s auditors examined whether Office of Parks revenue was incorrectly deposited into the custodial accounts of the trust. They found that more than $3.5 million in revenue was deposited into trust accounts incorrectly. Auditors recommended the Office of Parks recover the funds and improve controls over revenue.

Metropolitan Transportation Authority, Minority and Women’s Business Enterprise Reporting (2010-S-9)
State agencies and public authorities are required to promote the participation of minority-owned business enterprises (MBEs) and women-owned business enterprises (WBEs) in state contracts and procurement opportunities. Specifically, agencies must establish annual goals for such participation and make a “good faith” effort to achieve these goals. DiNapoli’s auditors examined the Metropolitan Transportation Authority’s performance and found that it was establishing annual goals and making a good faith effort to reach them. However, the MTA’s goals were unreasonable, and as a result, they have consistently fallen short of their goals. Auditors also found that the MTA was not accurately reporting results of its efforts and was overstating its procurements. Auditors recommended improvements to comply with contract requirements.