Scores of both black and Hispanic students on some state and national tests have gone up since Mayor Bloomberg took over the Department of Education in 2002, and by some measures the "proficiency gap" between black and Hispanic students and white and Asian ones has begun to close.
But by one other measure — not the test scores of black and Hispanic students alone or the percentage of them that met the bar called "proficiency," but a more subtle and relative measure known as the "achievement gap" — progress has been more elusive.
RELATED: Aaron Pallas's Achievement Gap Analysis | Department of Education Achievement Gap Analysis.
Three researchers studied that measure at the request of The New York Sun by analyzing detailed data the city Department of Education previously had not released. They found that the actual gap between different racial groups' test scores has not budged by most measures, and in some cases it has widened.
In the most encouraging case — the difference between black and white students' scores on an eighth-grade English test -- the gap has narrowed slightly. Yet it remains wide. In 2008, 74% of black eighth-graders in the city scored below the average white eighth-grader on the state English test, compared to 79% in 2002.
How much this achievement gap matters depends upon whom you talk to.
In an interview at Tweed Courthouse, the schools chancellor, Joel Klein, said the achievement gap is "an issue," but he said it should not obscure the significant gains black and Hispanic students have made under his watch.
Though in some cases the achievement gap is steady because no students have made gains on standardized tests, in other cases it is staying constant because scores have risen for all students — white, black, Hispanic, and Asian.
"If the way to close the achievement gap is to pull whites down, that's not a strategy that any intelligent person or any responsible school district would ever follow," Mr. Klein said.
Others say the relative differences between races the achievement gap exposes are important.
"In the real world, no one looks at proficiency scores to determine the likelihood of someone getting ahead," one of the researchers that analyzed the data for the Sun, Aaron Pallas, a professor of education and sociology at Columbia University's Teachers College, said. "What really matters is how people do in relation to one another. And so if you have groups remaining the same distance apart, then the group ahead is still going to remain ahead when you look at selection into colleges and the labor market."
Messrs. Bloomberg and Klein have previously said that New York City's achievement gap is narrowing.
"Over the past six years, we've done everything possible to narrow the achievement gap — and we have," Mr. Bloomberg said in testimony to Congress this summer. "In some cases, we've reduced it by half."
Their statements draw on two measures of progress. The first is actual improvements made by black and Hispanic students on standardized tests.
At some grade levels, more black and Hispanic students are passing both federal and state tests than ever before.
The percentage of black fourth-graders scoring above the "basic" level on a federal math test soared to 72% in 2007 from 58% in 2003.
In other grade levels -- notably the eighth grade -- there has been little if any change in the number of black and Hispanic students attaining proficiency on both state and federal tests, and in some cases the number of black and Hispanic students passing tests has declined.
The second measure is the percentage of students in different racial groups that meet the bar known as "proficiency." The gap between the percentage of, say, black students and white students who pass that bar is what Messrs. Bloomberg and Klein have been referring to as the achievement gap.
That measure has indeed been sliced in half in some cases. For instance, in 2002 the black-white gap on a state fourth-grade math test was 35 percentage points, with 76% of white students scoring "proficient" compared to 41% of black fourth-graders. By 2008, the gap had narrowed to 18 points: 91% of white students now score proficient compared to 73% of black students.
The proficiency gap is closing even as the achievement gap stays essentially the same because each gap represents a different kind of improvement. Proficiency rates detect movements across the proficiency bar, rising when students who had been below it learn enough knowledge and skills to reach the standard, but registering no change if students who were already meeting the standard surge even further above it. The achievement gap, on the other hand, is sensitive to changes both above and below the proficiency bar.
Black and Hispanic students are more likely to start out with scores below the "proficient" mark. That means that the proficiency gap can close even though all groups of students might be rising by equal amounts.
One of the three researchers who analyzed the achievement gap data for the Sun, Howard Everson, said that he puts more stock in proficiency scores, which are also the measure the federal government's No Child Left Behind law uses to judge schools and school districts.
Mr. Everson, a psychometrician at Fordham University, is the lead testing adviser to the New York State Education Department and an adviser to the federal government on its testing regime.
He said the proficiency figures are important because they contain not only information about test scores but also a professional judgment on whether a student has adequate skills and knowledge. Expert math and English teachers set "proficiency" bars, and Mr. Everson said he takes their judgments seriously.
On those grounds he said, "I think overall it looks to be pretty good news."
The third researcher who analyzed the data for the Sun, Robert Tobias, a New York University professor who was the city's testing director for 13 years, said he puts more stock in the achievement gap than in the proficiency gap because it reflects changes in students across the spectrum -- not just those who moved above or below the proficiency bar.
Mr. Tobias said the achievement gap's changes contradict claims the Bloomberg administration has made of impressive progress, which he said he has read in press releases and newspaper articles. "When one looks at this presentation, the picture is a lot more modest," he said.
The three researchers found that the gap between black and white students' average scores on an English test has closed slightly, as has the gap on that test between Hispanics and whites, and that no gap has closed on a mathematics test. It also found that the gap between Asian students and their black and Hispanic peers has widened slightly since 2002.
In some cases the gap remains wide because black and Hispanic students are making no progress and neither are white and Asian students. In other cases, black and Hispanic students are gaining some ground -- but white and Asian students' scores are improving at roughly the same pace.
The Department of Education also conducted an analysis of the achievement gap, reproducing the same process the outside researchers conducted. Pointing to small changes in figures, the department's analysis concluded that the gap has closed across the board even by this measure. "We are closing the achievement gap," the analysis said.
The researchers challenged that conclusion.
"This is not strong evidence that the gap is closing," Mr. Everson said. "The only thing you can say is that they're relatively flat, that the gap is relatively stable."
The achievement gap can also be examined by looking at New York City students' scores on a federal test, the National Assessment of Educational Progress, which is known as the nation's report card.
An analysis by the National Center for Education Statistics, the research arm of the federal Education Department, concludes that no achievement gaps have narrowed at all in New York City between 2003 and 2007. The only gap that moved in any significant direction is the one between poor students and the rest of the population, which widened slightly, that analysis said.
The National Center for Education Statistics also concludes that upward trends in the reading scores of black and Hispanic fourth-graders lauded by Mr. Klein are not statistically significant.
Mr. Klein criticized the National Center on Education Statistics analysis.
"Those are just confidence levels. Nobody is saying this is a science," Mr. Klein said. He added: "If three points is flat, and four points is statistically significant, then what you're doing is, you're playing something of a game."
Mr. Klein said he stands by his positive stance on New York City's success at closing the achievement gap.
"My view is that our black kids in the fourth grade are outperforming all black kids in America," Mr. Klein said. "Our kids are ripping the leather off the ball of the other kids around the country."
Others drew a more discouraging conclusion.
Mr. Pallas said the results should lead people around the country to question whether they should reproduce the policies being enacted in New York City.
"We need to be aware that what we're doing right now to close the achievement gap may not be working," Mr. Pallas said. "If what we're doing isn't working, we need to be aware of that and perhaps think about doing something else."
Lawrence Feinberg, the assistant director for reporting and analysis at the National Assessment Governing Board, the group that oversees the federal test, wrote in a memo last year that Mr. Klein's conclusions about progress by black students are "incomplete." The conclusions "may be questioned by his critics" because they depend on trends in test scores and do not take into account whether increases are statistically significant, the memo said.
The interim executive director of NAGB, Mary Crovo, to whom the memo was addressed, said in a telephone call to the Sun that the memo was an internal staff memo that was not meant to reflect the official position of the governing board. Asked whether she supported the memo's argument, she said, "I think the data in the memo are accurate, and that's as much as we can say."
A spokesman for the city's education department, David Cantor, called the memo "a politicized gloss" and said the outgoing executive director of NAGB, Charles Smith, had called Mr. Klein to apologize.
