By Christopher Connell, The Hechinger Report, 9/21/10
NASHVILLE — Offering middle-school math teachers bonuses up to $15,000 did not produce gains in student test scores, Vanderbilt University researchers reported Tuesday in what they said was the first scientifically rigorous test of merit pay.
The results (pdf) could amount to a cautionary flag about paying teachers for the performance of their students, a reform strategy the Obama administration and many states and school districts have favored despite lukewarm support or outright opposition from teachers' unions.
The U.S. Department of Education has put a great deal of effort into prodding school districts and states to try merit-pay systems as part of its Race to the Top competition, although teachers' unions have often objected on the grounds that they don't have fair and reliable ways to measure performance. In most school districts, teacher pay is based on years of experience and educational attainment levels.
The report's authors, of the National Center on Performance Incentives (NCPI) at Vanderbilt University's Peabody College of Education, stress that theirs is just one approach. The Nashville teachers who hit the mark based on their students' test scores received a bump in their paychecks but no additional mentoring or professional development. Neither their principals nor fellow teachers knew who participated in the experiment or who received bonuses.
Matthew G. Springer, director of the federally funded NCPI, said pay-for-performance is not "the magic bullet that so often the policy world is looking for."
At least in this experiment, Springer said, "it doesn't work."
The study was conducted from 2006 to 2009 in partnership with the nonprofit RAND Corporation. A local industrialist and Vanderbilt benefactor, Orrin Ingram, put up the nearly $1.3 million in bonuses.
Some 296 middle-school math teachers — two-thirds of the district's middle-school math teachers — volunteered to participate in the experiment. Half were placed randomly in a control group, while the rest were eligible for bonuses of $5,000, $10,000 or $15,000 if their pupils scored significantly higher than expected on the statewide exam known as the Tennessee Comprehensive Assessment Program.
One third of the eligible teachers — 51 of 152, or 34% — got bonuses at least once. Eighteen teachers received bonuses all three years.
Except for some temporary gains for fifth-graders, though, their students progressed no faster than those in classes taught by the 146 other teachers.
The local teachers' union in Nashville agreed to the experiment in collective bargaining, according to Erick Huth, president of the Metropolitan Nashville Education Association. He said the results were not at all surprising.
"I've believed for a long time that what improves instruction is having an instructional leader who is able to get all players in a school to collaborate," Huth said
The bonuses amounted to as much as 30% of teachers' yearly salaries here in the Music City, where the scale runs from $36,000 to $64,000, Huth said. The nation's 3.2 million public school teachers earned $53,910 on average in 2008-09, according to the National Center for Education Statistics.
The study was released at a two-day conference, "Evaluating and Rewarding Educator Effectiveness," at Vanderbilt's Peabody College that drew participants from Colorado, Georgia, Tennessee, Washington, D.C. and other places conducting their own experiments with performance pay. Some states have moved to tie teacher and principal evaluations to student test scores.
The study did not shake the faith of U.S. Secretary of Education Arne Duncan in merit pay.
"While this is a good study, it only looked at the narrow question of whether more pay motivates teachers to try harder," said Sandra Abrevaya, a spokeswoman for Duncan. It did not address the Obama administration's push to "change the culture of teaching by giving all educators the feedback they need to get better."
The Nashville experiment, known as POINT (Project on Incentives in Teaching), doled out the $15,000 bonuses to those teachers whose students performed "at a level that historically had been reached by only the top 5% of middle school math teachers." Teacher performance was calculated by using a value-added model, which predicts how students will do in a given year based on how they performed in the previous year. The teachers had to hit the 80th and 90th percentiles to pocket the $5,000 and $10,000 bonuses, respectively.
The study's design, in which teachers were judged against percentile benchmarks rather than their colleagues' performance, sought to preserve collaboration among teachers.
In surveys about the program, most teachers said they were already effective without the incentive of additional pay. Eight in ten said they didn't change the way they taught to improve their odds of earning a bonus. Many teachers came close to getting a bonus — so close that they would have qualified if their pupils answered two or three more questions correctly on the 55-question state exam.
The Nashville math teachers, according to the study, "expressed moderately favorable views toward performance pay in general, though less so for POINT in particular." The experiment ran smoothly, although the teachers became less enthusiastic over the three years. "They did not come away … thinking it had harmed their schools," the study said. "But by and large, they did not endorse the notion that bonus recipients were better teachers."
The fact that many fifth-grade teachers teach multiple subjects to the same students may have been a reason for the positive impact of merit pay found in fifth grade, according to the study's authors. But "the effect did not last. By the end of 6th grade it did not matter whether a student's 5th grade math teacher had been in the treatment or control group," the study said.
The researchers said the Nashville experiment didn't stir the negative reactions that have attended some other merit pay programs, but it "simply did not do much of anything."
Springer of NCPI said the study lays a foundation for further experiments on a topic that educators have been debating "for over a century."
Tennessee Commissioner of Education Timothy Webb said the Nashville study shows, "Money alone is not enough to encourage people to go into challenging schools and teach the most difficult students." He stressed the importance of improving teachers' working conditions, not just their pay.
Frederick M. Hess, director of education policy studies at the American Enterprise Institute in Washington, D.C., said he does not believe the study says much of value and worries it will only confuse the issue.
"The fact that that teachers don't respond to cash bonuses like rats do to food-pellets does nothing to diminish my confidence that it's good for schooling if teacher pay better reflects the contributions that teachers make," Hess said. "Serious proponents of merit pay believe the point is not any kind of short-term test-score bump but making the profession more attractive to talented candidates."
William Slotnik, executive director of the Boston-based Institute for Compensation Reform and Student Learning at the Community Training and Assistance Center, has argued that performance-based compensation tied directly to the educational mission of a school district can be a lever to transform schools. But he said it will take more than financial incentives to improve student achievement and that merit pay "is hard to get right. … If all you are doing is focusing on money, there is no track record in that resulting in the kind of changes needed to do this work well."
Contributing: Christopher Connell is a freelance writer. Liz Willen contributed to this article. It was produced by The Hechinger Report, a nonpartisan education-news outlet affiliated with the Hechinger Institute on Education and the Media at Columbia University
No comments:
Post a Comment