jay p. greene
in education reform
robert m. costrell
gary w. ritter
in education policy
in teacher quality
patrick j. wolf
in school choice
|Evaluation of Year One of the Achievement Challenge Pilot Project in the Little Rock Public School District|
by Joshua H. Barnett, Gary W. Ritter, Marcus A. Winters, and Jay P. Greene
Department of Education Reform
About the Authors
Joshua H. Barnett is a Distinguished Doctoral Fellow in the Public Policy Ph.D. program at the University of Arkansas. He has performed studies examining the effects of various education policies including merit pay, school discipline, and school finance in Arkansas, New Jersey, and Philadelphia. He works in the Office for Education Policy at the University of Arkansas. He earned his M.A. in communication studies from New Mexico State University in 2003.
Gary W. Ritter, Ph.D. is an Associate Professor of Education and Public Policy and holder of the Endowed Chair in Education Policy in the Department of Education Reform at the University of Arkansas. He is also the Associate Director of the inter-disciplinary Public Policy Ph.D. program and the Director of the Office for Education Policy at the University of Arkansas. His research interests include program evaluation, standards-based and accountability-based school reform, racial segregation in schools, the impact of pre-school care on school readiness, and school finance. He earned a Ph.D. in Education Policy in 2000 from the Graduate School of Education at the University of Pennsylvania.
Marcus A. Winters is a Doctoral Academy Fellow at the University of Arkansas. He has performed several studies on a variety of education policy issues including high-stakes testing, charter schools, and the effects of vouchers on the public school system. His op-ed articles have appeared in numerous newspapers, including The Washington Post, USA Today, and the Chicago Sun-Times. He received his B.A. in political science with departmental honors from Ohio University in 2002.
Jay P. Greene, Ph.D., is the Endowed Chair and Head of the Department of Education Reform at the University of Arkansas. He has conducted evaluations of school choice and accountability programs in Florida, Charlotte, Milwaukee, Cleveland, and San Antonio. He has also recently published research on high school graduation rates, social promotion, and special education. His articles have appeared in policy journals, such as The Public Interest, City Journal, and Education Next, in academic journals, such as the Teachers College Record, the Georgetown Public Policy Review, and the British Journal of Political Science, as well as in major newspapers, such as the Wall Street Journal,
We thank Kinsey Daven, Erica Fitzhugh, Crystal Green, Joe Ballard, Sharon Wrobel, Keith Nitta, Laine Gates, Brent Riffel, and Alison Hoffman for their assistance on this project. We also greatly appreciate the efforts of Ed Williams, Hugh Hattabaugh, Jim Wohlleb, Karen Dejarnette, Maurecia Robinson, Olivine Roberts, Sadie Mitchell, and Suellen Vann from the Little Rock School District, and Lisa Black of the Public Education Foundation of Little Rock. Finally, we thank the principals, teachers, and staff who welcomed us into their schools and provided information for this evaluation.
Can merit pay address Arkansas’ teacher quality dilemma? Supporters of merit pay programs in education argue that such programs would encourage teachers to be more innovative, work harder, and be more satisfied with their salary. The culmination of these advantages would result in better instruction and improved student achievement. However, opponents argue that such programs would increase negative competition, degrade the school environment, and encourage teachers to neglect low-performing students. The culmination of these disadvantages would result in poor instruction and declining student achievement. Despite the passionate arguments made by merit pay supporters and opponents alike, little is known about the actual impacts of merit pay programs on students or teachers because rigorous evaluations are rare.
The Achievement Challenge Pilot Project (ACPP) operating within the Little Rock School District provides an opportunity to conduct a rigorous evaluation to uncover the effects of a merit pay program on student achievement, as well as on the behaviors and attitudes of teachers with regard to innovation, working harder, satisfaction, competition, environment, and openness to challenges. Over the last year, we analyzed data reported by the District as well as data collected from the surveys of teachers. Based on our comparison of students and teachers in the ACPP schools and students and teachers not in the ACPP schools, we note eight key findings:
1. Students in schools where the ACPP operated in 2005-06 showed an improvement of 3.5 normal curve equivalent points. For the average student, this gain represents an improvement of nearly 7 percentile points.
4. Teachers in the merit pay program reported that they were more satisfied with their salaries than comparison teachers.
While the results from this first year study suggest positive impacts of the ACPP, we believe the second year study with five schools involved in the ACPP will greatly assist in expanding on and explaining the first year findings.
Can merit pay address Arkansas’ teacher quality dilemma? Supporters of merit pay programs in education argue that such programs would encourage teachers to be more innovative, work harder, and be more satisfied with their salary. The culmination of these advantages would result in better instruction and improved student achievement. However, opponents of merit pay programs in education argue that such programs would increase negative competition, degrade the school environment, and encourage teachers to neglect low-performing students. The culmination of these disadvantages would result in poor instruction and declining student achievement. Despite the passionate arguments made by merit pay supporters and opponents alike, little is known about the actual impacts of merit pay programs on students or teachers because rigorous evaluations are rare (Eberts, Hollenbeck, & Stone, 2002; Figlio & Kenny, 2006; Podgursky & Springer, 2006).
Difficulties in assessing merit pay programs fit into two general categories. The first problem deals with available resources. Due to lack of funding, many merit pay programs do not last long enough to gauge actual impacts. Also, the developers of many merit pay programs are unable to provide a large enough bonus to serve as an incentive, and/or the programs developed are not implemented on a large enough scale to determine impacts (Murnane & Cohen, 1986; Odden, Kellor, Heneman, & Milanowski, 1999). Additionally, many programs do not last long enough to determine impacts due to the opposition against merit pay (Ballou & Podgursky, 1997). The second problem deals with measuring student performance. Student impacts are difficult to assess because schools often do not take the same standardized tests across years. That is, before NCLB in 2002, public school students were generally not tested in each grade on a vertically scaled standardized test; rather they were tested every few years (e.g. grades 4, 8, and 11).
Thus, the question remains – Can merit pay programs significantly improve the academic performance of students, while rewarding and retaining teachers rather than dividing and discouraging them? The present evaluation responds directly to this question by examining the effects of the Little Rock merit pay program on student achievement, as well as examining the behaviors and attitudes of teachers with regard to innovation, working harder, satisfaction, competition, environment, and openness to teaching challenges. Using a regression model, where each student is compared to himself, we are able to determine the change in students’ test scores after the program was introduced in the student’s school. The change in normal curve equivalent points indicates the degree of influence of the ACPP on student performance. We also administered a survey to teachers in the two schools where the program was operating during 2005-06 and to teachers in three comparison schools where the program did not operate during 2005-06. The behaviors and attitudes of ACPP school teachers and comparison school teachers were compared to detect differences.
The results of this evaluation provide information for the Little Rock School District, the Arkansas Department of Education, and other state and local policymakers as they consider ways to improve student performance and reward and retain high quality teachers.
Why Merit Pay?
Merit pay plans are believed to have both motivational and compositional effects. First of all, proponents contend that the possibility of earning merit rewards will motivate current teachers to improve the performance of students through additional effort and innovation, where teachers work to learn and implement new effective teaching strategies. Further, merit pay advocates hope that this type of reform will result in a changing (and improved) composition of the teacher workforce. As the most effective teachers consistently earn the greatest bonuses, a natural selection occurs within the school where more effective teachers remain and less effective teachers leave the field due to a lack of bonuses. As less effective teachers leave the profession, more teachers begin applying for positions in these schools due to the increased salary potential. And, these new teachers entering the profession may be those who would expect to perform well under a merit pay program. Therefore, administrators have larger applicant pools from which to select effective beginning teachers. This process can lead to a systematic change of the teacher workforce.
With these goals in mind, the Little Rock School District and the Public Education Foundation of Little Rock partnered to create the Achievement Challenge Pilot Project (ACPP) at Meadowcliff Elementary School in 2004. The ACPP operated in Meadowcliff in 2004-05 and expanded to Wakefield Elementary in 2005-06. The ACPP further expanded in 2006-07 to three include three more elementary schools – Geyer Springs, Mabelvale, and Romine.
The Achievement Challenge Pilot Project
The Achievement Challenge Pilot Project operates within five of the Little Rock School District’s (LRSD) 34 elementary schools. The LRSD is Arkansas’ largest district serving nearly 25,000 students. The five elementary schools were selected based on their high percentages of minority students and students who qualify for free and reduced lunch, as well as their low academic achievement. The data show that 70 percent of the LRSD students qualify for the federal free and reduced lunch program, and 68 percent of the students in the LRSD are African American. The schools participating in the ACPP serve a more disadvantaged group of students, 89 percent of whom qualify for the federal free and reduced lunch program and 78 percent of whom are African American.
The ACPP uses student performance as the only basis for financial rewards for teachers and staff. Under the ACPP, all students in Meadowcliff were administered a nationally-normed test (the Stanford Achievement Test Version-9) at the beginning of the school year and the results were used by teachers to target instruction to meet student academic needs throughout the 2004-05 school year. In the spring 2005, the students took the SAT-9 again and their academic gain was measured. The gain for each student was determined by the change in his or her normal curve equivalent (NCE) score between the fall 2004 and spring 2005 tests.
Student and Teacher Information from the Schools
For the 2004-05 school year, Meadowcliff teacher bonuses were linked to growth in NCE points for each student, while staff bonuses were linked to the gains of the school as a whole. A bonus was provided to teachers for every student who gained, with higher bonuses provided for students who showed higher gains. The bonuses were as follows: a $100 bonus was provided for each student who gained from zero to four NCE points, a $200 bonus for each student who gained from five to nine NCE points, a $300 bonus for each student who gained from 10 to 14 NCE points, and a $400 bonus for each student who gained over 15 NCE points. For example, if all 20 students in a teacher’s class gained between 10 and 14 NCE points, that teacher’s bonus would be $6,000 (20 x $300). The bonuses were substantial, ranging from $1,800 to $8,600. In total, $134,800 was awarded to the teachers and staff as a result of student gains. Overall, the Meadowcliff students moved from the 18th percentile in the fall to the 30th percentile in the spring.
Due in large part to the perceived success of the ACPP and the motivation of the Public Education Foundation of Little Rock, the ACPP expanded to Wakefield Elementary for the 2005-06 school year. In their first year with the ACPP, students at Wakefield moved from the 16th percentile in the fall to the 29th percentile in the spring. In 2005-06, Meadowcliff teachers and staff were awarded $200,926 while their peers at Wakefield were awarded a total of $228,300.
For the 2006-2007 school year, the ACPP expanded to include three more elementary schools: Geyer Springs, Mabelvale, and Romine. The LRSD became financially responsible for the program at Meadowcliff and Wakefield, while various private foundations assumed financial responsibility for Geyer Springs, Mabelvale, and Romine. In September 2006, teachers in each of the five schools voted to accept the changes to their salary schedule for the 2006-07 year, and the Little Rock School Board approved the program to operate in the schools. For the 2006-07 school year, Table 1 provides a description of how the payouts will be awarded at each school, except Meadowcliff, where the program began in 2004-05. When Meadowcliff began the program, the payouts were arranged differently (see Table 2). The primary difference between Meadowcliff and the other four schools is the calculation of the payouts. At Meadowcliff the payouts are based on the number of individual students who achieve growth, while the payouts at Mabelvale, Geyer Springs, Romine, and Wakefield are based on the average growth of each teacher’s class.
For example, if a first grade teacher at Geyer Springs has 20 students and 10 students improve by 10 NCE points and 10 students decline by 10 NCE points, then the class average growth is 0. The teacher receives a bonus of $1,000 ($50 times 20 students). If this first grade teacher is at Meadowcliff and 10 students improve by 10 NCE points and 10 students decline by 10 NCE points, then the class average growth is 0. However, the Meadowcliff teacher receives a bonus of $3,000 ($300 times 10 students who gained 10 NCE points plus $0 for each of the 10 students who decline 10 NCE points). The potentially negative incentive for teachers to focus only on selected students in the Meadowcliff payout plan contributed to the payout change in the other schools. However, since the teachers at Meadowcliff approved the program in 2004-05 based on the per student payout structure, the payout scheme was not changed.
(click on the table for a larger view)
Achievement Challenge Schools and Comparison Schools
For the sake of analyzing the initial impact of the ACPP on teachers, three comparison schools were selected. The comparison schools were selected based on their comparable school-level demographics and performance scores. In 2005-06 at comparison schools, 93 percent of students qualified for the federal free and reduced lunch program and 85 percent of the students were African American. The comparison schools were also selected because of their comparable achievement scores. To match the ACPP and comparison schools on performance, the two-year (spring 2003 and spring 2004) average proficiency percentage on the Arkansas Benchmark math and literacy exams were used. In math, 47 percent of the ACPP students tested at the proficient level compared to 38 percent of the comparison students. In literacy, 52 percent of the ACPP students tested at the proficient level compared to 48 percent of the comparison students. All ACPP and comparison schools serve students in kindergarten through fifth grade. Table 3 provides a description of the characteristics of the ACPP and comparison schools. The section that follows describes how these schools are employed in our analysis to estimate the program impacts of the ACPP in Little Rock.
Achievement Challenge Pilot Project Evaluation
The evaluation plan for the ACPP is guided by two questions.
• How has the ACPP affected student performance?
Determining Effect of ACPP on Student Performance
The Little Rock School District’s Planning, Research, and Evaluation Department provided student-level demographic and achievement test data. In order to determine the effects of the ACPP on student performance, we use the data provided by the LRSD to predict a math change score. The variables incorporated into the prediction model are:
• Change in math performance from 2002-03 or 2003-04 to 2004-05 by student;
These four variables were used to predict the change in math performance by student and by year. The most appropriate type of analysis to determine the effect of the ACPP on student performance was to compare each student to himself or herself through a student-level fixed effects model. The model uses information about each student and predicts an achievement growth value for said student. The model then compares the predicted values to the actual values for each student. The estimated effect of the ACPP is the actual change in achievement for the ACPP students compared to the expected change in achievement. The fixed effects model has gained widespread use in the social science and education research community, and is the most appropriate methodology to respond to our question regarding the effect of the ACPP on student achievement.
For the fixed effects model to estimate correctly, we needed students with three years of test scores. Prior to 2004-05, elementary schools in the LRSD administered standardized tests only to second grade students. The nationally norm-referenced test used was the Stanford Achievement Test-Version 9 (SAT-9). In 2004-05, the statewide standardized test was switched from the SAT-9 to the Iowa Test of Basic Skills (ITBS). Since the SAT-9 was administered to second grade students only, we could obtain three years worth of data for students who were in second grade in or before 2003-04. Therefore, the only students who possessed three years worth of test scores are the students who were in fourth and fifth grade in 2005-06 (see Table 4).
Selecting the Appropriate Measure
Only the math test score was used due to available data limitations. Language scores could not be used because fourth and fifth grade students only possess two years of language testing. Reading scores could not be used because the fourth and fifth grade students did not take the reading total ITBS test in 2005; consequently, fourth and fifth grade students only possess two years of reading total tests. Therefore, we rely on the math scores for fourth and fifth grade students because they possess a math total score for three different points in time: 2002-03 or 2003-04, 2004-05, and 2005-06. Table 4 highlights the nationally-normed tests administered to Arkansas students since 2002-03. The subtests administered are noted by grade and year. The darkly shaded areas indicate that no test was given. The fourth and fifth grade math scores are lightly shaded to indicate these are the tests used in the analysis.
Finally, for the regression model to estimate correctly, the students need to have a pre-program and post-program growth score. In 2005-06, the ACPP operated in Meadowcliff and Wakefield. However, Meadowcliff began the program in 2004-05. Therefore, the students at Meadowcliff do not possess a pre-program growth score because Meadowcliff students only have one score before their school began the ACPP. Due to the limits of the available data, we base our estimate of the ACPP impact on fourth and fifth grade students in Wakefield and the three comparison schools.
Determining Effect of ACPP on Teacher Behaviors and Attitudes
The teachers within ACPP schools (Meadowcliff and Wakefield) and teachers in the three comparison schools (Baseline, Chicot, and Franklin) were surveyed on their behaviors and attitudes with regard to concepts often discussed in the literature regarding merit pay. Based on the literature and our discussions with administrators and teachers, our survey investigated these perceived advantages and disadvantages of merit pay to answer the following questions.
We asked teachers about the three possible advantages often posited in favor of merit pay: innovation, working harder, and pay satisfaction. We also asked teachers about the three potential disadvantages used as arguments against merit pay: counterproductive competition, degraded work environment, and viewing low-performing students as burdensome. Furthermore, we asked teachers about becoming a better teacher and improving student performance. These two issues are heralded as the eventual benefits from those supporting merit pay, while those who oppose merit pay posit that instruction will worsen and student achievement will decrease.
Surveys were administered in-person to teachers at each school. Through the assistance of the Little Rock School District, we made contact with each school principal and scheduled times to administer the surveys. Two graduate students and a professor from the University of Arkansas-Little Rock and two graduate students and a professor from the Clinton School of Public Policy assisted with administering the surveys. The survey packet for each teacher included the survey and a letter of informed consent. We also provided refreshments during the survey process, which took place at the end of the school day, generally between 2:45 and 3:30.
The survey results presented here are based on completed surveys from 58 of the 69 total teachers, specialists, and coaches in the two ACPP schools and 74 of the 121 total teachers, specialists, and coaches in the comparison schools.
Based on our analysis, we found that the ACPP significantly improved student test scores by 3.5 normal curve equivalent points on a nationally-normed math test. Notice that the pre-program mean score is similar for all students; however, the students in the ACPP schools improved, while the students in the comparison schools decreased.
This gain in achievement after one year’s time is roughly equal to one-sixth of the nationwide average test score gap between black and white students. If the observed benefit of the merit pay program were to compound for six years, it would close the black-white test score gap. Figure 1 further illustrates the gain of the ACPP students over the comparison students.
After collecting surveys from the ACPP teachers in Meadowcliff and Wakefield and the comparison teachers in Baseline, Chicot, and Franklin, we compared their responses. Overall, we found that the two groups of teachers differed significantly in their responses to the perceived advantages and disadvantages associated with merit pay.
Perceived Advantage – Findings
In addition to these findings, some specific question responses are worth noting. These questions fall into two main categories. Overall, we find that the majority of respondents report that the foundations of the current salary system – experience and advanced degrees – are not related to teacher effectiveness (questions 6 and 7). Finally, we find that the respondents in the ACPP schools report that merit pay programs increase collaboration (questions 2 and 11).
Views of Current Salary System
Table 5 provides the mean scores and percent agreeing/disagreeing to each of the constructs and questions of interest. Appendix A is a copy of the survey given to the teachers. Appendix B shows the questions under each corresponding construct. Appendix C shows the percentage of responses for each question of the survey. Appendix D shows the mean score and statistical information for each question from the survey.
So, can merit pay address Arkansas’ teacher quality dilemma? Based on the findings from the first year evaluation of the Achievement Challenge Pilot Project, the future of merit pay is promising. Student performance improved in schools where the program operated. With respect to the findings from the survey, the merit pay program is positively viewed by ACPP teachers. Of the positives often cited with merit pay programs, we found that ACPP teachers are more satisfied with their salaries than comparison teachers. However, comparison teachers were more likely to claim that they expended increased work effort in the past year. Of the potential negatives often associated with merit pay programs, we found that counterproductive competition did not increase, that the school environment became more positive, and teachers did not view low-performing students as a burden.
Therefore, we find that at least one perceived advantage of merit pay (increased satisfaction with compensation) held true in Little Rock, while none of the perceived negatives held for the teachers in the ACPP schools. Overall, we find that the ACPP teachers view themselves to be more effective compared to the comparison teachers. Notably, we find that ACPP teachers were significantly more likely to report that student performance improved over the past year – 93 percent compared to 46 percent.
This is perhaps the most important survey finding. The teachers perceived that the ACPP improved student performance, and the test data support this belief. This suggests that teachers witnessed measurable academic improvement for their students in the classroom in addition to the performance of the students on the Iowa Test of Basic Skills.
In addition to the comparison of the ACPP and non-ACPP teachers, we learned much about all teachers in the five Little Rock elementary schools. Of all respondents, the majority report that neither more teaching experience nor advanced degrees makes teachers more effective at teaching. This result indicates that most teachers do not report that the current salary system rewards effectiveness, since the current system rewards tenure and advanced degrees with higher salaries. Beyond the dissatisfaction with the current salary system, we learned that teachers view their job to be more difficult compared to previous years. However, we also learned that the majority of all teachers (ACPP and comparison) researched new teaching strategies, spent more time preparing for their job, enjoyed teaching more, and reported being better teachers than previously.
Ballou, D., & Podgursky, M. (1997). Teacher pay and teacher quality. Kalamazoo, MI: W.E. Upjohn Institute for Employment Research.
Eberts, R., Hollenbeck, K., & Stone, J. (2002). Teacher performance incentives and student outcomes. Journal of Human Resources, 37(4), 913-927.
Figlio, D. N., & Kenny, L. (2006). Individual teacher incentives and student performance.
Murnane, R. J., & Cohen, D. K., (1986). Merit pay and the evaluation problem: why most
Odden, A., Kellor, E., Heneman, H., & Milanowski, A. (1999). School-based
university of arkansas | department of education reform | 201 graduate education building | fayetteville | ar | 72701
Ph: 479|575-3172 Fax: 479|575-3196 | e-mail: email@example.com