| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
MPCP Longitudinal Educational Growth Study Baseline Report John Witte Patrick J. Wolf Josh M. Cowen David J. Fleming April 7, 2008 |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Executive Summary This report focuses on the initial design, implementation and baseline results of the five-year Longitudinal Educational Growth Study (LEGS) of the Milwaukee Parental Choice Program (MPCP) being conducted by the School Choice Demonstration Project (SCDP). The LEGS will be the first evaluation of the participant effects of the MPCP using student-level data to be implemented since the initial pilot program expanded dramatically in 1995. Included this initial report are baseline descriptions of achievement tests for a representative sample of MPCP students in grades 3 through 9, as well as outcomes for comparable samples of students in Milwaukee Public Schools (MPS). Also included are a descriptive analysis of survey results of MPCP parents and a carefully matched sample of MPS parents as well as a brief description of the results of student surveys for both samples. Introduction The Milwaukee Parental Choice Program The Milwaukee Parental Choice (Voucher) Program (MPCP) was the first parental choice program in the United States to allow urban parents to send their children to private schools with public funds covering the entire costs.[1] Often simply referred to as the “Choice” program, it is the largest of 12 voucher programs in the United States. In 2006-07, the baseline year of this report, a total of 17,749 students received vouchers that averaged $6,501 to attend one of 122 participating private schools in Milwaukee.[2] Research Questions in the Longitudinal Educational Growth Study METHODOLOGY One of our first research tasks was to identify two comparable samples of students—one from the MPCP and one from MPS. Samples were required (rather than entire populations) because of resource limitations on surveying parents, testing students, and tracking students over time. After estimating the size of the samples needed for identifying any statistically significant results after five years (given anticipated attrition), a grade-stratified random sample of MPCP students was drawn for grades 3 to 8. Those grades were selected because MPS administers the Wisconsin Knowledge and Concepts Examinations (WKCE) annually in those grades for federal accountability purposes. We selected all the Choice students in the 9th grade to follow so that we had a large enough sample to withstand attrition for our five-year attainment study. [4] Figure 1 (click for a larger view) Table 1 displays demographic data on students in our MPCP sample and students in both of the potential comparison groups. Perhaps the most important statistic is the proportion of white students in each sample, which indicates disproportionately high participation in MPCP among minority students. Nine percent of students in the MPCP sample and our MPS Match sample are white compared to 15 percent in the MPS Random sample. In both the MPCP and MPS Matched samples, African-Americans represent roughly two-thirds of the students, compared to 60 percent in the Random MPS sample. Compared to the MPCP sample, Hispanics are slightly underrepresented in the MPS Matched sample, where the 20 percent of students classified as Hispanic is closer to the MPS Random sample. The MPCP sample and our MPS Matched sample are more similar in gender and ELL breakdowns than the MPCP sample is to the MPS Random sample. Table 1: Comparisons of Student Characteristics in Three Possible Study Samples, 2006-07
Note: Cell values represent proportions of the total sample. BASELINE DESCRIPTIVES Table 2: Sample WKCE Scores (Grades 3-8), 2006-07
Note: “SD” stands for standard deviation. As expected by our algorithm, the MPCP students are generally very close to the MPS matched sample on most tests. And where there is a statistically different mean, the mean differences are quite small. Thus, as outlined in the last section, our algorithm for constructing the MPS Matched sample worked very well. Figure 2 (click for a larger view) Ninth graders in MPS do not take the WKCE. Instead they are administered the Benchmark Exam, which we similarly administered to the 801 MPCP 9th graders in our sample. Because we matched test scores by grade, this did not require a separate matching procedure, although the test scales were different (scale scores for the WKCE; percent proficient in subject for the Benchmark). [11] We report the results of 9th grade Benchmark scores in Table 3. As in earlier grades, the MPCP 9th graders are closer to the MPS Match than to the random sample of MPS students in the same grade. Unlike the earlier grades, however, MPCP 9th graders appear to exceed their MPS counterparts in one subject area (reading), and this difference carries over to the combined math/reading scores. Observed with the pattern of converging scores (albeit from separate testing instruments) evident in Figures 1 to 3, these results indicate relative parity between MPCP students, the MPS Match students, and the MPS population in the later grades. Across the grade levels, there is little evidence that MPCP schools are “cream-skimming” the “best” students, at least in terms of test scores. Table 3: Sample Benchmark Scores (Grade 9), 2006-07
***Different from MPCP at p<0.01 Parent and Student Views: MPCP and MPS Survey Results
The survey results provide us with a nuanced understanding of education circumstances. The main purposes of the survey were: (1) to understand who participates in the program; (2) to understand the school choice environment in the city of Milwaukee; (3) to understand parental participation in and expectations for their children’s achievement; and (4) to gauge levels of parental satisfaction with their children’s educational experiences. In the future, we will be able to use information from the surveys to model growth in student achievement. Who Participates in the Choice Program? The complete results of the parental survey are available in Appendix D. The MPCP parental results include 1,856 fully completed surveys and 4 partially completed surveys from those in the total sample (including testing parental refusals). Results from 1,438 completed surveys from MPS parents are also included. Analyzing the demographic characteristics of parents provides some insight into who chooses and who does not. However, one must remember that we matched students based on test scores, census tracts and, in some cases, student demographic information. Therefore, we did not expect large differences between MPCP and MPS parents, at least not at baseline.
Note: respondents could select more than one category Education. There are some important differences when examining the highest education level of respondents (Q38). About 51 percent of MPCP respondents have a high school diploma or less, while 14.0 percent have at least a four-year college degree. On the other hand, MPS respondents are comparatively less-well educated with roughly 57 percent of parents having a high school diploma or less and 11.1 percent having a college degree or more. This result is consistent with Witte’s (2000) previous analysis of the Milwaukee choice program from 1990 to 1994. Witte found that 45 percent of mothers with a child enrolled in the MPCP were a high school graduate or less, while only 8 percent had a four-year college diploma or post-graduate work. One of the major differences in Witte’s (2000) results and ours is that Witte found that 46 percent of MPCP mothers had some college education, while only 30 percent of MPCP parents and 26.3 percent of MPS parents in our sample are in the “some college education” category. Thus, although MPCP parents are somewhat more educated than MPS parents, both groups are less well-educated now than in the 1990-94 period.
Employment and Income. Although there are differences in the education levels of MPCP and MPS respondents, an examination of the employment questions finds no large differences (Q39 and Q40). When asked about the government assistance received by anyone in the household, MPCP and MPS respondents tended to reply similarly (Q59). However, MPS respondents were slightly more likely to receive assistance from various government programs.
Home Lives. The survey results allow us some insight into the home lives of the MPCP and MPS students. We find that MPCP students are about as likely as MPS students to be living with married parents (38.2 percent vs. 35.7 percent). This is an increase for both groups from 1990-1994 (23 percent for MPCP, 35 percent for low-income MPS families). According to our survey, about 57 percent of MPCP and MPS respondents said that the child’s other parent or guardian does not live in the household (Q42). There is almost no difference in family size for the two groups, while MPCP respondents are slightly more likely to own their own home (41 percent vs. 38 percent) (Q56). Given that there are more Hispanics in the MPCP sample than the MPS sample, it is unsurprising that more MPCP respondents said that they speak Spanish at home with their children (19.3 percent) than MPS respondents (11.3 percent).
Finally, there are few differences between and MPS and MPCP respondents when asked how long they have lived at their current address (Q55). When asked how long they have lived in Milwaukee, the majority of both MPS and MPCP parents answered more than 25 years (Q57).
The Choice Environment Tenure in Schools. Beyond demographic information, the survey results provide a deeper understanding of the education environment in Milwaukee. For example, there is great mobility in Milwaukee’s schools. About 39 percent of MPCP students and 37 percent of MPS students have been at their school for one year or less (Q2). On the other hand, 41.2 percent of MPCP students have been in their schools for three or more years. There are, however, clearly some highly mobile families in that 28.2 percent of MPS parents and 21.3 percent of MPCP parents said that their child has attended four or more schools (Q20).
Parental information levels are an important component of school choice programs. About 38 percent of MPCP parents learned about the voucher program from friends or relatives (Q7, first response). In the same way, over 33 percent responded that they heard of the program from their school. The importance of social networks is also evident in choosing a specific school. When asked how they initially heard of their child’s school, the first response of 54 percent of MPCP parents and 42 percent of MPS parents was from friends or relatives (Q15). The major difference between the groups regarding how they heard about their current schools is that MPCP parents received information from their church (9.4 percent) or from other private schools (4.5 percent) while MPS parents did not. How did you learn about the Parental Choice Program, also known as the voucher program?
Charter schools play a prominent role in Milwaukee’s education environment. MPS parents (72.0 percent) were more likely than MPCP parents (65.1 percent) to have heard of these public schools of choice (Q12). Of those who have heard of charter schools, roughly one-fifth of both MPCP and MPS parents have applied to a charter school (Q13). About 7 percent of MPCP parents and 10 percent of MPS respondents said that their child has attended a charter school. Which school characteristic is the most important?
Was this school your first choice for your child?
Parental Involvement, Expectations, and Importance of Education The survey included several measures of parental involvement. MPCP parents were more involved in school activities than MPS parents, including volunteering at their child’s school, attending parent/teacher conferences, taking part in the activities of a parent/teacher organization, and belonging to other education organizations (Q26). In terms of contact with schools (Q25), results were mixed between the two groups. Both groups were contacted most often concerning their child’s academic performance. MPCP parents were more likely to be contacted for volunteering or fundraising.
We also asked parents how many times in a normal week they participate in activities that are educationally beneficial to their children. On this measure, MPS respondents are more likely than MPCP respondents to participate in these types of behaviors (Q27). About 74 percent of MPS parents said that they help their child with their homework three or more times a week, while 62 percent of MPCP parents do the same. Likewise, MPS parents were more active than MPCP parents in reading with their children, working on math, helping with writing and watching educational television programs. One might think that MPS parents are more active than MPCP parents because MPS students are assigned more homework. We find no evidence of this, as MPCP students report spending more time on homework than MPS students (Q29). Weekly Participation in Child’s School-Related Activities*
When asked about the educational expectations they have for their child, MPCP parents were more positive than their MPS counterparts. Over 54 percent of MPCP parents expect their child to graduate from college or go to graduate school, whereas only 46.7 percent of MPS parents expect the same (Q33). Likewise, 16.2 percent of MPS parents believe their child will achieve a high school diploma or less, as opposed to 8 percent of MPCP parents. In addition, MPCP parents believed their school had higher expectations for their children. Roughly 47 percent of Choice parents strongly agreed with the statement that their child’s school has high expectations for academic achievement compared to 36.1 percent of MPS respondents (Q31).
In order to measure how important education is to parents, we asked them to compare the importance of education to other goals (Q34). For example, about three-fourths of both MPCP and MPS parents said that education is more important than having a good job. Examining the survey results for these questions, one finds few differences between MPCP and MPS respondents. Overall, MPCP parents are slightly more likely to say that education is just as important as these other goals. In another attempt to determine the importance of education to parents, we asked them if they have ever moved so that their children may attend a better school. We found that this is a relatively rare occurrence, as only 15.7 percent of MPCP respondents and 14.1 percent of MPS respondents said that they moved for this reason (Q30).
Parental Reports of Student Success The survey results provide us with the opportunity to learn more about our MPCP and MPS students beyond what we know from administrative data. Asked what was the average grade their child received in school this past year, three-fourths of MPCP parents and 64 percent of MPS parents said their child generally received As or Bs (Q32). According to the results from this question, it appears as if MPCP students in our sample received slightly higher grades than their MPS matches (MPCP GPA: 3.0 vs. MPS GPA: 2.8).
A relatively small percentage of MPCP students (2.7 percent) and MPS students (4.2 percent) in our sample have physical handicaps (Q46). Those parents with handicapped children were also asked how well their child’s school meets their child’s needs. MPS parents seemed more satisfied, as 51.7 percent of MPS parents said that the facilities met their child’s needs very well, while only 44 percent of MPCP parents responded similarly (Q47). Parental Reports of School Satisfaction Using the survey results, we can examine children’s educational experiences, as well as the relationships between parents and schools. Perhaps one of the most important measures of school effectiveness is parental satisfaction. We asked parents a battery of questions regarding their levels of satisfaction with their child’s school across a variety of school characteristics (Q22). Overall, the results reveal that parents are quite satisfied in both MPCP and MPS. For all but one of the 15 characteristics, the majority of both MPS and MPCP parents said that they were very satisfied or satisfied. The one exception concerns how much students can observe religious traditions in schools, as MPS parents are relatively less satisfied. While both MPS and MPCP parents appear quite satisfied with their children’s schools, MPCP parents are relatively more satisfied. This discrepancy arises as MPCP parents often said they were “very satisfied” with school characteristics, while MPS parents were generally “satisfied.”
Examining the mean scores for each school attribute (Q22, Appendix D), one finds that MPCP parents are most satisfied with what is taught in school, school safety, and the amount their child has learned. This is an especially important finding because school safety and the educational quality of the school were the most important school characteristics for MPCP parents. Therefore, MPCP parents are the most satisfied with those characteristics that they deem to be the most important. On the other hand, MPS respondents were the most satisfied in regards to opportunities for parental involvement, what is taught in school and how much teachers inform parents of students’ progress. While both MPCP and MPS parents said that the discipline in school was a very important school characteristic, they were relatively less satisfied with the amount of discipline in their children’s schools when compared to other characteristics.
Note: National average comes from The 39th Annual Phi Delta Kappa/Gallup Poll (2007). Finally, satisfaction can be measured by desires to continue in one’s current school. The survey results provide even more evidence of high levels of MPCP parental satisfaction in choice schools (Q5). Over 79 percent of MPCP parents said that they would re-enroll their child in their current private school next year, compared to 63.5 percent of MPS parents who gave a similar vote of confidence to their child’s public school.
Student Survey Results
Among the 9th grade general question results, both MPS and MPCP students have high mean agreement scores for the same two statements. The statement, “My school has high academic expectations of me,” received a mean score of 3.4 for MPCP students and 3.3 for MPS students, and the statement, “I have the books and supplies I need to do well in school,” received a mean score of 3.4 for MPCP 9th graders and 3.3 for MPS. The MPCP and MPS 9th grade respondents were similar in their great dissatisfaction regarding other aspects of their school environment. Only 8 percent of MPCP 9th grade students and 7 percent of their MPS counterparts strongly agreed with the statement that students at their school focus on learning (Q20). Over a quarter of both groups disagreed or strongly disagreed with the statement that students and adults respect each other at their school (Q19).
Many of the largest mean differences between MPCP and MPS 9th graders concern the general school atmosphere. MPCP students were more positive than their MPS counterparts toward the statement, “My school building has a positive atmosphere for learning” (Q7), as well as the statement, “My school has a friendly and welcoming atmosphere” (Q10). MPS high school students were more positive than MPCP ones when considering two statements: “I usually look forward to coming to school” (Q17), and “I am pleased with my academic progress” (Q18).
The 9th grade results for the safety/discipline, diversity and new teachers categories are similar to the elementary and middle school results, as both MPS and MPCP students gave generally positive responses, though the MPCP students were more positive. MPCP students had an almost 20 percentage point advantage (51.4 vs. 31.5) in the strongly agree category for the statement, “My school provides a drug-free environment” (Q21). Similarly, 42.6 percent of MPCP students compared to 28.9 percent of MPS students strongly agreed that their schools “make sure that classrooms are safe and orderly.” These results fit well with MPCP parental differences on satisfaction with discipline noted earlier.
Survey Summaries
2. Understanding the Choice Environment
3. Parental Involvement, Expectations, and Importance of Education
4. Parental Satisfaction with Their Current Schools
Our main findings from the school climate survey administered to students were twofold. 1. Overall, for both 4th through 8th graders and 9th graders (studied separately), attitudes were very positive in both MPS and MPCP schools. SUMMARY AND CONCLUSIONS
Later reports will expand upon the results presented here at baseline. Growth in student achievement, the rate of student entry, exit, and potential re-entry into the MPCP program, and the effect of these choices on education outcomes are all subjects of future stages of the Longitudinal Educational Growth Study. |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
REFERENCES Aaronson, Daniel. 1998. “Using Sibling Data to Estimate the Impact of Neighborhoods of Children’s Educational Outcomes.” Journal of Human Resources. 33(4): 915-946. Cullen, Julie Berry, Brian A. Jacob and Steven D. Leavitt. 2005. “The Impact of School Choice on Student Outcomes: An Analysis of the Chicago Public Schools” Journal of Public Economics. 89(5-6): 729-760. Figlio, David N. and Cecelia Elena Rouse. 2006. “Do Accountability and Voucher Threats Improve Low-Performing Schools?” Journal of Public Economics. 90(1-2): 239-255. Greene, Jay P., Paul E. Peterson, and Jiangtao Du. 1999. “Effectiveness of School Choice: The Milwaukee Experiment.” Education and Urban Society, 31, January, pp. 190-213. Greene, Jay P., and Marcus A. Winters. 2003. “When Schools Compete: The Effects of Vouchers on Florida Public School Achievement” (New York: Manhattan Institute Education Working Paper, No. 2, August). Howell, William G., Patrick J. Wolf, David E. Campbell, and Paul E. Peterson. 2002. “School Vouchers and Academic Performance: Results from Three Randomized Field Trials.” Journal of Policy Analysis and Management. 21(2), pp. 191-217. Leventhal, Tama and Jeanne Brooks-Gunn. 2004. “A Randomized Study of Neighborhood Effects on Low-Income Children’s Educational Outcomes.” Developmental Psychology. 40(4): 488-507. Ludwig, Jens, Helen Ladd, and Greg J. Duncan. 2001. The Effects of Urban Poverty on Educational Outcomes: Evidence from a Randomized Experiment.” In Urban Poverty and Educational Outcomes. Brookings-Wharton Papers on Urban Affairs. Edited by William Gale and Janet Rothenberg Pack. Washington, DC: Brookings Institution Press. pp.147-201. Metcalf, Kim K., Stephen D. West, Natalie Legan, Kelli Paul and William J. Boone. 2003. “Evaluation of the Cleveland Scholarship and Tutoring Program” Indiana Center for Evaluation Technical Report. Rose, Lowell C., and Alec M. Gallup. 2007. “The 39th Annual Phi Delta Kappa/Gallup Poll Of the Public’s Attitudes Toward the Public Schools.” Phi Delta Kappan. Vol. 89 (1), p. 33-45. Rosenbaum, Paul R. and Donald B. Rubin. 1983. “The Central Role of the Propensity Score in Observational Studies for Causal Effects” Biometrika. 70(1): 41-55. Rouse, Cecilia Elena. 1998. “Private School Vouchers and Student Achievement: An Evaluation of the Milwaukee Parental Choice Program.” The Quarterly Journal of Eco¬nomics, 113(2): 553–602. Sampson, Robert J., Jeffrey D. Morenoff, and Thomas Gannon-Rowley. 2002. “Assessing `Neighborhood Effects’: Social Processes and New Directions in Research.” Annual Review of Sociology. 28: 443-478. West, Martin and Paul Peterson. 2005. “The Efficacy of Choice Threats Within School Accountability Systems,” Harvard Program in Education Policy and Governance Working Paper 05-01, http://www.ksg.harvard.edu/pepg/PDF/Papers/West_Peterson_ChoiceThreats.pdf Wolf, Patrick J., Babette Guttman, Michael Puma, Lou Rizzo, Nada Eissa and Marcia Silverberg. 2007. Evaluation of the D.C. Opportunity Scholarship Program: Impacts After One Year. Institute for Education Sciences, National Center for Education Evaluation and Regional Assistance: U.S. Department of Education. Available at http://ies.ed.gov/ncee/pubs/20074009/
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
APPENDIX A Description of the Study Mandate The legal responsibility of the School Choice Demonstration Project under Act 125 is limited to collecting standardized data from the private schools and turning it over to the Legislative Audit Bureau each year beginning in 2007. The standardized test scores were collected from participating schools throughout the 2006-07 academic year, organized into a single database, and delivered to the Legislative Audit Bureau on December 28, 2007. Those data are discussed in the MPCP Annual School Testing Summary Report. This particular report is focused on fulfilling the SCDP’s vision of conducting a longitudinal evaluation of a representative panel of choice students closely matched to a panel of MPS students in order to estimate the actual effects of the MPCP on important student and family outcomes. This idea of a rigorous longitudinal evaluation of school choice in Milwaukee has been endorsed by the SCDP’s Research Advisory Board and is being supported by the six foundations that have thus far agreed to underwrite the School Choice Demonstration Project’s Milwaukee study. Although this project is being accomplished without state funds we are grateful to the state of Wisconsin for providing legislation in the form of Act 125 to authorize this important research project. We also thank the leadership and staff of the Wisconsin Department of Public Instruction and the Milwaukee Public School District for their critical assistance with this study. Finally, we acknowledge the extensive efforts to cooperate with this evaluation undertaken by the personnel at the various private schools that participate in the MPCP. We will continue to work closely with these entities, the legislature, the Legislative Audit Bureau, and other relevant state authorities to carry out the most complete, accurate, and informative study possible within the confines of the data and research circumstances that we encounter. APPENDIX B Constructing the Sample for Study To identify a representative sample of students to study over the 5-year duration of the LEGS Achievement and LEGS Attainment studies, we first selected a random sample of participants in the Milwaukee Parental Choice Program (MPCP) from a September 15, 2006 list of applied and accepted students. To obtain proportions of students for the LEGS Achievement study in grades 3-8 that were representative of the MPCP population, we stratified the selection by the number of students in each grade in the program. We then drew random samples for each grade, for a total of 2,184 students. For the LEGS Attainment study we selected all of those in 9th grade (911 students). The samples combined for a total of 3,095 students comprising 18 percent of the population of all MPCP participants in grades 3-9. We then examined the audited list of voucher recipients on the 3rd Friday count (September 15, 2006) from the Department of Public Instruction.[14] Two hundred twenty-seven students were not on this list or had duplicate records and were dropped from the study.[15] We informed each MPCP school as to which of their students had been selected. The parents or guardian of each student were informed via letter from their child’s school of their child’s selection, and were given the opportunity to decline participation. Of those students in the sample, 134 (4.7 percent) opted out of the study. An additional 7 students were not included in the study because their grade levels were no longer within grades 3 through 8. For those students who remained after these adjustments, we obtained information on students’ race, gender, and other variables through school records. To the extent possible, for those students missing administrative data, we added data with student survey data administered during the baseline testing session. The final analytic sample was comprised of 2,727 students in the MPCP program. See Appendix C for additional information regarding MPCP sample selection. The next step in the sample construction was the selection of students in Milwaukee Public Schools (MPS) who would constitute valid comparisons to the MPCP participants. Without such a process, we would not be certain that differences in student achievement growth and other outcomes of interest between MPCP and MPS would be attributable to differences in student baseline characteristics, or differences that influence both the decision to leave or remain in public schools and the outcomes of interest. To obtain a valid comparison group, we designed a multi-step procedure to incorporate students’ neighborhood information, prior achievement levels, and student demographics into the selection of our MPS sample. The basic sample design was to first match MPCP students to MPS students on their neighborhood, then on baseline test scores, and then to use propensity scores for being in MPCP to order the MPS students already matched on neighborhood and/or baseline test. The matching was done without replacement, meaning that each MPS student could only be matched to one MPCP student. The first step in this procedure was the inclusion of the census tract corresponding to the home address reported for each MPCP and MPS student. Census tracts are geographic locations given unique identification numbers by the United States Census Bureau.[16] In this step, potential MPS matches for a given MPCP student were limited to those in the MPCP student’s census tract. In our MPCP sample, there were students from 213 unique tracts in Milwaukee. We next narrowed potential MPS matches within each tract for grade 3-8 students with similar scores on the Wisconsin Knowledge and Concepts Examinations (WKCE). Similarly, we matched 9th graders based on the Benchmark Exam scores (see below). To define which scores were “similar” we divided the distribution of mean MPCP scale scores by grade, and then further into twenty test bands, each corresponding to every 5th percentile of the grade-specific distribution of scores. In the first band were students whose scores fell between the 1st and the 5th percentile. In the twentieth band were students whose scores fell on or exceeded the 95th percentile. We restricted test score matches for each MPCP student to those MPS students whose raw test scores fell within the same test band defined by the MPCP distribution. Finally, we narrowed potential matches further by estimating the influence of students’ race, gender, ELL status and baseline test score (the mean of the math and reading tests) on the likelihood that any student would select private education (MPCP=1). In this step, we estimated a propensity score, choice, using a logit model: choice = P(MPCP=1) = exβ/1 + exβ - Equation 1 for each student based on these characteristics, X, with models estimated separately by grade. Within census tracts and test bands, MPS students were matched to MPCP students by selecting the MPS student with the closest raw value of choice to the MPCP student. In our initial data collection at baseline, we were unable to gather complete data for all students in our sample, although the data are complete for the vast majority of students, as Table B-1 indicates. For MPCP students whose propensity scores we were unable to estimate due to missing data on race, gender, or ELL we simply drew an MPS student at random from within census tract and test band matches. If a missing test score caused the missing propensity score, we drew the MPS match at random from within the set of census tract-matched students. All selections were without replacement. Table B-2 summarizes the success of these matches. As the table indicates, nearly all (99 percent) of students were matched on census tract, while only 56 percent of students were subsequently matched on test band. Table B-1 : Complete MPCP Data
Reading and math percentages for grades 3-8 based on a denominator of 1,926 students; For grade 9 scores the denominator is 801 students. Table B-2: Matching By Tract and Testband
APPENDIX C: Data Collection Procedures and Protocols In this section, we describe the methodology for selecting the sample of panel students and the protocols used for each of the following data collection activities:
MPCP Sample Selection • In September 2006, the Milwaukee Department of Public Instruction (DPI) provided Westat with an un-audited data file of 16,892 students who were expected to be enrolled in the Milwaukee Parental Choice Program (MPCP) during the 2006-07 school year. This un-audited file was used to draw the sample of panel students because the audited data file was not available prior to the state mandated window for administering the WKCE-CRT-CRT. A sample of 3,095 panel students in grades 3 through 9 was drawn. Please note the census of 9th graders were included in the sample. After removing 20 duplicate records, the panel size was reduced to 3,075 students in grades 3 through 9.
Testing Protocol
The representative sample, or “panel,” of MPCP students in grades 3-8 were tested in reading and math using the Wisconsin Knowledge and Concepts Examinations - Criterion-Referenced Tests (WKCE-CRT). Panel students in grade 9 were tested in the same subjects using the Discovery Think Link Benchmark exam. These tests were chosen for panel testing because they were also being administered by Milwaukee Public Schools. The protocol for testing all panel students follows: WKCE-CRT Testing. Sampled panel students in grades 3-8 were administered the WKCE-CRT in their own schools by school personnel between November 7–24, 2006. The testing conditions replicated how MPS students are tested and therefore allow proper comparisons between the performance of MPCP and MPS students. The testing window was required by DPI and coincided with testing being conducted by MPS schools. Benchmark Testing. The census of panel students in grade 9 were tested using the Milwaukee Benchmark exam between December 8-20, 2006 in accordance with DPI requirements. Prior to testing the panel students, the study team participated in a MPS sponsored web-cast training on administering the WKCE-CRT-CRT. This training formed the basis for developing the materials used to train MPCP staff who would be administering the WKCE-CRT to panel students. Listed below is a summary of the training related activities that occurred prior to testing.
MPCP Parent and Student Surveys Paper and Pencil Student Surveys – Fall 2006. In Fall 2006, panel students were administered a paper and pencil survey immediately after completing the WKCE-CRT or Benchmark exam. The survey was not mandatory. Students could refuse to participate in the survey component of the study. This survey is the same one that MPS administers every year to their students in grades four and higher. The question wording was slightly different in the elementary/middle school questionnaire and the high school questionnaire. Student and Parent Telephone Surveys – Spring/Summer 2007. In April 2007 Westat received the audited filed from DPI. Subsequently, a first attempt was made to match the 2,894 panel students against the audited file. 2,750 matches were found.
MPS Parent and Student Surveys
Survey Response Rates For the Parent Survey:
For the Student Survey:
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Footnotes
[1] Since the late 1800s, Maine and Vermont have provided school vouchers to students in certain rural areas that lacked a public school, originally allowing them to attend any public or private school of their choosing in the area at public expense. The Maine program was subsequently limited to public or non-sectarian private schools. There is no clear consensus in the school choice literature regarding whether or not Maine and Vermont’s “town tuitioning” programs are actual voucher programs, though in concept and operation they are similar instruments of parental school choice. [3] For more on the School Choice Demonstration Project’s study mandate, see Appendix A. [4] See Appendix C for more information on the MPCP sample selection process. [5] Previous research on educational choice has used both of these methods. For the potential problems with using “rejects” from school lotteries as a control group, see Witte, 2000, pp. 136-42. However, the problem with waiting lists may be even more severe in that the schools with waiting lists are not randomly assigned across schools. One would have to assume that waiting lists indicate more desirable schools, and thus the full impact of the program (i.e. the full set of private schools) would not be adequately assessed using wait-listed students as a control group. [7]See Appendix B for a detailed discussion of the sample construction. [8]Evidence for neighborhood effects on social outcomes is presented across several social science disciplines. See, e.g., Aaronson (1998) for evidence of neighborhood effects on educational outcomes even after family characteristics are taken into account; Ludwig, Ladd and Duncan (2001) and Leventhal and Brooks-Gunn (2004) for experimental evidence linking neighborhood improvements to improvements in student outcomes; and Sampson, Morenoff and Gannon-Rowley (2002) for a general discussion. See also Cullen, Jacob and Leavitt (2005) for use of census tract information in research on school choice. [9] After creating the random sample of MPS students, we compared this sample to the population of MPS students. We found some significant differences between our random sample and the MPS population. It is unclear if a closer match would be to use a random sample of all MPS students or just those MPS students eligible for reduced free lunch. The MPCP income limit is 175% of the poverty line or less; the reduced free lunch cutoff is 185% of the poverty line. Eighty-three percent of MPS now qualify for free lunch, with a number undoubtedly under identified (especially at the high school level). Complicating the decision is that some students in MPCP can now have incomes up to 220% of the poverty line, but no current data exist on how many. The issue is not critical because our major comparison will be with the matched MPS sample we constructed using the procedures outlined above. For this report we merely included a random sample of all MPS students as another possible comparison. [10] For more information on testing protocols see Appendix C. [11]The Benchmark test is not nationally normed and is used primarily as a classroom diagnostic test given multiple times in a year. We used it for matching purposes because it was all that was available as a measure of baseline achievement for the 9th graders in each sample. We will not include these ninth graders in the LEGS Achievement Study. Whether we include these test scores as covariates in the LEGS Attainment Study will be determined next year when we will have 10th grade WKCE scores for the 9th graders still in school in 2007-08. [12]For more information on the survey protocols, see Appendix C. [13] Calculated by taking averages of the response categories: for the middle four categories we simply took the middle of the range (e.g. respondents in category 3 were given a value of $15,000). For the lowest income category, respondents were given an income value of $3,750, and for the highest category respondents were given a value of $62,500. [14]The 3rd Friday in September is used in Wisconsin as the official enrollment count for all public schools. State aid and other formulas and aid programs depend on this count. [15]The 3rd Friday list included only students who applied, were accepted and were enrolled on that date in the private schools. The students who were dropped were on the original September 1 list, but were not in the schools on September 15th, the third Friday of the month. That is very common in Milwaukee where students often apply to multiple schools under a number of choice programs (e.g. charter schools, magnet schools, and suburban schools). [16]Census tract was selected as the critical neighborhood proxy rather than a distance variable on advice of the Milwaukee Public Housing Authority, and after investigation that confirmed that the census tracts in Milwaukee were carefully drawn to represent neighborhood demarcations. |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Department of Education Reform University of Arkansas 201 Graduate Education Building Fayetteville, AR 72701 http://www.uark.edu/ua/der | Ph: 479/575-3172 Fax: 479/575-3196 edreform@uark.edu |