UARK

 

 

Education Working Paper Archive   

Keyword: School Choice
.pdf version adobe
 

MPCP Longitudinal Educational Growth Study Baseline Report

John Witte
University of Wisconsin-Madison

Patrick J. Wolf
University of Arkansas

Josh M. Cowen
University of Wisconsin-Madison

David J. Fleming
University of Wisconsin-Madison

Juanita Lucas-McLean
Westat

April 7, 2008


Executive Summary

This report focuses on the initial design, implementation and baseline results of the five-year Longitudinal Educational Growth Study (LEGS) of the Milwaukee Parental Choice Program (MPCP) being conducted by the School Choice Demonstration Project (SCDP). The LEGS will be the first evaluation of the participant effects of the MPCP using student-level data to be implemented since the initial pilot program expanded dramatically in 1995. Included this initial report are baseline descriptions of achievement tests for a representative sample of MPCP students in grades 3 through 9, as well as outcomes for comparable samples of students in Milwaukee Public Schools (MPS). Also included are a descriptive analysis of survey results of MPCP parents and a carefully matched sample of MPS parents as well as a brief description of the results of student surveys for both samples.

The first section of the baseline report discusses the construction of a sample of 2,727 MPCP students in grades 3-9. The report also discusses the selection of 2,727 similar Milwaukee Public School students. For both samples the core of this longitudinal study will be to track the educational progress across the two samples through school year 2011-12. We demonstrate that the sample of MPS students constructed by the SCDP is more similar to the representative MPCP sample along demographic and initial achievement criteria than other potential comparison groups of MPS students.

The baseline results indicate that MPCP students in grades 3 to 5 are currently scoring slightly lower on the math and reading portions of the Wisconsin Knowledge and Concepts Examinations (WKCE) than their MPS counterparts. However, no such difference exists for students in grades 6 to 8. Benchmark Test results for 9th graders are also similar between the two groups. The differences in grades 3 to 5 are almost exclusively due to lower MPCP math scores that disappear in grades 6 to 8.

According to our surveys of parents and students, MPCP parents had lower incomes, but higher levels of education than MPS parents. The two groups were also quite similar on how they learned of their child’s school and the qualities they sought in schools. A key difference was that MPCP parents got more information from churches and valued religious instruction more than MPS parents. In both groups, over 70 percent of students were attending their parents’ first choice of schools. Both MPCP and MPS parents and students showed high levels of satisfaction with their schools—in some cases higher than national averages. However, MPCP parents and students were generally more positive about their schooling experience than their counterparts in MPS. MPCP parents were less likely to report problems at school such as school violence, and had slightly higher educational expectations for their children, than comparable MPS parents. Students were also very positive about their schools, differing only slightly in their evaluation of their school climate depending on whether they were in the MPCP or MPS.

The LEGS is the student-based component of the evaluation of the MPCP authorized in WI Act 125, Sec. 8, 119.23 (7)(e), 2 to be carried out by the SCDP based at the University of Arkansas. This project is being funded by a diverse set of philanthropies including the Annie E. Casey, Joyce, Kern Family, Lynde and Harry Bradley, Robertson, and Walton Family foundations. We thank them for their generous support and acknowledge that the actual content of this report is solely the responsibility of the authors and does not necessarily reflect any official positions of the various funding organizations, the University of Arkansas, the University of Wisconsin, or Westat, Inc. We also express our gratitude to MPS, the private schools in the MPCP, and the state Department of Public Instruction for willing cooperation, advice, and assistance.


Introduction

The Milwaukee Parental Choice Program

The Milwaukee Parental Choice (Voucher) Program (MPCP) was the first parental choice program in the United States to allow urban parents to send their children to private schools with public funds covering the entire costs.[1] Often simply referred to as the “Choice” program, it is the largest of 12 voucher programs in the United States. In 2006-07, the baseline year of this report, a total of 17,749 students received vouchers that averaged $6,501 to attend one of 122 participating private schools in Milwaukee.[2]

The MPCP began in September 1990, based on legislation passed the year before as part of the biennial budget bill. For the first time in the United States, urban public school students were allowed to attend private schools with publicly funded vouchers. The program was targeted to low-income students (175% of the poverty line or less) who either had attended Milwaukee Public Schools (MPS) or were not in school the prior year. Only secular private schools were allowed to participate, and they had to limit their voucher students to 49 percent of total enrollment. The program was capped at 1 percent of the MPS student population (approximately 1,000 students in 1990). The voucher was equal to the annual per pupil MPS state aid ($2,446 in 1990). Annual evaluations and reports were required and the program was to terminate with the 1994-95 year.

There have been three major modifications in the program. In 1993-94 the maximum enrollment of Choice students in a school was raised from 49 to 65 percent and the total program cap was raised to 1.5 percent of the MPS enrollment. All sunset provisions were removed. Major changes in 1995 allowed sectarian schools to participate; allowed students who were in private schools in kindergarten through third grade to participate; eliminated any cap on voucher students per school; increased the program cap to 15 percent of MPS students; and eliminated all evaluation and reporting requirements.

Another series of major changes was made in 2005, with Wisconsin Act 125. The students’ prior school criteria were completely dropped, so that any student living in Milwaukee was potentially eligible. The family income cap for continuing Choice students was raised to 220 percent of the poverty line if they had been admitted earlier with an income under 175 percent. The program limit was raised to 22,500 students. Private schools faced a set of new accreditation and testing requirements. Those new testing requirements of Act 125 were coupled with the call for an independent evaluation of the MPCP by the School Choice Demonstration Project, to be reviewed annually by the Legislative Audit Bureau.[3]

The general purposes of this report are two fold. The first is to outline the general research and methodological issues that will guide the research over the five-year period of the Longitudinal Educational Growth Study (LEGS). Those issues and the research design are focused on longitudinal changes in the educational outcomes of comparable students over the period. The second purpose is to present baseline descriptions of MPCP participants both in terms of baseline test scores, from which we will judge growth in achievement, and baseline survey results of samples of both Choice and MPS parents and students.

Prior Research
Prior research on publicly funded voucher programs in the United States has been limited to four locations: Milwaukee, Cleveland, Florida, and the District of Columbia. Privately funded voucher programs have also been studied, with most attention being given to programs in New York City, Washington, D.C. and Dayton, OH (Howell et al, 2002). Despite the limited number of voucher programs, numerous researchers have contributed a considerable body of evidence to the debate over vouchers. The type of research falls into two broad areas: 1) the differences in achievement and other outcomes between students in voucher schools and various comparison groups; and 2) the systemic, “competitive” effects of voucher programs on the remaining public schools. The LEGS deals only with the former question. Future reports from the School Choice Demonstration Project will address the latter area.

The evidence on achievement from prior studies has yet to produce a strong academic consensus, with reported voucher impacts ranging from 0 to statistically significant gains of 3-4 percentile points per year. Prior research on Milwaukee from 1990 to 1994 found mixed results on student achievement, with all researchers agreeing that there were not significant differences between private (voucher) students and a random sample of similarly eligible MPS students on reading scores even after as long as three years in the program. There was considerable disagreement on possible math differences, with the three studies differing over the size of the advantage of private schools (Witte, 2000, chapter 6; Rouse, 1998; Greene et al, 1999). In terms of who participated in the prior Milwaukee voucher program, there was agreement that the private schools did not “cream skim” the best students, although choice parents, while very poor and often heading single-parent households, were more educated, had higher levels of prior parental involvement, and placed somewhat more importance on education than otherwise comparable MPS parents (Witte 2000, chapter 4). Finally, researchers agreed that choice parents were less satisfied with their prior public schools and more satisfied with their subsequent (private) schools than MPS parents.

Studies of other voucher programs have found similar results on parental satisfaction but have mixed, and contested results on student achievement. In Cleveland there appeared to be no overall differences on achievement test scores between voucher recipients and public school students (Metcalf et al, 2003). The same was true in Washington, DC based on the first year outcome report of an experimental evaluation (Wolf et al, 2007). In Florida there are a series of different programs including a vouchers-for-failing-schools program (recently ruled in violation of the Florida constitution); a program for students with disabilities; and a program awarding scholarships to poor students to attend private schools financed by corporate contributions in lieu of some state taxes. Research on these programs has thus far focused primarily on the effects of voucher competition on student achievement in public schools. Three different research teams have all reported that the Florida voucher programs have resulted in public (i.e. non-voucher) students achieving at higher levels, but disagree regarding how much of the public school gains are due to voucher competition and how much are simply due to the desire to avoid the stigma of failure (Figlio and Rouse 2006; West and Peterson 2005; Greene and Winters 2003).

Research Questions in the Longitudinal Educational Growth Study
A number of important research questions surrounding educational choice reside within the domain of “participant” effects that is the focus of the LEGS. They fall into two broad categories. The first concerns the effects of the Choice program on students themselves. The second is an understanding of how parents and students react to choice environments and how these families are impacted by such programs. The research in this study covers both of these areas in which school choice might affect participants.

Program Impacts on Students. There are two general sets of issues that have repeatedly been analyzed in choice intervention studies, as well as studies comparing public and private schools. The two issues deal with educational achievement and attainment. The former, relying on standardized tests as key outcome variables, is often the major focus of education evaluations. To the extent that standardized tests are proxies for meaningful educational outcomes, these studies are necessary and valuable. Achievement gains are also important in that they are linked to what we know absolutely has lifelong value – educational attainment. Attainment is usually measured by years of education completed or by reaching certain milestones, such as high school graduation. Hundreds of studies have linked the number of years of education completed to outcomes such as higher lifetime incomes, and lower chances of living in poverty, incarceration, or having children out of wedlock. Thus both achievement and attainment are crucial outcome measures in studying the effects of educational policies and interventions.

One of the primary research tasks for the LEGS is to compare the pupils in the MPCP to a similar group of students in MPS in terms of growth in both achievement and attainment. The LEGS Achievement study will be based on the progress of students beginning in grades 3 through 8 on the Wisconsin Knowledge and Concepts Examinations (WKCE). The LEGS Attainment study will be based on following 9th graders in 2006-07 through high school and beyond.

Achievement outcomes will be judged by estimating growth in scale scores on the WKCE controlling for prior levels of achievement and other student and family characteristics. In the future, we will look at attainment on a yearly basis in terms of remaining in school and progressing from one grade to the next. We will follow the 2006-07 9th graders through what would typically constitute the four years of their high school education, and measure outcomes such as college enrollment or employment after either completing or dropping out of high school. We will also study how the LEGS components are related. For example, do achievement gains predict high school graduation? What are the effects of grade retention on subsequent achievement gains or on the probability of graduating?

Another major issue that affects a number of aspects of education, but specifically achievement and attainment, is student mobility. Mobility is defined as students moving between schools when they have not reached a terminal grade in a school. Mobility may be between schools in the same system or between school systems or districts. In the last several decades educators and researchers have become keenly aware of the extremely high mobility rates for inner-city students. In the 1990s in Milwaukee such mobility at the elementary school level was judged to be between 22 and 28 percent per year (Witte, 2000, pp. 143-44). Thus we will be measuring and studying student mobility both as an outcome measure and as a variable that we hypothesize will negatively affect both achievement and attainment.

Data for both these studies will come from administrative records, test records, and parental and student surveys of comparable groups. The precise methodology and data collection protocols are described below and in the appendices.
Families and Educational Choice. Another important set of research questions revolves around families and choice environments. There are two broad sets of issues involving families. The first is to try to understand how families make school choices and what impact those choices may have on school enrollments. One critical issue is the characteristics of families who elect to use vouchers to attend private schools. One hypothesis is that families with higher socio-economic status (SES) are more likely to use vouchers, thus leaving the traditional public schools with lower SES families and more difficult to educate children. An alternative hypothesis is that choosing parents will do so because their children are not doing well in traditional public schools and they look to the private schools as a way to improve the educational outcomes of their low-performing children.

As noted above, earlier studies of the MPCP provided some evidence for each hypothesis (Witte 2000). Voucher students averaged beginning test scores that were equal to or lower than comparable MPS students. The Choice families also had considerably lower income and parents were very unhappy with their prior public schools. However, Choice parents also were more educated, spent more time on their children’s education, and valued education more than comparable MPS parents. The research in this study may shed further light on the critical question of whether voucher programs primarily serve advantaged or disadvantaged students.

Family characteristics are also fundamental to understanding achievement and attainment issues. Parent education, parent involvement, and family status all have been linked to higher student achievement and attainment. Thus it is imperative to have controls for these variables in estimating the effects that private or MPS schools may have on achievement and attainment gains.

A second set of family related issues, affecting both parents and students, are the levels and changes in attitudes toward the schools students attend. How do they judge their schools overall? How satisfied are they? How do they “grade” their schools? More specifically, what aspects of schools do they find most and least effective? The answers to these questions will provide us with clues as to what parents believe to be most important in schools, and if programs like the choice program allow more families to find schools with the qualities they desire. Given that higher income families have more residential and hence public school choice, this study will seek to determine if the Choice program facilitates parental matching of schools for lower income families. The answer to that question may come from parent surveys over time, but also from student mobility between schools and into and out of the program.

METHODOLOGY

One of our first research tasks was to identify two comparable samples of students—one from the MPCP and one from MPS. Samples were required (rather than entire populations) because of resource limitations on surveying parents, testing students, and tracking students over time. After estimating the size of the samples needed for identifying any statistically significant results after five years (given anticipated attrition), a grade-stratified random sample of MPCP students was drawn for grades 3 to 8. Those grades were selected because MPS administers the Wisconsin Knowledge and Concepts Examinations (WKCE) annually in those grades for federal accountability purposes. We selected all the Choice students in the 9th grade to follow so that we had a large enough sample to withstand attrition for our five-year attainment study. [4]

The question then was what to use for a comparable sample of MPS students. The ideal arrangement would be to have an experimental design in which all students desiring a voucher would be randomly selected to receive one or not. However, there is no statutory provision for random assignment under Wisconsin Act 125. Furthermore, since the student limits on the program had been increased in anticipation of increasing demand, it was impossible to construct a comparable group from waiting lists or students not picked in school and grade-level lotteries.[5]

Because the study will rely on changes over time in measures of achievement and attainment, it was important to be able to match samples on three critical criteria: 1) current levels of achievement, indicated by baseline test scores; 2) observable demographic characteristics known by researchers to be related to educational outcomes, and 3) variables that might serve as proxies for unobservable factors that may affect the comparable outcomes for each group.[6] To meet these criteria, we developed a unique and innovative approach in the construction of a sample of MPS students comparable to the representative MPCP sample. We used a combination of matching on specific student characteristics and a method of matching based on the propensity of certain students to participate in the program. Sample selection thus involved a three-stage process.

In the first stage we matched students in the same grade on their neighborhoods in Milwaukee. We did this in sequence for each student in the sample of MPCP students. Following the advice of demographers and city planners, we used census tracts to identify student neighborhoods. Census tracts are drawn by the U.S. Census Bureau to follow neighborhood boundaries. In our sample, MPCP students come from 213 different census tracts. In this stage, for any given MPCP student in our sample, we restricted the list of potential MPS matches to students in the same tract. One indication that our categorization of neighborhoods has substantive support comes from the survey results outlined in Appendix D (question 58). These results indicate nearly identical responses from MPCP and MPS parents regarding the safety of their neighborhoods. In particular, roughly 8 percent of respondents from both groups designated their neighborhoods “very unsafe,” while 20 percent responded to the “very safe” designation.

In the second stage, we matched students within census tracts within the same 5 percentile bandwidth of WKCE baseline test scores. In the final stage, if more than one MPS student was matched to the MPCP student based on census tract and test scores, or if there was missing values for either variable for an MPCP student, we matched by estimating propensity scores (Rosenbaum and Rubin 1983). The propensity score matching process identifies characteristics commonly associated with enrollment in the MPCP that are present in MPS students, making those public school students a good “match” for Choice students. In this step, we estimated the propensity of MPCP participation as a function of the mean of math and reading baseline test scores, gender, race and an indicator for students with English Language Learning status. The MPS student with the closest propensity score to the MPCP student was then selected. If missing predictors made it impossible to construct a propensity score for the MPCP student, a MPS student was selected at random from those remaining after matching on census tract and prior test. If the missing predictor was student test score, matches were made at random within tract. [7]

Propensity scores have been increasingly used in recent evaluation research when random assignment is not possible. We believe that by first selecting students using neighborhood and prior test scores and then using propensity scores to break ties or augment matches, we improve on simply matching on or controlling for baseline tests scores and therefore reduce the threat of selection bias. The leverage on selection effects hinges on the assumption that similar neighborhoods will serve as a control for unobserved variables that may affect outcomes. [8]

Although our future models for estimating gains in achievement and attainment will also include a large number of observed control variables that we know affect achievement, it is important to note that our matching method results in a comparison sample of MPS students with very similar observable characteristics to the MPCP sample. As indicated below in Table 1 and Figure 1 our matched MPS sample is closer to the MPCP sample on test scores, gender, and race than to an obvious alternative comparison group: a random sample of MPS students.[9]

Figure 1 indicates that for test scores for students in grades 3, 4 and 5, the MPS students matched via our algorithm (MPS Matched) are considerably closer to our MPCP sample than scores from a random sample of MPS students. Test scores for the various groups converge in the latter grades. If the goal for selecting a comparison group is to simply align baseline test scores as closely as possible to ensure that any diverging scores in future years are compared to a similar starting point, our method is at least as accurate as other methods in grades 6, 7 and 8, and preferable to other methods for the earlier grades. The fact that, in our comparison group, students’ neighborhood, race, gender, ELL status and the joint effect of these characteristics on school choice (students’ propensity scores) are accounted for further supports our method.

Figure 1 (click for a larger view)

Figure 1

Table 1 displays demographic data on students in our MPCP sample and students in both of the potential comparison groups. Perhaps the most important statistic is the proportion of white students in each sample, which indicates disproportionately high participation in MPCP among minority students. Nine percent of students in the MPCP sample and our MPS Match sample are white compared to 15 percent in the MPS Random sample. In both the MPCP and MPS Matched samples, African-Americans represent roughly two-thirds of the students, compared to 60 percent in the Random MPS sample. Compared to the MPCP sample, Hispanics are slightly underrepresented in the MPS Matched sample, where the 20 percent of students classified as Hispanic is closer to the MPS Random sample. The MPCP sample and our MPS Matched sample are more similar in gender and ELL breakdowns than the MPCP sample is to the MPS Random sample.

Table 1: Comparisons of Student Characteristics in Three Possible Study Samples, 2006-07

Sample

Black

Hispanic

White

Female

ELL

FRL

MPCP

0.66

0.23

0.09

0.55

0.11

0.87

MPS-Matched

0.67

0.20

0.09

0.53

0.10

0.86

MPS-Random

0.60

0.20

0.15

0.48

0.13

0.84

Note: Cell values represent proportions of the total sample.

BASELINE DESCRIPTIVES

A Snapshot in Time: Baseline Test Scores in MPS and MPCP

The matching procedure was necessary to produce appropriate samples for comparisons between MPS and MPCP students in subsequent years of the study. Because the matching procedure included neighborhood, student characteristics, and baseline test scores, future reports will be able to estimate the effects of MPCP participation on student educational growth. These “value-added” estimates for individual students are now the standard in estimating the effects of education interventions. What we will be estimating in the future are gains in educational achievement for individual students, comparing the results for the MPCP Random sample and MPS Matched sample. In this report, however, only a general description of the baseline test scores of the MPCP sample, the MPS Matched sample and the MPS Random sample of students are possible. This description is simply a “snapshot” in time and cannot be used as evidence of any effect of MPCP participation on student achievement.

Table 2 reports average math and reading scale scores on the Wisconsin Knowledge and Concepts Examinations (WKCE) for 3rd through 8th graders among MPCP, MPS Matched students and a grade-stratified random sample of MPS students.[10] We include the random MPS group for additional comparison and to further indicate the success of our matching procedure. The table also reports the combined scale score—the mean of math and reading scores—for the three groups. Results from difference-in-means tests between MPCP and the two MPS groups are indicated in the table key.

Table 2: Sample WKCE Scores (Grades 3-8), 2006-07


Grade

Sample

N

Mean Reading Scale Score

Reading SD

Mean Math Scale Score

Math SD

Combined Mean Scale

Combined SD

3

MPCP

341

427.9

44.1

380.5

48.1

404.1

42.6

3

MPS-Matched

341

429.9

42.9

390.1**

47.8

410.1*

41.1

3

MPS-Random

341

440.3***

38.2

401.8***

49.8

421.7***

40.3

4

MPCP

324

436.0

49.4

414.8

49.8

425.3

44.7

4

MPS-Matched

324

437.5

49.8

423.0**

49.9

430.0

45.2

4

MPS-Random

324

447.2***

53.3

434.4***

46.6

440.8***

46.5

5

MPCP

338

441.8

47.2

437.9

43.2

439.6

41.9

5

MPS-Matched

338

440.6

51.1

444.6**

41.5

442.7

41.0

5

MPS-Random

338

448.2

56.4

452.4***

44.3

450.8

46.7

6

MPCP

330

463.7

48.4

467.5

38.5

465.4

39.8

6

MPS-Matched

330

466.3

50.0

471.5

42.5

468.9

41.4

6

MPS-Random

330

464.3

54.3

469.4

43.7

467.9

43.8

7

MPCP

303

472.0

51.0

492.1

44.9

481.3

46.6

7

MPS-Matched

303

467.9

49.8

494.2

41.9

481.1

42.5

7

MPS-Random

303

476.7

48.9

498.6*

46.5

488.1*

44.0

8

MPCP

290

487.2

53.9

495.9

42.9

490.7

44.0

8

MPS-Matched

290

483.6

58.8

500.3

46.6

492.2

47.6

8

MPS-Random

290

488.4

55.3

497.3

50.1

493.0

49.5

TOTAL

MPCP

1926

453.5

53.1

445.7

61.1

449.4

53.0

TOTAL

MPS-Matched

1926

453.5

53.9

452.1***

59.6

453.0**

51.8

TOTAL

MPS-Random

1926

460.0***

54.0

457.2***

58.0

459.2***

51.6

Note: “SD” stands for standard deviation.
***Different from MPCP at p<0.01
**Different from MPCP at p<0.05
*Different from MPCP at p<0.10

As expected by our algorithm, the MPCP students are generally very close to the MPS matched sample on most tests. And where there is a statistically different mean, the mean differences are quite small. Thus, as outlined in the last section, our algorithm for constructing the MPS Matched sample worked very well.

When compared to the MPS random sample, the MPCP Matched test scores are generally closer to the MPCP scores. When there are statistically significant differences between MPCP and MPS Matched test scores, the MPS Matched scores are still closer to the MPCP scores than the MPS random sample scores are. Difference of means tests indicate that test scores in reading for students in grades 3 to 5 were generally higher in 2006 for MPS students than for MPCP students, but that these differences disappear for later grades. The trend in math is slightly more persistent, as MPS students also score higher in grade 7. Slight but significant differences between MPCP and MPS Matched students are evident in math achievement for grades 3 to 5. However, with one exception (grade 3), these differences are not found in the combined score.

The previously discussed Figure 1, and Figures 2 and 3 below, provide pictorial summaries of the results reported in Table 2. We note three patterns. The first is that scale scores in each grade and each subject increase by grade. The second feature is the apparent gap between MPCP and MPS students in early years in math achievement, and the relative lack of such a difference in reading. That is consistent with the data in Table 2. The final pattern is the convergence of math and reading scores between each group as grade increases. Although a slight difference in reading between MPCP and MPS students is apparent in grade 3, and a pronounced difference in math in grades 3-5, these differences nearly disappear in grades 6, 7 and 8. Thus whatever baseline differences there are between these groups they are due to lower math scores in the early grades for the MPCP students.

Figure 2 (click for a larger view)

Figure 2

Ninth graders in MPS do not take the WKCE. Instead they are administered the Benchmark Exam, which we similarly administered to the 801 MPCP 9th graders in our sample. Because we matched test scores by grade, this did not require a separate matching procedure, although the test scales were different (scale scores for the WKCE; percent proficient in subject for the Benchmark). [11] We report the results of 9th grade Benchmark scores in Table 3. As in earlier grades, the MPCP 9th graders are closer to the MPS Match than to the random sample of MPS students in the same grade. Unlike the earlier grades, however, MPCP 9th graders appear to exceed their MPS counterparts in one subject area (reading), and this difference carries over to the combined math/reading scores. Observed with the pattern of converging scores (albeit from separate testing instruments) evident in Figures 1 to 3, these results indicate relative parity between MPCP students, the MPS Match students, and the MPS population in the later grades. Across the grade levels, there is little evidence that MPCP schools are “cream-skimming” the “best” students, at least in terms of test scores.

Table 3: Sample Benchmark Scores (Grade 9), 2006-07


Grade

Sample

N

Mean Reading

Reading SD

Mean Math

Math SD

Combined

Combined SD

9

MPCP

801

49.6

18.8

42.6

16.7

46.1

15.6

9

MPS-Matched

801

46.2***

19.1

43.9

18.2

44.8

16.3

9

MPS-Random

801

44.6***

20.6

42.8

18.9

42.7***

18.2

***Different from MPCP at p<0.01
**Different from MPCP at p<0.05
*Different from MPCP at p<0.10

Parent and Student Views: MPCP and MPS Survey Results

Introduction

In order to more fully understand the school choice environment in Milwaukee, we surveyed the parents and students of our MPCP sample and the MPS Matched sample. Westat telephoned MPCP and MPS Matched parents. MPCP students in grades 4 to 9 and their MPS matches also completed surveys subject to parental consent. During the testing period, MPCP students were given a written survey to be completed on their answer key. Those students who did not complete the survey at this time were noted and were later telephoned, as were all MPS Matched students. [12]

The response rates for the parent and student surveys are presented in Table 4. For the MPCP parent survey, two different samples are offered. The first MPCP column includes response rates for the parents of students in the final sample (n = 2727) plus those parents who refused to have their children participate in the testing portion of the study (n = 134). Although we did not analyze the test scores for any student whose parent refused participation in the study, we did include these parents and their children in the survey sample. The total completed survey response rate for this sample was 64.9 with 12.3 percent of the sample completing surveys in Spanish. This rate is quite impressive given the mobility of the survey population. The second MPCP sample pertains to the response rates for the final sample only (n = 2727). Excluding study refusals the MPCP response rate was 65.4 percent. The response rate for MPS parents was 51.6 percent. The MPS survey was conducted later in the year, so it is unsurprising that the number of parents who were not locatable was higher (39.3) in the MPS sample than the MPCP sample. It is encouraging that only 1.8 percent of MPCP parents and 3.9 percent of MPS parents outright refused to be interviewed once they were contacted by survey administrators.

The response rates for the student survey are in the bottom of Table 4. Since third grade students were not surveyed, the sample sizes are smaller for the student survey than the parental survey. MPCP students could complete the survey in one of two ways: with paper and pencil at the time of test taking, or in a phone survey. Almost three-quarters of MPCP students responded on the paper survey. The total completed survey response rate is 83 percent for the total sample. All of the MPS student surveys were completed over the phone. The MPS student response rate is 46.6 with 4.0 percent of MPS student surveys completed in Spanish.

Table 4: Response Rates 2006-2007

 

 

 

 

 

 

 

Parental Survey

MPCP

MPCP

MPS

 

Total Sample With Testing Parental Refusals

Final Sample

Matched Sample

Completed-Total

64.9

65.4

51.6

Completed-English

52.5

52.6

47.0

Completed-Spanish

12.3

12.8

4.6

Partial Completion

0.1

0.2

1.1

Ineligible

0.8

0.8

0.4

Language Problem

0.2

0.2

0.4

Refused to be Interviewed

2.0

1.8

3.9

Not Locatable

23.0

22.5

39.3

Missing

9.0

9.1

3.2

TOTAL (N)

2,861

2,727

2,727

 

 

 

 

 

 

 

 

Student Survey

MPCP

MPCP

MPS

 

Total Sample With Testing Parental Refusals

Final Sample

Matched Sample

Completed-Total

83.0

84.5

46.6

Completed-Paper Survey

74.6

77.8

 

Completed-Phone Survey

8.4

6.8

46.6^

Child Disabled

0.1

0.1

0.5

Child Moved

0.7

0.7

2.2

Ineligible

0.6

0.7

0.6

Language Problem

0.0

0.0

0.5

Refused to be Interviewed

0.3

0.3

4.7

Not Locatable

8.0

6.6

41.2

Missing

7.2

7.1

3.7

TOTAL (N)*

2,499

2,386

2,386

Note: The cell values represent percentages.

 

 

*There are fewer student surveys than parental surveys because 3rd graders were not surveyed. 

^4.0 percent of the MPS student surveys were completed in Spanish.

 

The survey results provide us with a nuanced understanding of education circumstances. The main purposes of the survey were: (1) to understand who participates in the program; (2) to understand the school choice environment in the city of Milwaukee; (3) to understand parental participation in and expectations for their children’s achievement; and (4) to gauge levels of parental satisfaction with their children’s educational experiences. In the future, we will be able to use information from the surveys to model growth in student achievement.

Who Participates in the Choice Program?

The complete results of the parental survey are available in Appendix D. The MPCP parental results include 1,856 fully completed surveys and 4 partially completed surveys from those in the total sample (including testing parental refusals). Results from 1,438 completed surveys from MPS parents are also included. Analyzing the demographic characteristics of parents provides some insight into who chooses and who does not. However, one must remember that we matched students based on test scores, census tracts and, in some cases, student demographic information. Therefore, we did not expect large differences between MPCP and MPS parents, at least not at baseline.

Race. The majority of survey respondents are African-American (Q37). While a greater percentage of MPS (62.5 percent) respondents were African-American when compared to the MPCP (57.6 percent), Hispanics make up a greater proportion of MPCP respondents (24.5 percent) than in MPS (18.7). The number of white parent respondents is similar in each program.

What is your ethnicity?

 

MPCP

MPS

American Indian

3.3

2.2

Asian or Pacific Islander

2.1

2.8

Black

57.6

62.5

Hispanic

24.5

18.7

White

15.8

14.3

Other/Refused/Don't Know

1.5

3.0

Note: respondents could select more than one category


Education. There are some important differences when examining the highest education level of respondents (Q38). About 51 percent of MPCP respondents have a high school diploma or less, while 14.0 percent have at least a four-year college degree. On the other hand, MPS respondents are comparatively less-well educated with roughly 57 percent of parents having a high school diploma or less and 11.1 percent having a college degree or more. This result is consistent with Witte’s (2000) previous analysis of the Milwaukee choice program from 1990 to 1994. Witte found that 45 percent of mothers with a child enrolled in the MPCP were a high school graduate or less, while only 8 percent had a four-year college diploma or post-graduate work. One of the major differences in Witte’s (2000) results and ours is that Witte found that 46 percent of MPCP mothers had some college education, while only 30 percent of MPCP parents and 26.3 percent of MPS parents in our sample are in the “some college education” category. Thus, although MPCP parents are somewhat more educated than MPS parents, both groups are less well-educated now than in the 1990-94 period.

What is the highest educational level that you have completed?

                               

MPCP

MPS

Eighth grade or below

10.0

5.6

Some high school

12.6

17.4

GED

3.0

3.8

High school graduate

25.2

30.3

Post graduate (technical school)

4.0

2.8

Some college

30.0

26.3

4 year college degree

10.8

7.8

Post-graduate work

3.2

3.3

Other/Refused/Don't Know

1.2

2.7

Employment and Income. Although there are differences in the education levels of MPCP and MPS respondents, an examination of the employment questions finds no large differences (Q39 and Q40). When asked about the government assistance received by anyone in the household, MPCP and MPS respondents tended to reply similarly (Q59). However, MPS respondents were slightly more likely to receive assistance from various government programs.

Given that there is an income cap in the voucher program, it is not surprising that MPS respondents have a higher income, on average (Q43). There is very little difference between voucher and public school parents at very low levels of income, but, at the other end of the distribution, about 15 percent of MPS respondents have a total household income of more than $50,000, while only 4.7 percent of MPCP parents had an income in that highest category. Using these figures, we estimate the average income of MPCP parents to be $23,371 and the average income of MPS parents to be $27,577.[13]

Including everyone in your household, what was the
total income for your household in the last calendar year
(before taxes and other deductions)?

 

MPCP

MPS

Less than $5,000

8.5

9.9

Between $5,001 and $10,000

12.3

12.5

Between $10,001 and $20,000

23.9

18.0

Between $20,001 and $35,000

31.1

23.7

Between $35,001 and $50,000

13.0

12.9

$50,001 or more

4.7

15.0

Other/Refused/Don't Know

6.5

8.0



Home Lives. The survey results allow us some insight into the home lives of the MPCP and MPS students. We find that MPCP students are about as likely as MPS students to be living with married parents (38.2 percent vs. 35.7 percent). This is an increase for both groups from 1990-1994 (23 percent for MPCP, 35 percent for low-income MPS families). According to our survey, about 57 percent of MPCP and MPS respondents said that the child’s other parent or guardian does not live in the household (Q42). There is almost no difference in family size for the two groups, while MPCP respondents are slightly more likely to own their own home (41 percent vs. 38 percent) (Q56). Given that there are more Hispanics in the MPCP sample than the MPS sample, it is unsurprising that more MPCP respondents said that they speak Spanish at home with their children (19.3 percent) than MPS respondents (11.3 percent).

Home Life 2006-2007

 

MPCP

MPS

Married

38.2

35.7

Single parent household

57.9

57.4

Spanish spoken in household

19.3

11.3

Lived in Milwaukee more than 25 years

50.8

53.4

Homeowners

41.0

38.0

Finally, there are few differences between and MPS and MPCP respondents when asked how long they have lived at their current address (Q55). When asked how long they have lived in Milwaukee, the majority of both MPS and MPCP parents answered more than 25 years (Q57).

Religion. Since 1996 religion and religious schools have played an integral role in the Choice program. We asked respondents several questions regarding the role of religion in their lives. In terms of religious preferences, there are more Catholics (30.0 percent vs. 20.5 percent) and Lutherans (9.1 percent vs 3.3 percent) in MPCP than MPS (Q52). Twelve percent of MPS parents said that they had no religious affiliation, while only 5.9 percent of MPCP parents answered similarly. When trying to understand why some parents participate in the choice program and some do not, the level of religiosity may be as important as parents’ religious preferences. MPCP respondents are more religious than MPS respondents (Q54). Almost two-thirds of the MPCP parents said that they attend religious services once a week or more, while only about 54 percent of MPS parents said the same.

What is your religious preference?

 

MPCP

MPS

7th Day Adventist

1.1

0.4

Apostolic/Pentecostal

7.2

7.2

Catholic

30.0

20.5

Christian, Non-Denominational

16.8

20.1

Church of God in Christ

2.5

2.2

Islamic

2.9

1.0

Jewish

1.1

0.1

Lutheran

6.3

3.3

Baptist

21.2

24.3

None

5.9

12.0

Other/Refused/Don't Know

5.0

8.9

The Choice Environment

Tenure in Schools. Beyond demographic information, the survey results provide a deeper understanding of the education environment in Milwaukee. For example, there is great mobility in Milwaukee’s schools. About 39 percent of MPCP students and 37 percent of MPS students have been at their school for one year or less (Q2). On the other hand, 41.2 percent of MPCP students have been in their schools for three or more years. There are, however, clearly some highly mobile families in that 28.2 percent of MPS parents and 21.3 percent of MPCP parents said that their child has attended four or more schools (Q20).

How many years has your child been at this school?

 

MPCP

MPS

Less than 1 Year

10.9

3.7

1 Year

28.1

33.1

2 Years

13.5

20.7

3 Years

15.3

14.5

4 Years

9.8

7.8

More than 5 Years

22.1

20.1

Other/Refused/Don't Know

0.3

0.1



Choosing Schools. The many and varied school choice options for parents to consider make Milwaukee an interesting case study. Less than one half (45.1 percent) of MPS students attend their residentially assigned school (Q4). Before choosing a school, MPCP parents visited or contacted about one public school and one private school, whereas MPS parents contacted or visited 1.6 public schools but only .3 private schools, on average (Q18).

Is this your neighborhood assigned
or residentially assigned school?

 

MPS

Yes

45.1

No

52.8

Other/Refused/Don't Know

2.1

Parental information levels are an important component of school choice programs. About 38 percent of MPCP parents learned about the voucher program from friends or relatives (Q7, first response). In the same way, over 33 percent responded that they heard of the program from their school. The importance of social networks is also evident in choosing a specific school. When asked how they initially heard of their child’s school, the first response of 54 percent of MPCP parents and 42 percent of MPS parents was from friends or relatives (Q15). The major difference between the groups regarding how they heard about their current schools is that MPCP parents received information from their church (9.4 percent) or from other private schools (4.5 percent) while MPS parents did not.

On the MPS side, roughly 60 percent of parents had heard of the Milwaukee Parental Choice Program but only 13.8 percent of them had applied for a voucher at one point (Q8, Q9). Of those that applied, roughly 14 percent had children that previously participated in the MPCP, while about 15 percent were ineligible for the program (Q10). We also asked those who did not apply to the MPCP why they did not apply (Q11). The majority blamed a lack of information: 43.3 percent said they did not know the MPCP existed, 18.6 percent said they did not know enough about the program, and 1 percent said they did not know enough about individual Choice schools. Another one-fifth of MPS respondents said that they did not apply to the MPCP because they were satisfied with their current school.

How did you learn about the Parental Choice Program, also known as the voucher program?

MPCP

                                 

1st Response

2nd Response

Friends or Relatives

38.4

24.4

My Child's School

33.5

32.0

Other Private Schools

5.2

10.5

Newspaper/Magazine

4.4

8.1

Church

4.0

4.7

Television or Radio

3.6

12.2

Community Center

1.2

1.7

Internet

0.7

0.6

Other 

8.5

5.8

Refused/Don't Know

0.5

0.0

 

How did you initially hear about your child’s current school?

 

MPCP

MPS

Friends or Relatives

54.0

41.6

Church

9.4

0.3

Other Private Schools

4.5

----

Flyers/Brochures

3.7

4.6

Call From School

2.8

3.9

Community Center

2.2

2.2

Community Events

1.7

0.5

Newspaper/Magazine

1.3

0.9

Home Visit

1.0

0.2

Television or Radio

0.9

0.8

Internet

0.6

0.8

Other

17.3

43.7

Refused/Don't Know

0.6

1.1

Note: Parents asked to choose all that apply. First response.

Charter schools play a prominent role in Milwaukee’s education environment. MPS parents (72.0 percent) were more likely than MPCP parents (65.1 percent) to have heard of these public schools of choice (Q12). Of those who have heard of charter schools, roughly one-fifth of both MPCP and MPS parents have applied to a charter school (Q13). About 7 percent of MPCP parents and 10 percent of MPS respondents said that their child has attended a charter school.

What do parents want in their children’s schools? The survey results indicate that MPCP parents and MPS parents have similar desires (Q16). The educational quality of the school is by far the most important characteristic, while student safety and teacher quality are also quite important (Q17). The largest difference between MPCP and MPS parents concerns the importance of religious instruction. About 9 percent of MPCP parents considered religious instruction their most important criterion, while only 1.7 percent of MPS parents did. In a school choice environment with many possible options, roughly three-fourths of both MPCP and MPS parents said that their child’s school was their first choice (Q3).

Which school characteristic is the most important?

 

MPCP

MPS

Educational quality of the school

53.2

48.5

Teacher quality

11.9

16.4

Safety in the school

11.3

16.6

Religious instruction

8.6

1.7

Discipline in the school

4.3

3.6

School leadership

2.7

3.0

Financial considerations

1.6

0.4

Class size

1.6

2.6

Location of the school

1.3

1.9

Extracurricular activities (sports, etc.)

1.3

1.0

Racial diversity

0.9

1.5

School facilities (library,  gym, textbooks)

0.5

0.6

Special programs offered by the school

0.3

1.5

Other children in the family attending the same school

0.2

0.2

Other/Refused/Don't Know

0.3

0.5

Was this school your first choice for your child?

 

MPCP

MPS

Yes

77.9

74.1

No

21.7

23.9

Other/Refused/Don't Know

0.4

2.0

Parental Involvement, Expectations, and Importance of Education

The survey included several measures of parental involvement. MPCP parents were more involved in school activities than MPS parents, including volunteering at their child’s school, attending parent/teacher conferences, taking part in the activities of a parent/teacher organization, and belonging to other education organizations (Q26). In terms of contact with schools (Q25), results were mixed between the two groups. Both groups were contacted most often concerning their child’s academic performance. MPCP parents were more likely to be contacted for volunteering or fundraising.

Did you (or someone in your household) do any of the following at your child’s school this past year?

MPCP

MPS

Volunteer at your child’s school

54.7

37.9

Attend parent/teacher conferences

94.5

91.8

Take part in activities of a parent/teacher organization

47.4

32.1

Belong to other organizations dealing with school matters

26.7

18.5

Note: cells report percentages of respondents answering “yes.”

We also asked parents how many times in a normal week they participate in activities that are educationally beneficial to their children. On this measure, MPS respondents are more likely than MPCP respondents to participate in these types of behaviors (Q27). About 74 percent of MPS parents said that they help their child with their homework three or more times a week, while 62 percent of MPCP parents do the same. Likewise, MPS parents were more active than MPCP parents in reading with their children, working on math, helping with writing and watching educational television programs. One might think that MPS parents are more active than MPCP parents because MPS students are assigned more homework. We find no evidence of this, as MPCP students report spending more time on homework than MPS students (Q29).

Weekly Participation in Child’s School-Related Activities*

 

MPCP

MPS

Help your child with homework

62.0

73.7

Read with or to your child

45.9

55.2

Work on arithmetic or math

46.1

59.3

Work on penmanship or writing

30.8

44.2

Watch educational programs on TV with your child

40.2

45.7

When asked about the educational expectations they have for their child, MPCP parents were more positive than their MPS counterparts. Over 54 percent of MPCP parents expect their child to graduate from college or go to graduate school, whereas only 46.7 percent of MPS parents expect the same (Q33). Likewise, 16.2 percent of MPS parents believe their child will achieve a high school diploma or less, as opposed to 8 percent of MPCP parents. In addition, MPCP parents believed their school had higher expectations for their children. Roughly 47 percent of Choice parents strongly agreed with the statement that their child’s school has high expectations for academic achievement compared to 36.1 percent of MPS respondents (Q31).

How far do you expect your child to go in school?

 

MPCP

MPS

Finished some high school

0.3

1.7

Graduated from high school

7.6

14.5

Go to vocational school after high school

1.3

1.9

Go to college

35.7

33.5

Graduate from college

29.7

33.7

Go to graduate school (law, medicine, masters degree)

24.5

13.0

Other/Refused/Don't Know

0.9

1.7

In order to measure how important education is to parents, we asked them to compare the importance of education to other goals (Q34). For example, about three-fourths of both MPCP and MPS parents said that education is more important than having a good job. Examining the survey results for these questions, one finds few differences between MPCP and MPS respondents. Overall, MPCP parents are slightly more likely to say that education is just as important as these other goals. In another attempt to determine the importance of education to parents, we asked them if they have ever moved so that their children may attend a better school. We found that this is a relatively rare occurrence, as only 15.7 percent of MPCP respondents and 14.1 percent of MPS respondents said that they moved for this reason (Q30).

How would you rate the importance of education in family compared to other goals?

                                 

Education is more important

Education is just as important

                                 

MPCP

MPS

MPCP

MPS

Having a good job

75.1

77.0

23.8

21.8

Having enough money in the family

59.0

61.9

37.5

33.8

Maintaining religious observances/faith

31.7

33.5

56.8

52.4

Maintaining family ethnic traditions

35.5

36.9

57.6

53.8

Having a healthy family

20.4

18.4

62.7

61.8

Having a good place to live

30.2

25.1

63.0

64.8

Parental Reports of Student Success

The survey results provide us with the opportunity to learn more about our MPCP and MPS students beyond what we know from administrative data. Asked what was the average grade their child received in school this past year, three-fourths of MPCP parents and 64 percent of MPS parents said their child generally received As or Bs (Q32). According to the results from this question, it appears as if MPCP students in our sample received slightly higher grades than their MPS matches (MPCP GPA: 3.0 vs. MPS GPA: 2.8).

What is the average grade your child
received in school this past year?

 

MPCP

MPS

A

30.4

22.5

B

45.1

41.8

C

19.3

26.6

D

2.3

4.4

F

0.9

1.6

Other/Refused/Don't Know

2.0

3.1

GPA

3.0

2.8

A relatively small percentage of MPCP students (2.7 percent) and MPS students (4.2 percent) in our sample have physical handicaps (Q46). Those parents with handicapped children were also asked how well their child’s school meets their child’s needs. MPS parents seemed more satisfied, as 51.7 percent of MPS parents said that the facilities met their child’s needs very well, while only 44 percent of MPCP parents responded similarly (Q47).

While there was only a small difference in regard to physical handicaps, there is a larger difference between MPS and MPCP students with regards to the prevalence of learning disabilities (Q48). The percentage of respondents who said that their child has a learning disability is twice as large in the MPS sample (18.2 percent) than in the MPCP sample (8.7). Discussions with our Milwaukee Principal Advisory Panel indicated that MPCP school personnel are less likely to identify slow learners specifically as “learning disabled” than are MPS school personnel. It is possible that some or even all of this large difference in the reported rates of learning disabled students across the two groups is due to this difference in labeling practices and not necessarily because MPCP schools are serving fewer learning disabled students. There is very little difference in opinions between public and choice school parents regarding how well the school meets their children’s needs regarding learning disabilities (Q49).

According to the parent survey, MPS students were more likely to miss school and have been suspended than MPCP students. About 29 percent of MPS respondents said that their child missed at least 3 days of school in the last month as opposed to only 17.4 percent of MPCP parents (Q50). During the past year, 33.5 percent of the children of the MPS respondents were suspended for disciplinary reasons (Q51). Less than 20 percent of the MPCP respondents’ children were suspended in that same time frame.

Parental Reports of School Satisfaction

Using the survey results, we can examine children’s educational experiences, as well as the relationships between parents and schools. Perhaps one of the most important measures of school effectiveness is parental satisfaction. We asked parents a battery of questions regarding their levels of satisfaction with their child’s school across a variety of school characteristics (Q22). Overall, the results reveal that parents are quite satisfied in both MPCP and MPS. For all but one of the 15 characteristics, the majority of both MPS and MPCP parents said that they were very satisfied or satisfied. The one exception concerns how much students can observe religious traditions in schools, as MPS parents are relatively less satisfied. While both MPS and MPCP parents appear quite satisfied with their children’s schools, MPCP parents are relatively more satisfied. This discrepancy arises as MPCP parents often said they were “very satisfied” with school characteristics, while MPS parents were generally “satisfied.”

Thinking specifically about your child’s school, how satisfied are you with each of the following?

                                 

Very Satisfied

Satisfied

                                

MPCP

MPS

MPCP

MPS

What is taught in school

54.9

35.3

36.8

49.4

School safety

55.2

36.0

36.7

45.8

Amount your child has learned