Working Paper: School Choice Lotteries and Validity | Teachers College Columbia University

Skip to content Skip to main navigation
Teachers College, Columbia University
Printer-friendly Version
Teachers College, Columbia Univeristy Logo

Working Paper: School Choice Lotteries and Validity

One of the most important questions for policymakers considering the merits of school choice programs is whether students in school choice program outperform their traditional public school counterparts. This question, however, has proven difficult to answer because the type of student who exercises choice may be fundamentally different than the type of student who does not. As such, researchers employ econometric techniques to control for observable differences between students across school sectors. Any remaining differences in achievement are therefore assumed to be attributable to the school choice program itself.

By: Ron Zimmer and John Engberg
Working Paper No. 220

One of the most important questions for policymakers considering the merits of school choice programs is whether students in school choice program outperform their traditional public school counterparts. This question, however, has proven difficult to answer because the type of student who exercises choice may be fundamentally different than the type of student who does not. As such, researchers employ econometric techniques to control for observable differences between students across school sectors. Any remaining differences in achievement are therefore assumed to be attributable to the school choice program itself.

A frequently used technique to control for differences between students who attend a selected school versus those who did not is a school choice lottery in which seats are randomly awarded if there are more applicants than seats. The theory is that if students are randomly assigned to enroll in a chosen school via lottery, the assignment may be analyzed similarly to a randomized controlled trial (RCT), which is generally considered a strong method of estimating the impacts of a program or policy. After seats are awarded by lottery, any remaining differences in achievement between lottery losers, most of whom remain in traditional public schools, and lottery winners, most of whom attend schools of choice, are considered effects of the school choice program itself.

In this paper Ron Zimmer and John Engberg argue that lottery analyses have an important limitation that can and should be tested empirically: not all students in school choice programs enter via lottery. As such, the results of a lottery study do not necessarily generalize to all students in a school choice program. Specifically, the authors present two scenarios that frequently occur in school choice programs but are not directly addressed in most lottery analyses: (1) undersubscribed schools of choice (which admit students on a first-come, first-served basis) and (2) a school of choice in which some students are enrolled in the school prior to the assignment of open seats via lottery.

While their analyses show limited evidence that results differ between lottery and non-lottery students in the magnet school program they study, the authors' more general conclusion is that generalizability is a significant concern that is often untested in some of the most prominent school choice lottery studies to date. The authors propose two types of sensitivity tests to address these concerns in future research: (1) compare achievement growth of students who enter via lottery with that of students who entered without a lottery; (2) compare the achievement growth of students in undersubscribed schools of choice with growth of students in oversubscribed schools of choice. These tests would help policy makers better understand the applicability of lottery analyses to non-lottery students in the same program.

View paper

Published Tuesday, May. 20, 2014

Working Paper: School Choice Lotteries and Validity

By: Ron Zimmer and John Engberg
Working Paper No. 220

One of the most important questions for policymakers considering the merits of school choice programs is whether students in school choice program outperform their traditional public school counterparts. This question, however, has proven difficult to answer because the type of student who exercises choice may be fundamentally different than the type of student who does not. As such, researchers employ econometric techniques to control for observable differences between students across school sectors. Any remaining differences in achievement are therefore assumed to be attributable to the school choice program itself.

A frequently used technique to control for differences between students who attend a selected school versus those who did not is a school choice lottery in which seats are randomly awarded if there are more applicants than seats. The theory is that if students are randomly assigned to enroll in a chosen school via lottery, the assignment may be analyzed similarly to a randomized controlled trial (RCT), which is generally considered a strong method of estimating the impacts of a program or policy. After seats are awarded by lottery, any remaining differences in achievement between lottery losers, most of whom remain in traditional public schools, and lottery winners, most of whom attend schools of choice, are considered effects of the school choice program itself.

In this paper Ron Zimmer and John Engberg argue that lottery analyses have an important limitation that can and should be tested empirically: not all students in school choice programs enter via lottery. As such, the results of a lottery study do not necessarily generalize to all students in a school choice program. Specifically, the authors present two scenarios that frequently occur in school choice programs but are not directly addressed in most lottery analyses: (1) undersubscribed schools of choice (which admit students on a first-come, first-served basis) and (2) a school of choice in which some students are enrolled in the school prior to the assignment of open seats via lottery.

While their analyses show limited evidence that results differ between lottery and non-lottery students in the magnet school program they study, the authors' more general conclusion is that generalizability is a significant concern that is often untested in some of the most prominent school choice lottery studies to date. The authors propose two types of sensitivity tests to address these concerns in future research: (1) compare achievement growth of students who enter via lottery with that of students who entered without a lottery; (2) compare the achievement growth of students in undersubscribed schools of choice with growth of students in oversubscribed schools of choice. These tests would help policy makers better understand the applicability of lottery analyses to non-lottery students in the same program.

View paper

How This Gift Connects The Dots
 
Scholarships & Fellowships
 
Faculty & Programs
 
Campus & Technology
 
Financial Flexibility
 
Engage TC Alumni & Friends