data poison language study
Aug. 13, 2004
10 weeks ago, a consortium of researchers from Arizona's universities and
research institutes released a comprehensive report indicating that a lack
of good data has the state largely in the dark about the effectiveness of
its education policies.
This was true of policies regarding English-language learners, too, as noted
in a chapter I co-authored with other researchers.
We reported that available data "are not suitable to evaluate the effects of
specific policies for ELL students," and that "no reliable or meaningful
conclusions" could follow from them.
Like other authors of the report, we recommended that the state base its
policy decisions on scientific evidence, separating research as far from
politics as possible, and made recommendations for achieving this goal.
Regrettably, concerns raised over the adequacy of available data did not
prevent state schools Superintendent Tom Horne from releasing a study of the
anti-bilingual education policy upon which his campaign for office had been
based, using the very data researchers had warned about.
Horne's study concluded that his department's vigorous ban on Spanish was
working wonders for students. "There is not a single exception," Horne told
The Republic. "It tells us that the students in English
immersion do substantially better."
A look at the study reveals that this is a phenomenal exaggeration.
Spanish-background students in all-English classrooms in grades affected by
the anti-bilingual law had anywhere from no advantage to a "two month"
advantage over students taught bilingually.
In most cases, immersion students in this group revealed a mere "one month"
Due to extreme limitations in the state's education data and flawed analysis
in the study, even modest boasting would be unjustified.
Here are some reasons why:
• The study ignored socio-economic differences. Students who attend schools
with higher poverty levels tend to have lower scores than students who
attend wealthier schools. Because the study did not consider this factor, we
simply do not know whether the reported gains are associated with greater
resources or with program placement, as claimed.
• The study ignored relevant background knowledge. For example, if students
in one group tended to know more English or have more years of schooling in
prior years, they'd end up with higher scores regardless of what program
they were in.
• The study confused classroom-level and program-level descriptions. For
example, in bilingual programs, students are taught in bilingual classrooms
in early grades and transitioned to all-English classrooms after linguistic
and academic benchmarks have been met. In these circumstances, the
highest-scoring bilingual program students would be mixed in with the
immersion program students, artificially inflating their average score.
That's all bad enough, but things get worse. The study took at face value
potentially incorrect "program placement" data, which have been found to
shift erratically from year to year. Thus, the indicator around which the
study revolves is itself highly unreliable, frequently coded incorrectly by
students and teachers who feel confused, pressured and fearful about the new
Arizona has become the single most regressive and language-restrictionist
state in the nation.
Although its policies are at odds with good program-evaluation research, the
Department of Education continues to overregulate
application of the English-only law, making other viable and promising
approaches essentially impossible.
And after four years, the public remains totally ignorant about the
consequences of these extreme policies for English-language learners in
MacSwan is an associate professor in the College
of Education at Arizona State University, and a visiting scholar in the
linguistics department at the Massachusetts Institute of Technology.