Bad data poison language study
Arizona Republic
Aug. 13, 2004

My Turn

  Jeff MacSwan

Just 10 weeks ago, a consortium of researchers from Arizona's universities and research institutes released a comprehensive report indicating that a lack of good data has the state largely in the dark about the effectiveness of its education policies.

This was true of policies regarding English-language learners, too, as noted in a chapter I co-authored with other researchers.

We reported that available data "are not suitable to evaluate the effects of specific policies for ELL students," and that "no reliable or meaningful conclusions" could follow from them.

Like other authors of the report, we recommended that the state base its policy decisions on scientific evidence, separating research as far from politics as possible, and made recommendations for achieving this goal.

Regrettably, concerns raised over the adequacy of available data did not prevent state schools Superintendent Tom Horne from releasing a study of the anti-bilingual education policy upon which his campaign for office had been based, using the very data researchers had warned about.

Horne's study concluded that his department's vigorous ban on Spanish was working wonders for students. "There is not a single exception," Horne told The Republic. "It tells us that the students in English immersion do substantially better."

A look at the study reveals that this is a phenomenal exaggeration.

Spanish-background students in all-English classrooms in grades affected by the anti-bilingual law had anywhere from no advantage to a "two month" advantage over students taught bilingually.

In most cases, immersion students in this group revealed a mere "one month" advantage.



Due to extreme limitations in the state's education data and flawed analysis in the study, even modest boasting would be unjustified.

Here are some reasons why:


• The study ignored socio-economic differences. Students who attend schools with higher poverty levels tend to have lower scores than students who attend wealthier schools. Because the study did not consider this factor, we simply do not know whether the reported gains are associated with greater resources or with program placement, as claimed.


• The study ignored relevant background knowledge. For example, if students in one group tended to know more English or have more years of schooling in prior years, they'd end up with higher scores regardless of what program they were in.


• The study confused classroom-level and program-level descriptions. For example, in bilingual programs, students are taught in bilingual classrooms in early grades and transitioned to all-English classrooms after linguistic and academic benchmarks have been met. In these circumstances, the highest-scoring bilingual program students would be mixed in with the immersion program students, artificially inflating their average score.

That's all bad enough, but things get worse. The study took at face value potentially incorrect "program placement" data, which have been found to shift erratically from year to year. Thus, the indicator around which the study revolves is itself highly unreliable, frequently coded incorrectly by students and teachers who feel confused, pressured and fearful about the new law.

Arizona has become the single most regressive and language-restrictionist state in the nation.

Although its policies are at odds with good program-evaluation research, the Department of Education continues to overregulate application of the English-only law, making other viable and promising approaches essentially impossible.

And after four years, the public remains totally ignorant about the consequences of these extreme policies for English-language learners in Arizona.


Jeff MacSwan is an associate professor in the College of Education at Arizona State University, and a visiting scholar in the linguistics department at the Massachusetts Institute of Technology.