Testing: One, Two, Three; Testing: One, . . . Can You Hear Me Back There?

March 22, 1999

Book Review
Philip J. Davis

Executive Summary, Third International Mathematics and Science Study (TIMSS). By William Schmidt, Curtis McKnight, and Senta Raizen
U.S. National Research Center for TIMSS, Michigan State University.
A TIMSS Primer: Lessons and Implications for U.S. Education. By Harold W. Stevenson, Thomas B. Fordham Foundation, July 1998.
The Rationality and Irrationality of International Comparative Studies.
By Christine Keitel and Jeremy Kilpatrick, in International Comparisons in Mathematics Education, G. Kaiser, I. Huntley, and E. Luna (eds.)
Falmer Press, London, 1998, pages 241-257.
Tinkering with TIMSS. By Gerald W. Bracey, Phi Delta Kappan, September 1998.

"No system of external tests which aims primarily at examining individual scholars can result in anything but educational waste."---Alfred North Whitehead, The Aims of Education, 1912.

The Third International Mathematics and Science Study (TIMSS) is the most extensive and exhaustive international comparison of education ever undertaken. In this study (c. 1995), which cost at least 50 megabucks, a half million students at five grade levels were tested in 41 countries. TIMSS confronted the students with 150 (mostly) multiple-choice questions, of which 50 were to be answered within a restricted time.

In its most sensational release, the study provided a rank ordering of countries as regards student accomplishment. Practically every newspaper in the U.S. reported as sad fact that American students came in way down on the list. The official TIMSS publications include three large volumes; also part of the raw data are more than 250 videotapes of classroom situations. The reaction to the report from pundits, educationists, mathematicians, politicians, foundation officers, think-tanks promoting national competitiveness, school superintendents, teachers, their union officials, statisticians, overheated bores, ideologues, amici curiae, and Johnny & Mary Q. Public has been enormous. "Everybody," as Jimmy Durante used to say, "wants to get into the act."

Search engines can find more than 5000 TIMSS-related Web sites. Even with the chaff discarded, if this were possible, it would still take several lifetimes and all the cash resources of one of our more plush foundations just to review the reactions to TIMSS. Along one particular axis, the reactions emerging from the wide, wide American world range from "experience to which we must heed" to "garbage." Along another axis, recommendations for change range from a nipponization or gallicization of U.S. education to total computerization wherein education would be entrusted to a very few computogogues. After talking to a long-experienced staff member of the Annenberg Institute for School Reform at Brown, I gather that despite the cogent doubts of not a few critics, the TIMSS report is now serving in the U.S. as a widely heeded clarion call. And this despite Stevenson's statement in A TIMSS Primer that:

"The TIMSS staff hesitate to draw any firm conclusions [about the poor U.S. performance] from the study."

This confession is followed by seven "possible explanations," and I can think of others not suggested that would probably get censored out.

The Executive Summary: "A Splintered Vision"
The executive summary of TIMSS (Schmidt, McKnight, and Raizen) is directed specifically toward U.S. consumption. There is no coherent vision in mathematical and science education, say its authors. There is no dominating leadership. The curricula and textbooks are unfocused. The coverage is a mile wide and an inch deep. The demands on students are slight, and what is considered to be basic minimal knowledge is less than the international norm.

"Fundamental changes are needed in teacher knowledge, working conditions, curricula quality, student expectations, and textbook content." Well, what of the reform movement? Is that movement in the singular? No: movements, plural. Movements abound: They are full of energy and enthusiasm and receive much publicity. But each separate proposed reform is, according to the executive summary,

"heard simply as one more voice in a 'babel' of competing voices. This babel becomes so overwhelming that it is hard for official actors to . . . prioritize the voices to which they will listen."

Apart from numerous bromides about the necessities and qualities of technical knowledge in an information-driven technological world, the chief recommendation of the executive summary seems to be that, in some order, we cut back the number of topics taught, we deepen them according to someone's vision, we set up a unified U.S.-wide model, and underneath it all---implicitly or wistfully---that we reform the whole of U.S. society and American civilization so as to admit such educational changes.

A Single Test for The Wide, Wide World?
In numerous countries, opponents of the TIMSS report have generated polemics rivaling in fury those of the recent U.S. Congress. Thus, Gerald Bracey, a research psychologist and fiery critic of international student comparisons, has written at the conclusion of a long series of objections,

"The test results of the TIMSS final year study . . . do not permit a comparison across nations. When people make such comparisons, they are engaging in a political exercise, not an intellectual one."

The second article under review employs calmer language:

"The funding, design and analyses of comparative studies (TIMSS) fail to represent not only the mathematics education community, but also the spectrum of participating countries."

So claim Keitel and Kilpatrick, both of them mathematical education theorists, the former at the Free University, Berlin, and the latter at the University of Georgia. Their article raises a good dozen significant unresolved issues, of which I present here the three that strike me as the most important:

1. How was it determined that the questions asked of the students properly reflected the actual curricula, the actual opportunities to learn that are in place in the various school grades in the various countries? This determination appears to have been made informally (perhaps subjectively) by a battery of experts. What appears not to have been taken into consideration by the experts was the total context of the learning situation: the way questions were developed in the local classes, the goals and purposes of these questions, the social and economic context of the school experience, the extracurricular opportunities and offerings, the materials that were available, and so forth.

"The improvements made in TIMSS over previous comparative studies have been tainted by the dominance of the USA in funding most of the research and directing the data gathering and analysis. The consequence is a study embedded within the research traditions of one country but too frequently having little or nothing to say to mathematics educators in other countries."

2. The high ranks of Asian students, admired and envied by the rest of the world, raise eyebrows. Asian mathematics instruction focuses on problem solving. More than 80% of primary and secondary students take private lessons to bone up for the exams. The Japanese classroom situations shown in the videos are not typical, says Keitel, who has visited Japanese classrooms, but rather were preselected.

3. The political consequences of TIMSS are considerable and not entirely desirable. Some TIMSS researchers are using the mediocre results reported for their countries to get money for more testing; as a consequence, there could simultaneously be less support for public schools and more for privatization, and greater constraints on teachers.

Where I'm Coming From
Splintered vision? My own feeling is that the U.S. is large enough and strong enough to support diverse mathematical visions. It has so many high schools and colleges of different types that I see no great danger, and a number of virtues, in having a wide variety of mathematics taught. There is no need for the lockstep approach, wherein one knows that on a given date, the law of sines is being taught in high schools all over America.

World technological competition? In addition to our native-born mathematicians, the presence in the U.S. of a large number of first-class mathematical émigrés, educated according to a variety of traditions, is made possible by liberal immigration policies and employment opportunities. On that score there is no need for worry over the next decade about a supply of good mathematicians.

The newspapers recently reported the temporary hiring of mathematics teachers from Austria (and other countries) to make up for shortages at the elementary and secondary levels in U.S. schools. These temps are given short courses in American culture and American modes of teaching, which all hope will not ruin their mathematics.

I used to hear that competition is good, that it fosters creativity. Social Darwinians love it. Bell Telephone was broken up, and a million communication systems blossomed. Well, perhaps too much splinterization is not good.

Does a single test for myriads of students make sense? Even as the editors of "A Splintered Vision" were putting together their document, new experimental programs were being devised for every age level, with no chance of analysis by the methodology of TIMSS. Here is one such program: A certain class is on line en masse. The instructor talks to the class and sets problems Web-wise. The students talk to each other and to the instructor Web-wise. They comment on the material, discussing their personal attacks on the problems and their solutions. The instructor answers Web-wise. At regular, but not exactly frequent, intervals, the class and the instructor materialize from their normal spectral condition and meet in Lecture Room A, Watson Hall. A grade is given somehow on the basis of this interplay. (Subjective judgment? A dirty word this, in some quarters.)

Could all the variations of this one teaching mode be reduced to a common basis? But why bother? TIMSS is out of date (like the "latest"-model PC) even before its gargantuan data base can be analyzed.

Regarding the traumatic emphasis on testing that exists in certain places-as seen in the practice among concerned parents of renting hotel rooms near the test site so that their children can come back and rest between sessions-I would say that this resembles the grueling training of young Olympic hopefuls in skating that is supported by their parents. The system produces good skaters, but world-class skating is for the skating elite.

And as regards overdosing on testing, even as the testing establishment in the U.S. has produced a number of test-crazy schools, the practice has been under attack (even in medical diagnostics!). Mind you, in some public schools whole months are devoted to test preparation, during which testing goes on al-most continuously. While this treatment may be appropriate for a hospital patient wired up for vital signs, teaching for knee-jerk responses may deprive students of an education. Some administrators have also suggested that teachers' pay be linked to student achievement, and one immediately envisions their paychecks fluctuating from month to month like the returns on a mutual fund.

How Much and for Whom?
Mathematics is an activity, and in part a language, that is created and supported by people. No one group has or should have a monopoly on its creation, its dissemination, or its use. Yet the strength of mathematics as a social creation depends on worldwide fluency in this language.

How much fluency is sufficient? As regards the average person living in a highly mathematized, but chipified, multimedia technological civilization, I think that rather less is required on a day-to-day basis than is often argued in official recommendations.

What is necessary is to teach enough so that the commonplace diurnal mathematical demands placed on the pop-ulation are readily fulfilled. What is also necessary is to infuse sufficient mathematical and historical literacy that people will be able to understand that the mathematizations put in place in society do not come down from the heavens: that they do not operate as pieces of inexplicable ju-ju, that mathematizations are human cultural arrangements and should be subject to the same sort of critical evaluation as all human arrangements.

At the risk of sounding like a traitor to my profession, I would say that high school algebra or beyond is not necessary to achieve this goal. A bit of elementary probability would be a good thing, as would knowledge of the major time-variation templates: linear, exponential, periodic. A mandatory short course called Skepticism 101 would help students hone critical skills in all areas. I and many others would like to see students' ability to be critical strengthened greatly; I know of some teachers who have been working toward just this goal, which may be more important than finding the volume of a cone by rote.

As for the higher levels of mathematical knowledge, I am an elitist and advocate special training for those who are up to it, who like it, and who want it. Few of us can break Babe Ruth's home run record, with or without an appeal to the pharmacy; few of us can dance like Baryshnikov. There is no need for it. A nation of Baryshnikovs is not required for a decent life. For most of us, watching the performance with admiration is enough.

Philip J. Davis, professor emeritus of applied mathematics at Brown University, is an independent writer, scholar, and lecturer. He lives in Providence, Rhode Island.


Renew SIAM · Contact Us · Site Map · Join SIAM · My Account
Facebook Twitter Youtube