Home » Education Canada » Telling Time with a Broken Clock - by Joe Bower

Telling Time with a Broken Clock - by Joe Bower

The trouble with standardized testing

What do standardized test scores tells us? This is a complex question, yet too many people assume to know its answer. What if standardized test scores are like a broken clock that is providing us misleading information? Authentic accountability is about providing the public with the information they need about their public schools. This means that public assurance is about transparency, but standardized tests tend to conceal far more than they ever reveal. It's time to replace standardized testing with more sophisticated and more demanding processes.

225 votes
+
Vote up!

“The more we learn about standardized testing, particularly in its high-stakes incarnation, the more likely we are to be appalled.” — Alfie Kohn

What do standardized test scores really tell us? Like many public policy issues, this is a complex question – yet too many people assume to know the answer. Whole school jurisdictions and entire nations define themselves by their standardized test results, including provincially administered examinations and international instruments such as the Program for International Student Assessment (PISA). All of these programs, justified by their so-called impartiality and objectivity, share the assumption that the scores must be the public’s “transparent” window into the quality of our schools.

Bestselling author and blogger Seth Godin reminds us that the worst kind of clock is a clock that randomly runs fast or slow. “If there’s no clock,” Godin writes, “we go seeking the right time. But a wrong clock? We’re going to be tempted to accept what it tells us.”[1] Godin’s message is that tracking the wrong data or misreading good data can get us into trouble. What if standardized test scores aren’t telling us what we think they are telling us? What if the scores are illusions that are giving us false confidence? What if our reliance on standardized testing to judge our schools is like relying on a broken clock for time?

If education policy pundits (many of whom are not teachers) and politicians expect the public to trust the scores as prima facie evidence of the quality of teaching and learning that goes on in our schools, then the public needs to know more about the costs and consequences of standardized testing. It is time to move past our historic reliance on standardized testing programs driven by educational bureaucracies satisfied to measure what is easiest to measure. Canada’s already strong public school systems will not punch past their current levels of performance unless we move to a new approach to public assurance. 

Measuring what matters least

Ask any parents what their long-term concerns and goals are for their children, and seldom will you hear about test scores and world rankings. Their concerns are compelling, existential and heartfelt. Parents want their kids to be happy, hard-working, motivated, responsible, honest, empathetic, intelligent, collaborative, creative and courageous. Of course we want our children to grow academically, but we also want them to grow emotionally, socially and physically, and this requires a well-rounded education that cannot be evaluated by standardized tests. This is not an argument against academics; academics should play a large role in school. However, even when it comes to numeracy and literacy, standardized tests tend to be limited to measuring forgettable facts while ignoring the higher-level creative and critical thinking. It makes a lot of sense to question the scores when you know that the tests are a contrived and unrealistic form of assessment that measures what matters least. It’s time we shifted from valuing what we measure to measuring what we value. 

Campbell’s Law

Well-known (but not well-known enough) in social science, Campbell’s Law tells us that “the more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it was intended to monitor.”[2] In the case of standardized testing, corruption and distortion can come in a variety of ways. Here are three examples: 

Teaching or testing? Teaching to the test and excessive test preparation invalidates inferences that can be drawn from the scores – yet they are the inevitable response to pressure to produce good test scores. Classroom time is devoured by not only the tests themselves but also practice tests, pre- and post-tests, field tests for the tests, benchmark tests, teacher tests, district tests, and state or provincial tests.  Because testing is not teaching, this ultimately leads to a loss of opportunities for students to have a broad range of educational experiences, and the first things to go usually end up being the arts and physical activity – which do not lend themselves to be easily tested.

Learning or cheating? The moment low-stakes test scores are publicized to rank and sort teachers or schools, they become high-stakes tests. Where there is smoke, there is usually fire, and where there is high-stakes standardized testing, there is cheating. We can bemoan this inconvenience or play the blame game, but it won’t change anything. Systems thinking tells us that cheating has less to do with the characteristics of individual teachers or students and more to do with the priorities of schools and school systems.

Raising children or raising scores? When schools are encouraged to focus on test scores, some come to see children less as individuals of worth regardless of their academic ability, and more as score increasers and score suppressors. Sadly, the more the scores are made to count for teachers and schools, the more the scores count against the children who need the most help. They will be seen as undesirable; after all, they are the students most likely to score low, dragging down the school’s ranking. This becomes even truer when (as in the U.S.) merit pay schemes marry teacher pay to the scores and/or when a school’s reputation hinges on being publicly ranked. Under these twisted circumstances, schools may come to know more about raising scores than raising children.

We can no more skirt the real-world ramifications of Campbell’s Law than Wile E. Coyote could avoid the punishing effects of gravity. Nichols and Berliner explains, “apparently, you can have (a) higher stakes and less certainty about the validity of assessment or (b) lower stakes and greater certainty about validity. But you are not likely to have both high stakes and high validity. Uncertainty about the meaning of test scores increases as the stakes attached to them become more severe.”[3]

Socio-economics

Alfie Kohn begins his article “Fighting the Tests: A practical guide to rescuing our schools” by stating, “Don’t let anyone tell you that standardized tests are not accurate measures. The truth of the matter is they offer a remarkably precise method for gauging the size of the houses near the school where the test was administered.”[4] The inconvenient truth about standardized testing is that socio-economic status is responsible for an overwhelming proportion (50 to 70 percent) of the variance in test scores. The strongest predictor of student performance on achievement tests is socio-economic status, which is why it is a mistake to believe that the scores tell us about school quality when really they are reflecting affluence or poverty.

No school or school system has ever become great without great teachers, but what can an excellent teacher do about a child who needs glasses or is hungry? To say that teacher or school quality is the most important variable in education is at best naive. Education historian Diane Ravitch writes, “Reformers tell us that teachers are the most important influence within the school on student scores, and that is right. But the teacher contribution to scores is dwarfed by the influence of family and other out-of-school factors.”[5]

Ultimately, great teachers make great schools, but great teachers can’t do it alone – they require the support of an equitable society. If we are not careful, we risk misinterpreting the scores, and instead of waging war on poverty and inequity, we end up waging war on teachers and schools.

Dispelling the corporate “reform” agenda

Alongside the rush to introduce unproven technologies into classrooms, standardized testing in the United States has become a political instrument wielded by organizations such as Students First, Democrats for Education Reform and the American Legislative Exchange Council. Linking teacher pay to student test scores, and eliminating tenure and collective bargaining, have become popular methods for undermining confidence in public schools so that education entrepreneurs can pour private equity and venture capital into companies that aim to profit from public education.

After a decade of intense standardized testing and sanctions under No Child Left Behind, California Democrats recently passed a resolution stating, “The reform initiatives of Students First rely on destructive anti-educator policies that do nothing for students but blame educators and their unions for the ills of society, make testing the goal of education, shatter communities by closing their public schools, and see public schools as potential profit centers and children as measurable commodities.”[6]

Over the last two decades, the U.S. has proven to be a cautionary tale for how Canada, and the world, should not reform education, and Canadians would be wise not to think that the 49th Parallel offers any inherent insulation from the corporate education reform agenda.

Building a better clock

The way forward is not to build schools that are a better version of yesterday. It is telling that the demand for standardized testing intensifies the further you get from classrooms, and yet authentic accountability and public assurance needs to happen within schools and communities.

A move away from standardized testing is not a case for the absence of accountability – it is a pathway to supporting innovation and creativity. Ruth Sutton reminds us, “The issue is not whether we need information about the learning and achievement of our children and young people, but what kind of information we need, and how best to gather it.”[7] Once we can see that standardized testing is like a broken clock, we can work together to figure out how to build a better clock.

Building a better clock and better schools starts with asking tough questions – which is precisely the spirit behind an Action Canada Task Force report titled Real Accountability or an Illusion of Success?[8] that invites Canadians to take a deeper look at the goals of public education and the role of standardized testing.

Alberta should be congratulated for their recent move away from their Grades 3, 6 and 9 Provincial Achievement Tests. Now Alberta requires the insight and courage to see that the replacement for the old tests should not be new tests. From a policy perspective, accountability models based on census testing of entire student populations do little to support student progress and are not cost effective. A shift to a sample program using performance-based assessments would be less obstructive, cost less and provide more meaningful information.

Given the highly relational nature of the teaching and learning process, the best teachers know that assessment is not a spreadsheet – it’s a conversation. They also know that there is no substitute for what teachers observe while their students are actively learning, and this is why the best assessments ask students to actually do something that is in a context and for a purpose. Unlike standardized tests, which cannot provide anything more than an incomplete snapshot of a student on a single day, a collection of performance assessments assembled in a learning portfolio can inform the teaching and learning process in a timely fashion, while simultaneously assuring the public that students are receiving a high-quality education.

To enhance public assurance, Andy Hargreaves suggests departments of education should shift from a “bureaucratic accountability model to a locally focused, student-driven assurance model based on school-development plans and teachers as leaders in innovation.”[9]  The means for accomplishing this can be found in programs like Alberta’s Initiative for School Improvement (AISI), which was recently axed by the Alberta government. A number of similar school improvement initiatives exist throughout Canada, but are also vulnerable to cuts by governments that use times of fiscal belt-tightening as an excuse to reduce investments in innovation.

It might be argued that standardized testing has allowed us to build good enough schools; after all, even a broken clock is right twice a day. However, business guru Jim Collins reminds us that “good is the enemy of great.” If we aspire to create great schools for all children, we need to seek an end to standardized testing and replace it with more sophisticated and demanding processes for public assurance.

First published in Education Canada, June 2013

EN BREF - Que nous révèlent les résultats d’examens normalisés? La question est complexe, pourtant trop de personnes présument en connaître la réponse. Et si les résultats aux examens normalisés s’assimilaient à une horloge déréglée diffusant des renseignements erronés? Une véritable reddition de comptes, c’est fournir au public les renseignements qu’il requiert sur les écoles publiques. C’est donc une question de transparence. Mais les examens normalisés tendent à obscurcir plus de choses qu’ils n’en révèlent. Le temps est venu de les remplacer par des processus plus évolués et plus exigeants.


[1] Seth Godin, “The Worst Kind of Clock,” Seth Godin (Blog), September 21, 2012, http://sethgodin.typepad.com/seths_blog/2012/09/the-worst-kind-of-clock.html.

[2] Sharon Nichols and David Berliner, Collateral Damage: How high stakes testing corrupts America’s schools (Massachusetts: Harvard Education Press, 2008), 26-27.

[3] Nichols and Berliner, Collateral Damage, 27.

[4] Alfie Kohn, “Fighting the Tests: A practical guide to rescuing our schools,” Phi Delta Kappan (January 2001). http://www.alfiekohn.org/teaching/ftt.htm

[5] Diane Ravitch, “Why VAM Is Junk Science,” Diane Ravitch’s Blog, July 16, 2012, http://dianeravitch.net/2012/07/16/why-vam-is-junk-science

[6] The California Democratic Party, Resolution 13-04.47: Supporting California’s Public School and Dispelling the Corporate “Reform” Agenda (April 14, 2013).http://www.cadem.org/admin/miscdocs/files/Resolutions-Report-FINAL-2.pdf

[7] Alberta Assessment Consortium, A New Look At Public Assurance: Imagining the possibilities for Alberta students. http://www.aac.ab.ca/resources/pdf/Public%20Assurance%20Doc_final_may31.pdf

[8] Real Accountability or An Illusion of Success? A Call to Review Standardized Testing in Ontario. Action Canada Task Force Report.http://testingillusion.ca/wp-content/uploads/2013/01/illusion_of_success_EN.pdf.

[9] Andy Hargreaves, foreward to A Great School for All: Transforming education in Alberta (Edmonton: Barnett House, 2012).