Complaints about MCAS have come from many sources and have focused on a range of issues. Students, teachers, parents, school administrators, legislators and interested bystanders have all chimed in on the debate. How good are these objective tests at measuring what students have learned? Do MCAS tests actually measure the skills they claim to measure? Aren't there other things we want our schools to accomplish beyond the specific forms of information and reasoning that MCAS tests? Should we be trying to measure these things as well? If we make changes in our schools on the basis of these scores, will those actions improve things? Will such actions be fair? That is, will they reward and punish the appropriate people and programs? The stakes are high, and the potential consequences great.

Consider just one of the issues raised in the debate about MCAS. Do the scores on these tests really reflect differences in the quality of the schooling throughout the state? That is a critical question given the way the issue has been framed by the policy makers who would make changes on the basis of MCAS scores. They speak of  "schools that fail," and propose that schools with low MCAS scores be subjected to a range of remedies which might include reductions in salaries for their teachers, provision of vouchers for students to leave such schools, and even closing "failing schools." The underlying premise is that schools that produce low MCAS scores do so because the teaching is bad. 

This spring a private research firm conducted a study of MCAS scores in Massachusetts and found what so many critics of the testing sensed from the scores, but had not pinned down with hard numbers. The fact that MCAS scores have been high in wealthier school districts and low in poorer ones was a consistent and worrisome fact since the inception of the program of testing. It seemed reasonable to conclude that in our poorest cities and towns the schooling was inferior. The teaching staffs were overloaded with entrenched and incompetent teachers, and the life and creativity had gone out of the classrooms. Perhaps they needed some sort of "renewal," to use a currently popular term in education. Kevin Clancy, chairman and CEO of Copernicus, the company that did the study, reported on its findings in the Boston Globe. Here is what they found. 

If you take all the MCAS scores produced in a given testing session, the variation in the scores between schools does, indeed, change with the wealth of the town in which the scores were produced. Consistently Weston, Medfield, Harvard and Wellesley outscored Brockton, Lowell and Chelsea. In fact, the research combined scores for five social economic factors and tested how they were related to MCAS scores. The five factors were:

1. Rate of welfare dependency – in this case, researchers used the percent of families receiving Aid to Families with Dependent Children, or AFDC

2. Family structure – measured by the percent of families that had two parents

3. Family income – measured by the percent of families below the official, government poverty line which is currently set at less than $18,500 for a family of four

4. Race – measured as the percent of families that are white

5. Educational level – measured as the percent of families in which a member holds a college bachelor's degree or higher. 

The researchers added these five factors together into a statistical lump and correlated them with the MCAS scores. What they found was that over 90% of the variation in MCAS scores of students was explained by the combination of these five factors. The statistical procedure they used to arrive at this conclusion is not as complicated as some people make it sound. When we try to explain problems like the extreme variation in MCAS scores, we should never, ever begin with the assumption that one factor is THE CAUSE. Personal experience and a great deal of research have taught us that in human behavior a number of factors are extremely likely to be contributing to a specific outcome. For example, consider how successful a person is in his or her career. Do not take seriously the person who claims his success is due to a single factor ("Work hard, my boy."), since lots of people work hard with no results. Other factors must be present for success to occur. Among them are intelligence, knowledge, educational level, educational prestige, social skills, contacts, health, luck, capacity for ruthlessness, honesty, luck and so on. So, when these researchers combined five factors and set them against the MCAS scores, they were trying to account for variation in the scores with more than one variable. To find these factors they sifted through a large number of variables before they settled on these five. They were chosen because they were the most efficient in explaining why MCAS scores were higher in some towns and lower in others. 

Notice that if 90% of the variation in MCAS scores could be accounted for by these variables, that leaves just 10% due to other factors. You can speculate what such factors might be, but even if all of the remaining ten percent were due to differences in quality of teaching, it would be small in comparison with the 90% due to the social-economic variables identified by the research. How, then, should we use such findings? To begin with, it makes no sense to punish teachers for the fact that they teach in poorer towns. Their students come to class with a range of burdens imposed by social economic disadvantages and these harm their ability to learn. In addition, these poorer towns cannot afford the salaries and budgets that support the most effective educational programs, making the rate of learning worse yet for the students. In wealthier towns the children carry no such burdens when they arrive at school, and are given all the advantages of higher spending on their schools and greater emphasis on learning at home. 

But MCAS is not worthless. It is merely misused. Rather than use MCAS scores to punish poorer towns for being poorer, this research proposed a realistic way to find the under-performing schools we may wish to either assist or punish. If advantages like those identified in this research so directly lead to higher scores on tests like MCAS, then we can expect that students in wealthier towns should score higher and those in poorer towns score lower. We should then set our expectations for the performance of students on a scale taking these advantages into account. So, in a wealthy town like Weston, their advantages might translate into an expected average MCAS score in the 95th percentile of schools while students in Chelsea might be predicted to score in only the 20th percentile. Simply plugging values for a given town into Kevin Clancy's model would yield predictors like this. An under-performing school then would be identified as one in which the MCAS scores are below those predicted by the level of advantage or disadvantage of living in the town. The Copernicus research discovered that by this formula, relatively wealthy Marblehead under performed while relatively poor Chelsea over-performed.

Before we decide to punish teachers in poor schools for the low MCAS scores of their students, we ought to identify the factors underlying student performance. If we paid doctors only if their patients stayed well they'd scream bloody injustice. They can't control their patients' genetic propensities for disease, or whether they take their medications, or whether they eat decently, can afford to live in safe and healthy homes, can read the labels on the bottles or are afraid of hospitals. Teachers should be held responsible only for the portion of students' educations over which they have influence. Now we know what percent of an education that normally is.

William C. Levin is Associate Editor of the Bridgewater Review.

Top | Table of Contents | Home