Investing in What Works

A major US study¹ led by David Grissmer from RAND tackled this challenge through a meta-analysis of the longitudinal achievement data from 44 states on the National Assessments of Education Progress (NAEP). Grissmer’s findings provide insight into which state policies and programs may account for substantial differences in achievement scores that cannot be explained by student demographics.

Making Useful Comparisons

The researchers examined NAEP achievement results in math and reading at the 4^th and 8^th grade level between 1990 and 1996. These scores provided the first set of data permitting statistically valid achievement comparisons across states. By using federal census data, Grisssmer also established student family characteristics for the testing samples. The raw achievement scores of states were then compared by the progress of students from similar family demographics and by improvements over time. Although the 1998 test results arrived too late to be included in the tables provided in the study, the authors report subsequent analysis confirms the trends established.

The findings suggest some states are doing far better than others in causing achievement gains and in elevating poor and minority performance in comparison with like groups of students elsewhere. There were statistically significant differences –as large as 11-12 percentile points – among pupils with similar family characteristics across states.

For example, although California and Texas have similar demographic profiles, Texan students on average scored 11 percentile points higher on reading and math tests than their peers in California. Fourth grade NAEP math results for 1996 show non-Hispanic white and black students in Texas ranked first, and Hispanic students ranked fifth among the states. On the same test, California non-Hispanic white students ranked third from the bottom, black students last, and Hispanic students fourth from the bottom.

Grissmer observes that differences in state scores for students from similar families can be explained in large part by per pupil expenditures and how these are allocated. He suggests three factors appear to account for about 2/3 of the Texas-California gap: lower class sizes in primary grades, higher participation in public pre-kindergarten and better teaching resources for teachers.

Cost-Effective Resources

Some educational expenditures are far more cost-effective than others. This was especially noted in the case of states with large groups of poor and minority students.

Grissmer found, other things being equal, NAEP scores are higher in states with:

Higher per pupil expenditures, lower pupil-teacher ratio in lower grades
Higher percentages of teachers reporting adequate teaching resources
More children in public pre-kindergarten programs an Lower teacher turnover

These findings are predictable and intuitive. But the evidence also showed, other things being equal, some inputs may not have the expected effect on achievement:

States with higher teacher salaries or percentage of teachers with master’s degrees do not have higher scores. Teacher educational level is a weak and inconsistent predictor of achievement.
A lower pupil-teacher ratio showed statistically significant positive effects for grades 1-4, but was negatively correlated for grades 5 through 8.

Grissmer concluded the impact on scores often depends on precisely how the expenditures are directed, the socioeconomic level of the state, the current level of resourcing and the specific grades targeted.

Four years of small classes appear to provide resiliency against later larger classes, but two years or less do not. Programs targeted towards less disadvantaged populations generally showed no net cost savings. In calculating the effects on 4^th grade achievement of increasing pre-kindergarten participation by 10%, low family income was significantly correlated with gains, whereas middle and high incomes were not.

Policy Counts, Too
Resource variables do not explain all of the scoring trends. Grissmer found the trend coefficients were only slightly reduced by including the resource variables in the regression. Thus the researchers looked to reform efforts beyond spending to find plausible evidence that may have produced strong gains. High-performing North Carolina and Texas appeared to benefit from similar education policies over this period, including clear standards for each grade, assessment linked to those standards, good feedback to the schools, some accountability measures and deregulation of the teaching environment.

Lessons Learned
Until now, policymakers and educators have had little help from the research community in identifying what is effective and efficient. If we want to invest wisely in our children, R & D must play the same role that it does in every other sector of our economy. Grissmer’s work has already identified some key factors:

"There are significant differences in state gains that cannot be explained by changes in level or allocation of the major resource variables. Structural reforms within the education system are most plausible cause of these gains. This implies that providing more resources for public education is not the answer to school improvement without fundamental reforms that can change the organizational climate and incentives in education.."(p. 101)

The report may be downloaded in PDF format from: http://www.rand.org/publications/MR/MR924