Kenneth Prewitt’s contemplation of the prospects for useful social science challenges us to place this problem—familiar to students of public policy and the history of science—in its contemporary context. In an age of intense focus on accountability in nearly all matters of public policy and civil society, is it any wonder that the benefits and costs of investments in scientific research are increasingly subjected to measures of its promised and actual utility? Here I will focus on a central theme in Prewitt’s eloquent essay, which should be on the “must-read” list for anyone worried about [social] science policy.

“Why shouldn’t the public be entitled to evidence that these dollars are being spent smartly and usefully?”

“Accountability” is a concept with more complexity than one might think given its cliché status in popular and professional rhetoric. The distinguished social scientist James March once observed that “the demand for accountability is a sign of pathology in the social system.”1 Quoted by Lee Cronbach, in “A Valedictory: Reflections on 60 Years in Educational Testing,” Board on Testing and Assessment, National Academy of Sciences, 1995.
Read more here →
Without a doubt, we are experiencing a widespread, some would argue rampant, epidemic of hardball-style insistence on a variety of metrics as evidence that public funds are being spent wisely. Indeed, in some areas this seemingly natural extension of principles of diffused authority and distrust of centralized government, part of our historical DNA, has morphed into an obsessive reliance on measures that are, at best, approximations of progress toward ambiguous, complex, and uncertain outcomes; and worse, has resulted in the gradual and pernicious substitution of the measures for that which they are ostensibly measuring. I am reminded of Alfred Binet’s exasperation when, having devised an early version of “intelligence testing” was hectored by those wanting a simple definition of “intelligence,” facetiously said, “It is what my test measures!”2 Quoted in Phillipe Malrieu, Hommage a Henri Wallon (Toulouse: Universite de Toulouse, 1981) His views on intelligence were in fact substantially more nuanced, and as the distinguished psychologist Sheldon White noted, Binet condemned those who with ‘brutal pessimism’ and ‘deplorable verdicts’ were promoting the concept of intelligence as a single, unitary construct.”3 Sheldon White, “Conceptual Foundations of IQ Testing,” Psychology, Public Policy, and Law 6, no. 1 (Mar 2000): 33–43.  More info →

Anyone who has toiled in the swamps of educational assessment understands the hazard: good standardized measures of academic achievement, intended as estimates of complex domains of human information processing, teaching, and learning, can be vulgarized and transformed into the elements of curriculum and the definition of educational goals. With this creeping (and somewhat creepy) misuse of measurement come inevitable behavioral consequences: “teaching to the test,” narrowing of curricula, distorting incentives for instruction and student performance, and generating misleading information about the quality of schools and school systems.

“What mechanisms and metrics might reduce the unintended negative effects of high-stakes accountability?”

In other words, and with all respect to Professor March, accountability is not necessarily a symptom of pathology of social systems (it would be nice if more societies embraced basic rules of democratic accountability), but too much of it undertaken without regard to the fragility of its measures can undermine its own legitimate purposes. Accountability, per se, is not a bad thing; the danger occurs when too much emphasis is placed on the “count” part of the word, and not enough on our “ability” to make sense of the measures we use.4See also Michael Feuer, “Measuring Accountability When Trust Is Conditional,” Education Week, September 26, 2012.
Read here →

Given that the federal investment in scientific R&D grew by about 600 percent since 1955, to a level of approximately $135 billion for fiscal year 2015, it is neither surprising nor wrong for the concept of accountability to have become increasingly salient.5For trends in federal science funding see the NSF supplement on federal R&D funding.
Read here →
Why shouldn’t the public be entitled to evidence that these dollars are being spent smartly and usefully? As Prewitt correctly notes,

Democracy rests on holding politicians accountable for their errors of judgment or corrupt practices. Financial markets and commercial enterprises are accountable to shareholders and consumers, trade unions to their members, and professors to their students and peers. Why should science not be accountable to its funders?

But that’s where the agreement ends, for it is in the choice of metrics and the ways in which they are interpreted and applied that the debate inevitably degenerates. And, its political origins notwithstanding, the pursuit of viable and useful metrics is sullied, ironically, by the subtle and often not-so-subtle intrusions of politics. As I argue in a new book, The Rising Price of Objectivity,

Today’s political stalemates over science funding are shrouded in ideology and religion: conservative factions in Congress, cheered on by the most extreme of the Tea Party yahoos, brazenly propose cuts in science funding for anything that they claim lacks a clear national security rationale; but alert observers see through the veneer to what is clearly a disdain for any research, whether funded publicly or privately, on issues such as climate change, women’s reproductive biology, educational standards, and the general social good.6 Cambridge, MA: Harvard Education Press, 2016.More Info →

Although we have always had vigorous debates about the funding and governance of science, the more recent advent of ideology-driven intransigence has reached new heights (or lows) in civic and policy discourse.

The challenge, then, is avoiding either extreme of the seesaw, between the naïve fantasy that science can or should be exempt from public accountability on one end; and the detrimental consequences of political overreach and overreliance on imperfect estimates of scientific purpose and performance on the other. This is a key message in the call for a “theory of use,” one of the more compelling results of the National Research Council (NRC) effort that Prewitt chaired.7 Washington, DC: National Academy Press, 2012More Info → At the risk of some circularity of reasoning, I would argue that understanding and relaxing this fundamental tension—in social science generally and in education perhaps as the most problematic (“messy”) case given its complicated mosaic of culture, history, values, and ideology—requires, well, good science. Social (and behavioral) science can and should be responsible for crafting smarter and more sensible systems of accountability and, more generally, for developing a use theory, given the record of notable accomplishments in the understanding of incentives and motivation and the effects of risk, uncertainty, and complexity. There is no room here for a full elaboration, but I would emphasize the contributions of Herbert Simon (and his students) as a beginning point for consideration of strategies of accountability and the design and use of measures.8 See Herbert Simon, “From Substantive to Procedural Rationality,” in Method and Appraisal in Economics, edited by S. J. Latsis, 129–148 (Cambridge, UK: Cambridge University Press, 1976).
More Info →
My application to education research and policy is in Michael Feuer, Moderating the Debate: Rationality and the Promise of American Education (Cambridge, MA: Harvard Education Press, 2006).

More Info →
How much evidence, following what type and amount of deliberation, is relevant and “useful” to decisions about the funding of science, generally and in the social sciences specifically? Under what organizational arrangements and agreements can such a theory be expected to flourish? What mechanisms and metrics might reduce the unintended negative effects of high-stakes accountability?

Prewitt’s essay raises other questions. For example, to what extent are the social sciences more (or less) vulnerable than the physical and biological sciences to the pressures of funding that may compromise scholarly integrity and the credibility of findings? (I discuss this issue in The Rising Price, with attention to the combined effects of the private and public funding sectors.) To what extent should the social sciences be given special leniency in accountability, on grounds that their scope is complex, the goals of policy are less-easily articulated, and (human) subjects are so inherently varied? This was one of the questions posed in another NRC report, focused on education research, which warrants revisiting.9 See Richard Shavelson, et al, Scientific Research in EducationWashington, DC: National Academies Press, 2002.More Info → Given that “social science is steadily pushing its way deeper into our lives … [and] the more social science matters, the more society wants a say in what it does…” what are constructive ways to infuse into research the realities of policy and practice? On ethical grounds, potential users of research ought to have a say in how it is framed, especially if the results are expected to have consequences (intended or unintended); in addition, though, I believe that science is actually improved when the “wisdom of practice” is consciously incorporated at the level of theory-building and research design. Economics, for example, became a better science with attention to realities of behavior and thought.10I touch on this point in my essay, “Evidence and Advocacy,” found in Past as Prologue: The National Academy of Education at 50.
Washington, DC: National Academy of Education, 2015More Info →
I borrow the phrase “wisdom of practice” from Lee Shulman, with gratitude.
Finally, at least for now, is it accurate to assert that social science is special for its dual goals to “build a better science and to build a better world…”? I imagine our colleagues in chemistry, medicine, physics, geology, and biology might wonder why they are omitted from this remarkable duality.

Prewitt’s is not the last word on this topic, but it’s a big word that warrants careful attention and continued debate. At a minimum, heightened sensitivity to the issues he raises should elevate public and professional understanding; whether we should hold him accountable for the ultimate application of his wisdom is, of course, another matter.

Posted on July 19, 2016

References:

1
Quoted by Lee Cronbach, in “A Valedictory: Reflections on 60 Years in Educational Testing,” Board on Testing and Assessment, National Academy of Sciences, 1995.
Read more here →
2
Quoted in Phillipe Malrieu, Hommage a Henri Wallon (Toulouse: Universite de Toulouse, 1981)
3
Sheldon White, “Conceptual Foundations of IQ Testing,” Psychology, Public Policy, and Law 6, no. 1 (Mar 2000): 33–43.  More info →
4
See also Michael Feuer, “Measuring Accountability When Trust Is Conditional,” Education Week, September 26, 2012.
Read here →
5
For trends in federal science funding see the NSF supplement on federal R&D funding.
Read here →
6
Cambridge, MA: Harvard Education Press, 2016.More Info →
7
Washington, DC: National Academy Press, 2012More Info →
8
See Herbert Simon, “From Substantive to Procedural Rationality,” in Method and Appraisal in Economics, edited by S. J. Latsis, 129–148 (Cambridge, UK: Cambridge University Press, 1976).
More Info →
My application to education research and policy is in Michael Feuer, Moderating the Debate: Rationality and the Promise of American Education (Cambridge, MA: Harvard Education Press, 2006).

More Info →
9
See Richard Shavelson, et al, Scientific Research in EducationWashington, DC: National Academies Press, 2002.More Info →
10
I touch on this point in my essay, “Evidence and Advocacy,” found in Past as Prologue: The National Academy of Education at 50.
Washington, DC: National Academy of Education, 2015More Info →
I borrow the phrase “wisdom of practice” from Lee Shulman, with gratitude.