Julian Reiss, Error in Economics. Towards a more evidence-based methodology, London, Routledge, 2007.

Julian Reiss’ monograph presents a normative case for an empiricist methodology in economics. According to the author, economists should focus on measuring social variables in order to find causal relationships among them. They should investigate the stability of these relationships in different contexts and, in case it exists, inquire about the underlying abstract factors and establish economic laws. In this process, economists should bear in mind the practical purposes their research is supposed to serve (if there are any) so that the best methods to achieve those aims are always chosen.

The case for this evidence-based approach is articulated in nine chapters divided into three parts. In part one, Reiss discusses economic measurement in a single case study: the Consumer Price Index (CPI) controversy. The second part focuses on causality, drawing on different instances of actual economic research: experimental economics, thought experiments, instrumental variables, etc. The author is pessimistic about the number of economic factors with stable causal capacities found by economists so far, and therefore the third part is not about economic laws, but rather about the prospects of getting them and using them in policy-making. Provided there are good evidential grounds, the more pluralistic economists are in their methodologies, the more likely it seems they will be successful, even if the author makes no promises about it: “the empirical road has not been walked yet and we do not know where it would lead us if we do walk it” (183).

Reiss’ argument for a more evidence-based methodology does not rely on the virtues of a particular technique he wants to promote, but rather on the problems of the standard theory-driven approach discussed in the book. His trust in the possibility of an evidence-based alternative is a matter of philosophical conviction: if the epistemic prejudices of the economic profession against purely inductive methodologies can be removed, these could flourish and yield a more convincing economics. Reiss’ conviction owes much to Nancy Cartwright’s philosophy of the social sciences. This allows him, for instance, to be critical about the ability of mainstream economists to use clinching methods of causal inference. One possible response that Kevin Hoover explores below is that they may have been walking the vouching road to causality, a less demanding one but with no result in the end either. This does not imply that it cannot be done, but it should temper the author’s methodological hopes (and perhaps explain the profession’s pessimism). Aris Spanos challenges in turn the conceptual foundations of this optimism: Reiss can invoke Bacon and be confident on the possibility of correcting errors through a variety of methods to deal with evidence, but it takes a bit more statistical sophistication to define evidence in a way which warrants real error-correction.

Reiss’ programme hinges on evidence and causality, but also on a pragmatic view on values: evidence is sound when it gives a licence to act and attain our epistemic and non-epistemic aims. Depending on our purposes, we should choose the more adequate methodology to gather the relevant evidence. Reiss denounces how economists focus on certain kinds of epistemic values which prevent them from gathering the evidence politicians request (ch. 5, 10). Methodologists and economists are also criticised for adopting epistemic values that constrain empirical research without much scientific profit (ch. 8-9).

Error in economics opens with the explicit admission that evidence alone will not solve neither our theoretical nor practical disagreements: values are required to decide which is the correct CPI (ch. 2-4). But values are diverse and make us disagree, no matter how much causal evidence is gathered. Why should we expect evidence-based economics to overcome such controversies about values? An inspiring example for Reiss seems to be the community of physicists who constructed a unified scale to measure temperature throughout a century of experiments, even if there was no definition of temperature on which they could agree a priori. In his Inventing Temperature (2007) Hasok Chang shows that their success is better explained by a coherentist epistemology with a minimum of theoretical commitments. In order to measure temperature, a scale is necessary, but at first there were many and all of them arbitrary to a certain point. I.e., there was no self-evident basis to ground the inquiry. The choice between all these scales, claims Chang, was made possible by an implicit agreement on a set of epistemic values that presided over the experiments, independently of the theoretical commitments of each physicist. Gradually, the accumulation of evidence made them agree on a scale of measurement under the coherentist pressure of their shared values. In a similar vein, Reiss argues that if economists reached an empirically oriented agreement on their epistemic and non-epistemic aims, their measurements would progress (more than it did in the past century, at least). Yet, he admits, non-epistemic values make economists disagree as much as epistemic values. Can we expect that, once they agree on this latter, the former will cease to disturb their agreement?

I suspect that Reiss optimism about evidence-based economics presupposes some sort of affirmative answer. My own pessimism is based on the existence of empirical traditions in economics where, despite a certain number of shared values, disagreement in measurement persists. My example of choice is Milton Friedman, whose 1953 methodological piece is surprisingly parallel to Reiss’ in some respects. Friedman wanted the economic profession to align around the accuracy of predictions, keeping theoretical commitments to a minimum. The lighter the theoretical apparatus, the easier would be to classify data in separate markets (Marshallians are epistemic contextualists) and obtain predictions that should be judged for their success alone. Besides, for our practical purposes, argued Friedman, we need nothing more than predictive success (close to pragmatism as many were in Chicago). As Kevin Hoover puts it below, Friedman was an empiricist “just as Reiss thinks we ought to be”.

My pessimism about evidence-based economics à la Reiss comes thus from the fate of Friedman’s own evidence-based agenda. I have studied elsewhere how Friedman’s work on consumption usually elicited controversies that never yielded a professional consensus, despite both its solid methodological grounds and the accuracy of his predictions (Teira forthcoming). An issue very often at stake in these controversies was the definition of the variables in the analysis. Friedman’s predictions were always grounded on a classification of data under economic variables without much theoretical clinching. As he once put it, a commodity X is just a label, “not a word for a physical or technical entity to be defined once and for all independently of the problem at hand”. When the economist defined a commodity X once and for every market (as Walrasians intended), empirical analysis became increasingly complicated. More strict definitions, thought Friedman, made data sets unusable and, yet, economists are very often urged to use them by police-makers for whom a suboptimal response is better than none. And economists, thought Friedman, should provide it. Coherence in the definition of variables across their many uses did not count as much for Friedman as it did for Chang’s thermologists, who could safely ignore the practical implications of their experiments.

Yet, no matter how accurate Friedman’s predictions were, if anyone disagreed, one could always contest the definition of the variables used to obtain the predictions: “these data are not X” or “these data should be X as well” – Reiss himself notices this regarding money (p. 130). Unlike temperature, economic variables are difficult to define on the basis of a set of uncontroversial experiments acknowledged by everyone in the profession. As the CPI controversy shows, when disagreement on values occurs, disagreement on measurement procedures follows. Hence progress in economic measurement will only happen if either the community shares epistemic and non-epistemic values (so that they don’t have practical incentives to disagree) or the measurement procedure is strong enough to overcome political divergences.

Good science, says Reiss, “reduces the prejudices and preconceptions of the individual scientist to a minimum”. We may legitimately ask how evidence-based economics achieves this reduction and succeeds when other empiricist agendas in economics failed. Evidence-based economics is a “research project under construction” and indeed one of its many merits is that it raises issues that other approaches in economics neglect. It is open to debate whether Reiss’ evidence-based alternative offers any real hope of solving them, but epistemic optimists should certainly try to walk this road.

{January 2009}
{Economics and philosophy, 25.2 (2009), pp. 199-201}