Following my previous post on Haavelmo, here is a longer description of why I like the guy and what our disagreements are in the light of his major paper.
In 1944, Trygve Haavelmo published a 124 pages paper in Econometrica that would set the stage for almost four decades of research in econometrics. It is extremely interesting to examine this fundational milestone, especially to contrast it with Haavelmo's subequent writings when he became president of the Econometric Society in 1958 and when he received the Nobel prize in 1989. I think we can see Haavelmo's thought slowly changing and a way for empirics to enter economics emerge, thrive and slowly come to a halt. I think it is important to understand where we come from, where we stand and where we go. Haavelmo is also an extremely clean example of mixing engineering with science, and of putting too much faith on economic theory. In the meantime, Haavelmo's legacy is beautiful, and many of the concepts that he and his fellows at the Cowles Commission have built are extremely useful today, especially to think about causality and how it relates to the observed data. The theoretical apparatus that they set up is nothing short of impressive and awesome. In this post, I want to guide you through three moments of Haavelmo's life and of that of econometrics.
What is extremely surprising in Haavelmo's writings is how intricately the scientific and engineering aspects of economics are intertwined. The scientific ambition is especially strong in the beginning of the 1944 monograph. But then, Haavelmo seems to lose sight of this aim when he becomes more practical and technical. I'm going to start with a description of the scientific aspect of econometrics as Haavelmo sees it. Then, I'll describe some strange renouncements when he becomes more practical and I'll finish with a description of the lore limited goal he sets for econometrics.
The role of Econometrics for testing economic theories
In the beginning of the paper, Haavelmo defines econometric research:
The method of econometric research aims, essentially, at a conjunction of economic theory and actual measurements, using the theory and technique of statistical inference as a bridge pier.
So we are going to try to relate the theories in economics to their counterpart in the actual real world. Theory is necessary, but it is only the first leg of a science, as Haavelmo recognizes building upon Pareto:
Theoretical models are necessary tools in our attempts to understand and "explain" events in real life. Within such theoretical models we draw conclusions of the type, "if A is true, then B is true." Also, we may decide whether a particular statement or a link in the theory is right or wrong, i.e., whether it does or does not violate the requirements as to inner consistency of our model. As long as we remain in the world of abstractions and simplifications there is no limit to what we might choose to prove or to disprove; or, as Pareto has said, "Il n'y a pas de proposition qu'on ne puisse certifier vraie sous certaines conditions, A determiner." Our guard against futile speculations is the requirement that the results of our theoretical considerations are, ultimately, to be compared with some real phenomena. [All emphasis in the quotes are mine.]
I like a lot this part of Haavelmo's paper because he really puts forward that economics is an empirical science. Actually, economics is not all about mathematical economics, as he explains later:
One of the most characteristic features of modern economic theory is the extensive use of symbols, formulae, equations, and other mathematical notions. Modern articles and books on economics are "full of mathematics." Many economists consider "mathematical economics" as a separate branch of economics. The question suggests itself as to what the difference is between "mathematical economics" and "mathematics." Does a system of equations, say, become less mathematical and more economic in character just by calling x "consumption," y "price," etc.? There are certainly many examples of studies to be found that do not go very much further than this, as far as economic signifiance is concerned. But they hardly deserve the ranking of contributions to economics. What makes a piece of mathematical economics not only mathematics but also economics is, I believe, this: When we set up a system of theoretical relationships and use economic names for the otherwise purely theoretical variables involved, we have in mind some actual experiment, or some design of an experiment, which we could at least imagine arranging, in order to measure those quantities in real economic life that we think might obey the laws imposed on their theoretical namesakes. For example, in the theory of choice we introduce the notion of indifference surfaces, to show how an individual, at given prices, would distribute his fixed income over the various commodities. This sounds like "economics" but is actually only a formal mathematical scheme, until we add a design of experiments that would indicate, first, what real phenomena are to be identified with the theoretical prices, quantities, and income; second, what is to be meant by an "individual"; and, third, how we should arrange to observe the individual actually making his choice.
I really love this part: economics is not math because it says something about the world, and, even more critically, because it involves the design of an experiment. What are the experiments that we can perform in economics? Haavelmo distinguishes two:
A design of experiments (a prescription of what the physicists call a "crucial experiment") is an essential appendix to any quantitative theory. And we usually have some such experiments in mind when we construct the theories, although-unfortunately-most economists do not describe their designs of experiments explicitly. If they did, they would see that the experiments they have in mind may be grouped into two different classes, namely, (1) experiments that we should like to make to see if certain real economic phenomena-when artificially isolated from "other influences"-would verify certain hypotheses, and (2) the stream of experiments that Nature is steadily turning out from her own enormous laboratory, and which we merely watch as passive observers.
We can see that the first class of experiments corresponds to the classical definition of an experiment in the natural sciences. We try to devise a crucial experiment in order to test the predictions of the theory. In order to do so, we have to be able to isolate the phenomenon of interest from all the other influences. This is what scientific experiments are for. Haavelmo actually makes this clear in what follows:
In the first case we can make the agreement or disagreement between theory and facts depend upon two things: the facts we choose to consider, as well as our theory about them. As Bertrand Russell has said: "The actual procedure of science consists of an alternation of observation, hypothesis, experiment, and theory."'
And he goes on to acknowledge that economic science needs this kind of experiments badly and mostly:
Now, if we examine current economic theories, we see that a great many of them, in particular the more profound ones, require experiments of the first type mentioned above. On the other hand, the kind of economic data that we actually have belong mostly to the second type.
But what is this second type of experiments, the ones that Nature turns out from her enormous laboratory? Haavelmo tells more about them:
In the second case we can only try to adjust our theories to reality as it appears before us. And what is the meaning of a design of experiments in this case? It is this: We try to choose a theory and a design of experiments to go with it, in such a way that the resulting data would be those which we get by passive observation of reality. If we succeed in doing so, we become master of reality-by passive agreement.
So Havelmo, as Yule before him, ackowledges that we cannot run experiments in economics and that we have to take the facts as given to us by Nature, without the possibility of interfering with them. What do we do then? We have to assume that the data of passive observation results from our theory and somehow make them agree. We then master reality, at least by passive agreement. I think this is still confusing but it is extremely important because this view underlies most of the subsequent developments in econometrics. Haavelmo makes what he believes we should do clearer afterwards:
The economist is usually a rather passive observer with respect to important economic phenomena. He is not in a position to enforce the prescriptions of his own designs of ideal experiments. "Observational" variables, when contradicting the theory, leave the possibility that we might be trying out the theory on facts for which the theory was not meant to hold, the confusion being caused by the use of the same names for quantities that are actually different. The statistical observations available have to be "corrected," or the theory itself has to be adjusted. To use a mechanical illustration, suppose we should like to verify the law of falling bodies (in vacuum), and suppose our measurements for that purpose consisted of a series of observations of a stone (say) dropped through the air from various levels above the ground. To use such data we should at least have to calculate the extra effect of the air resistance and extract this element from the data. Or, what amounts to the same, we should have to expand the simple theory of bodies falling in vacuum, to allow for the air resistance (and probably many other factors). A physicist would dismiss these measurements as absurd for such a pur- pose because he can easily do much better. The economist, on the other hand, often has to be satisfied with rough and biased measurements.
I find this analogy with physics to be particularly enlightening. Because we only see the data of passive observation, we have to account for the fact that a lot of other relationships are confounding the relation of interest. Accounting for these influences means modeling all of these other influences. I think this is an impossible task if done at the same time as building the model and testing the theory. This is where Haavelmo mixes science with engineering and takes the field in a wrong direction. Let me explain. Science is about Cartesian slicing of reality into smaller subsets that are studied in isolation. Engineering is about combining these relationships in a computable model (which generally requires some degree of simplification, the choice of which makes modeling an art) and testing the predictions of the model against new data. What Haavelmo proposes is to blend all these steps into one unique estimation procedure: theory gives us all the theoretical slices and estimation from the data of a passive observation should be able to deliver the properties of the relationships of interest. This sounds crazy and unrealistic. It puts too much weight on the combination of data with theory. Actually, this is generally an ill-posed problem that does not have a solution with passive data alone. Some information is lacking and the system has too many unknown relationships and not enough independent information. Haavelmo was actually aware of that problem, that he called the problem of confluence and that we now call the problem of identification.
Autonomy and the problem of confluent relationships
Haavelmo makes it clear from the beginning that we are interested in relationships that it might not be possible to observe in the data, because they are confounded by other ones occurring simultaneously. The problem of estimating supply and demand curves out of a data set of price and quantities is the classical exemplification of this problem of lack of autonomy that we now call identification. Haavelmo recalls first the goal of economic research as Cartesian slicing:
Our hope in economic theory and research is that it may be possible to establish constant and relatively simple relations between dependent variables, y and a relatively small number of independent variables, x.
Just before that, he discusses the mere existence of constant economic laws:
We might be inclined to say that the possibility of such fruitful hypothetical constructions and deductions depends upon two separate factors, namely, on the one hand, the fact that there are laws of Nature, on the other hand, the efficiency of our analytical tools. However, by closer inspection we see that such a distinction is a dubious one. Indeed, we can hardly describe such a thing as a law of nature without referring to certain principles of analysis. And the phrase, "In the natural sciences we have stable laws," means not much more and not much less than this: The natural sciences have chosen very fruitful ways of looking upon physical reality. So also, a phrase such as "In economic life there are no constant laws," is not only too pessimistic, it also seems meaningless.
So, it is possible to find laws in economic life if we look at the phenomena in the right way. What is the problem of autonomy then?
Let us consider one such particular relation, say x1=f(x2, x3). In constructing such a relation, we reason in the following way: If x2 be such and such, x3 such and such, etc., then this implies a certain value of x1. In this process we do not question whether these "ifs" can actually occur or not. When we impose more relations upon the variables, a great many of these "ifs," which were possible for the relation x1=f(x2, x3) separately, may be impossible, because they violate the other relations. After having imposed a whole system of relations, there may not be very much left of all the hypothetical variation with which we started out. At the same time, if we have made a lucky choice of theoretical relations, it may be that the possible variations that are left over agree well with those of the observed variables.
So, because the data from passive observation are generated by the interaction of a lot of various phenomena, the actual variation that remains might be much less than the one we would need to test or validate one of the relationships of interest individually. Haavelmo then rightfully asks why we care about these fundamental relationships:
But why do we start out with much more general variations than those we finally need? For example, suppose that the Walrasian system of general-equilibrium relations were a true picture of reality; what would be gained by operating with this general system, as compared with the simple statement that each of the quantities involved is equal to a constant? The gain is this: In setting up the different general relations we conceive of a wider set of possibilities that might correspond to reality, were it ruled by one of the relations only. The simultaneous system of relations gives us an explanation of the fact that, out of this enormous set of possibilities, only one very particular one actually emerges. But once this is established, could we not then forget about the whole process, and keep to the much simpler picture that is the actual one? Here is where the problem of autonomy of an economic relation comes in.
We care about the deeper relationships because they give us insights into many more possible variations that the ones we actually see in the data. These relations are true even if some other relations in the system get altered by economic policy or some other event: they are autonomous from these changes, hence the term autonomy. Autonomy is nice because it gives a relationship a lot of power to predict things that would happen would we change the environment in some way. Haavelmo illustrates the notion of autonomy with what I think is a beautiful analogy: that of a car.
The meaning of this notion, and its importance, can, I think, be rather well illustrated by the following mechanical analogy: If we should make a series of speed tests with an automobile, driving on a flat, dry road, we might be able to establish a very accurate functional relationship between the pressure on the gas throttle (or the distance of the gas pedal from the bottom of the car) and the corresponding maximum speed of the car. And the knowledge of this rela- tionship might be sufficient to operate the car at a prescribed speed. But if a man did not know anything about automobiles, and he wanted to understand how they work, we should not advise him to spend time and effort in measuring a relationship like that. Why? Because (1) such a relation leaves the whole inner mechanism of a car in complete mystery, and (2) such a relation might break down at any time, as soon as there is some disorder or change in any working part of the car. We say that such a relation has very little autonomy, because its existence depends upon the simultaneous fulfilment of a great many other relations, some of which are of a transitory nature. On the other hand, the general laws of thermodynamics, the dynamics of friction, etc., etc., are highly autonomous relations with respect to the automobile mechanism, because these relations describe the functioning of some parts of the mechanism irrespective of what happens in some other parts.
We want deeper, more autonomous relationships because they enable us to make sound predictions when circumstances in the economy change because they stay the same when the policy change occurs. They are autonomous with respect to this policy change:
The principal task of economic theory is to establish such relations as might be expected to possess as high a degree of autonomy as possible. Any relation that is derived by combining two or more relations within a system, we call a confluent relation. Such a confluent relation has, of course, usually a lower degree of autonomy (and never a higher one) than each of the relations from which it was derived, and all the more so the greater the number of different relations upon which it depends.
A classical example of a non autonomous relationship is that of the correlation between prices and quantities on a market other time. This correlation breaks down as soon as the supply or demand function changes. It could be after the introduction of a tax or a subsidy. The very famous Lucas critique of econometric models is simply a mere restatement of the notion of autonomy, which Lucas actually ackowledges (footnote 3).
Identification: not enough data, too much theory
One of the key questions for econometricians aware of the confluence problem is to be able to extract relationships with a high degree of autonomy with observational data where the actual amount of variation is deeply limited by the interaction of several relationships:
In scientific research-in the field of economics as well as in other fields-our search for "explanations" consists of digging down to more fundamental relations than those that appear before us when we merely "stand and look." Each of these fundamental relations we conceive of as invariant with respect to a much wider class of variations than those particular ones that are displayed before us in the natural course of events. Now, if the real phenomena we observe day by day are really ruled by the simultaneous action of a whole system of fundamental laws, we see only very little of the whole class of hypothetical variations for which each of the fundamental relations might be assumed to hold.
How is it possible to resolve this tension? In my opinion, Haavelmo does not take this problem seriously enough and tends to consider it as a technical issue. He, and subsequent researchers at the Cowles Commission, merely tend to make the resolution of this problem a property of the theory itself. It is the so-called identification problem. Haavelmo is well-aware of the problem, even if he does not use the term identification itself, that will be coined later by Koopmans:
We may fail to recognize that one or more of the parameters to be estimated might, in fact, be arbitrary with respect to the system of equations. This is the statistical side of the problem of autonomous relations. Suppose that, in particular, it is possible to derive an infinity of new systems which have exactly the same form as the original system, but with different values of the coefficients involved. Then, if we do not know anything about the values of the parameters in the original equation system, it is clearly not possible to obtain a unique estimate of them by any number of observations of the variables. And if we did obtain some "estimate" that appeared to be unique in such cases, it could only be due to the application of estimation formulae leading to spurious or biased results. For example, the question of deriving both demand and supply curves from the same set of price-quantity data is a classical example of this type of problems.
Indeed, in the supply and demand case, a rigorous investigation of the theoretical model should make us aware that it is impossible to recover the properties of the demand and the supply curve from price-quantity data alone. If we were to run a regression between prices and quantities, we would have a spurious confluent coefficient.
This question (in the case of linear relations known as the problems of multicollinearity) is of great importance in economic research, because such research has to build, mostly, on passive observations of facts, instead of data obtained by rationally planned experiments. And this means that we can obtain only such data as are the results of the economic system as it in fact is, and not as it would be under those unrestricted hypothetical variations with which we operate in economic theory, and in which we are interested for the purpose of economic policy.
So how are we going to solve for the fact that the data obtained from passive observations might not give us enough variation to pin down (identify) the autonomous relationships? For Haavelmo, this is mainly a technical problem, a property of the system of simultaneous equations that we think has generated the data:
In the following we shall see that the investigation of this problem of indeterminate coefficients, as well as other questions of estimation in relation to economic equation systems, all come down to one and the same thing, namely, to study the properties of the joint probability distribution of the random (observable) variables in a stochastic equation system
This problem of going from the observed data to the deep invariant autonomous parameters of interest, that has been later called the identification problem, has fascinated economists for a long time. And it all started with Haavelmo attracting the attention of the profession on this issue:
This problem, however, is of particular significance in the field of econometrics, and relevant to the very construction of economic models, and besides, this particular mathematical problem does not seem to have attracted the interest of mathematicians.
So for Haavelmo, this is a both a theoretical and a technical problem: is there a sufficient amount of variation in the assumed economic system of simultaneous equations to be able to go from the data to the deep autonomous relationships? This problem has attracted a lot of research since then, and some of it is still underway. Nowadays, and following Anderson and Rubin (1949), that themselves follow Frish and Haavelmo, we talk about structural and reduced form relationships instead of autonomous and confluent. Frish used the terms superflux and coflux.
Where does this lead us? Well, identification is a property of the system of theoretical relations that we postulate. So that all our inference is going to be conducted conditional on this system being restricted enough so that it is identified. Koopmans, Rubin and Leipnick (1950) recognize this fact in a chapter of the famous Cowles Commission monograph #10: the structural model is identified under a set of a priori restrictions. For example, it was well known since Phillip Wright that supply and demand models are identified if there exists a shifter of supply that is restricted not to affect demand and a shifter of demand that does not affect supply. As a consequence, the identification of economic relationships and all of our empirical knowledge rests on a set of untestable assumptions. The key question now becomes: how to justify these assumptions? They tend to be extremely important, but for a long time in economics assuming these restrictions seemed almost unimportant and was left to footnotes. Since the early 90s, Josh Angrist and others have done a lot to put these restrictions back into the picture, and to discuss them. Basically, these restrictions are the experiments that we are postulating in the data to be able to tests our theories. They better be reliable. Hence the quest for natural experiments that would be a credible source of identification of autonomous relationships and of testing of economic theories.
As for Haavelmo, his discarding of the possibility of running experiments in economics and his faith into the mathematical analysis of the identifiability of a set of simultaneous equations stemming from economic theory as a surrogate for good experiments did not seem to bother him too much. Two things strike me. First, how can a man so adamant about testing theories and finding crucial experiments can settle for such an insatisfactory device as a priori restrictions when it comes to the actual implementation of econometric analysis? I suspect the influence of Frish, who had been his master and had already outlined the path of research along these lines. Frish was apparently a very energetic and charismatic figure, and it is possible that his respect for Frish explains the apparent schizophrenia of Haavelmo's paper. This approach of identification with a priori restrictions has persisted until today, and papers are published in leading journals studying the restrictions needed to identify some quantities of interest. Second, I am surprised by how much Haavelmo mixes science with engineering in his paper. The separate equations are studied simultaneously under the a priori identifying restrictions. Each of them is then interpreted as an autonomous relationship with causal implications, and as a test of some theories. This is the scientific aspect of the endeavor. The equations are then combined in order to predict a policy change, the engineering part. All of this analysis rests upon the validity of the a priori restrictions only. Some work in economics still uses this approach of causally interpreting structural models identified by a priori restrictions.
Actually, Haavelmo predates the credibility revolution on the engineering side when he advocates for the following cycle, that opens up the need for new data to validate the predictions of the model, a sound engineering advice:
If we have found a certain hypothesis, and, therefore, the model behind it, acceptable on the basis of a certain number of observations, we may decide to use the theory for the purpose of predictions. If, after a while, we find that we are not very successful with these predictions, we should be inclined to doubt the validity of the hypothesis adopted (and, therefore, the usefulness of the theory behind it). We should then test it again on the basis of the extended set of observations.
Haavelmo is also well-aware of the Popperian limitation to knowledge:
Now suppose that we have a set of observations that all confirm the statements that are permissible within our model. Then these statements become facts interpreted in the light of our theoretical model, or, in other words, our model is acceptable so far as the known observations are concerned. But will the model hold also for future observations? We cannot give any a priori reason for such a supposition. We can only say that, according to a vast record of actual experiences, it seems to have been fruitful to believe in the possibility of such empirical inductions.
Disenchantment
Along the years, Haavelmo has grown dissatisfied with the actual results of the research program that he had delineated in his 1944 landmark paper. In his 1958 presidential address to the econometric society, he goes as far as saying:
The concrete results of our efforts at quantitative measurements often seem to get worse the more refinement of tools and logical stringency we call into play!
He then describes the progress that econometricians, a lot of them linked to the Cowles Commission, have made in the direction of solving the identification problem and proposing estimation devices for simultaneous equations models. And he concludes:
But the concrete results of these efforts have often been a seemingly lower degree of accuracy of the would-be economic laws (i.e., larger residuals), or coefficients that seem a priori less reasonable than those obtained by using cruder or clearly inconsistent methods.
What is to blame for this apparent failure? For Haavelmo, the main responsibility lies within the insufficiencies of economic theory:
I think we may well find part of the explanation [...] in the shortcomings of basic economic theory.
Haavelmo then proposes two directions in which to improve economic theory: first, including what people actually think (their expectations) in models, and second to relax the stability of preferences and make them dependent on neighbors, friends and so on. But I think he does not touch upon the key issue. When talking about what has been learned, he says:
We have been striving to develop more efficient statistical methods of testing hypotheses and estimation. In particular, we have been concerned about general principles of consistent procedure in the important field of multiple regression technique. We have found certain general principles which would seem to make good sense. Essentially, these principles are based on the reasonable idea that, if an economic model is in fact "correct" or "true," we can say something a priori about the way in which the data emerging from it must behave. We can say something, a priori, about whether it is theoretically possible to estimate the parameters involved. And we can decide, a priori, what the proper estimation procedure should be. We have learned to understand the futility of arguing that the data "in practice" may behave differently, because such an argument would simply mean that we contradict our own model.
So he remains completely faithful to the a priori approach. And he goes as far as saying that the fact that the data might contradict the model is futile. If I can understand this in the context of identification analysis, it still is a very poor way of seeing empirical validation. In his 1989 Nobel prize reception speech, he blames the theory again for the failure of the econometrics program:
the possibility of extracting information from observations of the world we live in depends on good economic theory. Econometrics has to be founded on theories that describe in a reasonably accurate way the fashion in which the observed world has operated in the past. [...] I think existing economic theories are not good enough for this purpose.
A final word
As a conclusion, I cannot help thinking of the wonderful adventure of understanding the sources of the identification problem as a beautiful endeavor but at the same time as a waste of time. Would looking for actual crucial experiments in practice not been a sounder way of spending this huge amount of energy? I think Haavelmo and the Cowles Commission members were much too theoretically oriented to be really fascinated by the adventure of realistically looking at the data with a crucial experiment in mind. But their work was fascinating as a way to encode and understand causlaity and the difficulties iof causal inference. Nowadays, any applied empirical paper has to discuss its identification strategy: how it extracts causality from observational data. Also, the apparatus used to study causality and identification using simultaneous equation models have resurfaced recently in artificial intelligence: Judea Pearl has ackowledged the legacy of Haavelmo when thinking about extracting causal factors from observational data (p.158).
I have been fascinated by all these works and have studied them in detail. What I think now was that we were quick in declaring defeat over the data and quick to call a priori restrictions to the rescue. I am happy that the data makes its way into economics big time. I cannot help thinking about Claude Bernard, the father of experimental medicine, that said in his 1865 book on experimental medecine that:
Experimentation is undeniably harder in medecine than in any other science; but for that very reason it was never so necessary, and indeed so indispensable. The more complex a science, the more important is it, in fact, to establish a good experimental standard, so as to secure comparable facts, free of sources of error. Nothing os today, I believe, more important to the progress of medecine.
Replacing medicine by economics would seem to describe the status of our science nowadays. We are in the middle of a credibility revolution where data and crucial experiments enter the picture big time. Here, I cannot help thinking of the young Haavelmo, that, in the conclusion of his landmark paper, acknowledges that econometrics seems pretty technical, and tongue in cheek, that we might want to dispense with it:
The patient reader, now at the end of our analysis, might well be left with the feeling that the approach we have outlined, although simple in point of principle, in most cases would involve a tremendous amount of work. He might remark, sarcastically, that. "It would take him a lifetime to obtain one single demand elasticity." And he might be inclined to wonder: Is it worthwhile?
He then concludes, going back to his good initial intentions:
If economics is to establish itself as a reputable quantitative science, many economists will have to revise their ideas as to the level of statistical theory and technique and the amount of tedious work that will be required, even for modest projects of research. [...] In other quantitative sciences the discovery of "laws," even in highly specialized fields, has moved from the private study into huge scientific laboratories where scores of experts are engaged, not only in carrying out actual measurements, but also in working out, with painstaking precision, the formulae to be tested and the plans for the crucial experiments to be made. Should we expect less in economic research, if its results are to be the basis for economic policy upon which might depend billions of dollars of national income and the general economic welfare of millions of people?
I could not agree more.
No comments:
Post a Comment