I’ve decided to put SKY on hold, and to focus on my research instead. It is not that I do not think that SKY or something like SKY is not necessary. On the contrary. But I can see myself trying to juggle SKY with my research and with being a parent, and I can see myself failing at all three or trying to do two well. And SKY has to go.
I’ve learned a lot in the meantime.
The first thing I’ve learned is that everyone agrees that SKY or something like SKY is needed and would be much better than what we have now. I’ve heard no-one trying to defend the current publication system as being the best of all possible worlds. I’ve heard no-one arguing that paywalls are great and that they love inaccessible datasets and code. I’ve heard no-one saying that we do not need more aggregation, more collaboration, and less publication bias. I’ve heard a lot of people being frustrated with the current system of prestige publications, on both ends of the spectrum, both those who are in and those who are out.
The second thing I’ve learned is that it is feasible. We can preregister and publish individual research results, write comments and referee reports on them and vote for the ones we prefer, aggregate them with constantly updated meta-analysis and write collaborative consensus articles interpreting the results. We can do it. Together.
The third thing I've learned is that I was not original in bringing up SKY. Some people had already thought about it, and some even started implementing something very similar. As always in research, independent discovery is a reminder that you should stay humble.
The fourth thing I’ve learned is how change is hard. Oh my god, little did I know how hard it was. In retrospect, I was completely, ridiculously, naive with respect to how we can make change happen. I should have known better, as if any change has ever come easily. But I did not know better, wishful thinking, naiveté and probably lack of experience explain that. I’ve learned, and I’m happy I’ve learned.
Change is hard for several reasons. First, it is not clear how to make it happen. What is the politically most savvy and most respectful way to do things. I’ve tried a lot and failed a lot in that respect. When I started with my first blog on the empirical revolution in economics, I was sure that people would get the bottom part of my message, that I was worried about rampant publication bias and Questionable Research Practices in economics and that we would have something like a “Aha” moment, and start collectively thinking about how to make things better. But that part of the post went AWOL, people liked the self-congratulatory content, not the more difficult self-reflecting part. OK, I took that and I wrote a series on the problems with p-values. People seemed not to read much through this, and got lost in the details I guess. OK, so I wrote my blog post on why we cannot trust the published empirical findings in economics, and here, I’ve had two very different reactions: on the one hand, a lot of people, most of them outside of Econ, were on board and found what I was saying pretty clear and convincing. On the other hand, most people in Econ just ignored my post. No reaction. Deafening silence. I guess that’s the sound of you being right but also annoying. OK, so then I created SKY in order to show that it was not all talking and all criticising, that there were solutions. Here, two types of reactions: enthusiastic from some (almost 200 likes 🙂) and positive but skeptical from most others. “It seems very nice in practice, but how can we make that happen?" were they saying. “With the current way of promoting researchers and allocating research funds, it is impossible to get there. You would need a complete overhaul, good luck for making that happen.”
Second, change requires a lot of work and a lot of available time. I do not have time right now to put the work in, unless I decrease the quality of my research output or I decrease my quality as a dad, or I burn out. I cannot decrease the quality of my research output because this is the key for me to stay relevant within the field. So I have to buckle up, accept the current publication system with all its flaws, and accept to play along or become irrelevant, or get out of research and prioritise SKY, with the risk of losing expertise and respect in the field, and go against my intuition that SKY or something like SKY can only come from inside the research community. Maybe I’m wrong on that, I’m willing to listen. Maybe in the transition time from the current system to SKY or something like SKY, we need professional people getting into SKY or something like SKY full time to get it out of the ground. But then, I should work on SKY full time, and as a consequence completely jeopardise my research. And I cannot do that. I love my research projects and I have to see them through. I cannot let them down right now. They are my way to better things within my research community.
Third, change requires to understand the politics of human endeavours and a good strategy, and I came to understand that in two key ways. First, I did not see that by claiming that “we could not trust the published empirical record in empirical economics”, I was striking a blow against my own house, and criticising the track record of my very community. Not a very clever move. Even if the community is highly akin to self-reflection (which econ partly is), the consequences of acknowledging your mistakes might be dismal. At least, people are not too kin into doing it because of the huge uncertainties as to how the general public is going to receive these kind of statements. For example, I was only saying that because of an unknown amount of publication bias, we do not really know what we know. I never said that everything in empirical economics is wrong, I said we do not know how much is. But a cursory reading of what I was saying might have concluded as such. And I was putting the entire house in peril because I had doubts. The key thing is whether I had doubts about foundations or about details. But even if my doubts were foundational, the best way to address them is probably one at a time, starting by assessing the level of publication bias in one question. Then in another. Etc. Start slow, small and within the community, so as not to lose its trust. Message received. Second, I did not anticipate that by questioning the current track record and practices of empirical economists, I might alienate them, or upset them. I was expecting something like a self-reflection: “Oh, I see, maybe we’re wrong, what could we do to make things better”. Yep, I was that naive. In retrospect, I’m amazed at my ignorance of what makes people tick. And that’s something I’ve learned repeatedly since: you have to factor the human component in, people fallibilities and imperfections and limitations. You cannot ask them to be their better selves all of the time, otherwise you’ll fail. Message received as well.
So eventually, is SKY the way to go? I’m pretty sure it is, and that 10, 20 to 30 years from now, maybe more, most research in the social sciences will look like SKY or something like SKY. But the way to go there is not to erase everything and build up afresh from the ground. I think it goes through careful piecemeal change of the existing structure. And some people are already doing that, with the introduction of registered reports in economics, for example, the data and code requests at most top journals, the arrival of meta-analysis and of tests for publication bias in top journals. We are still a very long way from producing reliable reproducible aggregated evidence, but it is growing and I’m going to contribute to that as much as I can with my own research.
OK, so thanks to all of those who have supported SKY and contributed to it. Thanks to everyone that gave me feedback on it. It has been a great occasion for learning and growth. I end up being more convinced that we need slow piecemeal changes than a complete sudden overhaul, even though I’m convinced that the end result will look a lot like what I (and a lot of others) have delineated.
If you feel like taking the flame, everything is open source, so you can just take over 😉
No comments:
Post a Comment