Why I'm putting SKY on hold

I’ve decided to put SKY on hold, and to focus on my research instead. It is not that I do not think that SKY or something like SKY is not necessary. On the contrary. But I can see myself trying to juggle SKY with my research and with being a parent, and I can see myself failing at all three or trying to do two well. And SKY has to go. 

I’ve learned a lot in the meantime. 

The first thing I’ve learned is that everyone agrees that SKY or something like SKY is needed and would be much better than what we have now. I’ve heard no-one trying to defend the current publication system as being the best of all possible worlds. I’ve heard no-one arguing that paywalls are great and that they love inaccessible datasets and code. I’ve heard no-one saying that we do not need more aggregation, more collaboration, and less publication bias. I’ve heard a lot of people being frustrated with the current system of prestige publications, on both ends of the spectrum, both those who are in and those who are out. 

The second thing I’ve learned is that it is feasible. We can preregister and publish individual research results, write comments and referee reports on them and vote for the ones we prefer, aggregate them with constantly updated meta-analysis and write collaborative consensus articles interpreting the results. We can do it. Together. 

The third thing I've learned is that I was not original in bringing up SKY. Some people had already thought about it, and some even started implementing something very similar. As always in research, independent discovery is a reminder that you should stay humble.

The fourth thing I’ve learned is how change is hard. Oh my god, little did I know how hard it was. In retrospect, I was completely, ridiculously, naive with respect to how we can make change happen. I should have known better, as if any change has ever come easily. But I did not know better, wishful thinking, naiveté and probably lack of experience explain that. I’ve learned, and I’m happy I’ve learned. 

Change is hard for several reasons. First, it is not clear how to make it happen. What is the politically most savvy and most respectful way to do things. I’ve tried a lot and failed a lot in that respect. When I started with my first blog on the empirical revolution in economics, I was sure that people would get the bottom part of my message, that I was worried about rampant publication bias and Questionable Research Practices in economics and that we would have something like a “Aha” moment, and start collectively thinking about how to make things better. But that part of the post went AWOL, people liked the self-congratulatory content, not the more difficult self-reflecting part. OK, I took that and I wrote a series on the problems with p-values. People seemed not to read much through this, and got lost in the details I guess. OK, so I wrote my blog post on why we cannot trust the published empirical findings in economics, and here, I’ve had two very different reactions: on the one hand, a lot of people, most of them outside of Econ, were on board and found what I was saying pretty clear and convincing. On the other hand, most people in Econ just ignored my post. No reaction. Deafening silence. I guess that’s the sound of you being right but also annoying. OK, so then I created SKY in order to show that it was not all talking and all criticising, that there were solutions. Here, two types of reactions: enthusiastic from some (almost 200 likes 🙂) and positive but skeptical from most others. “It seems very nice in practice, but how can we make that happen?" were they saying. “With the current way of promoting researchers and allocating research funds, it is impossible to get there. You would need a complete overhaul, good luck for making that happen.” 

Second, change requires a lot of work and a lot of available time. I do not have time right now to put the work in, unless I decrease the quality of my research output or I decrease my quality as a dad, or I burn out. I cannot decrease the quality of my research output because this is the key for me to stay relevant within the field. So I have to buckle up, accept the current publication system with all its flaws, and accept to play along or become irrelevant, or get out of research and prioritise SKY, with the risk of losing expertise and respect in the field, and go against my intuition that SKY or something like SKY can only come from inside the research community. Maybe I’m wrong on that, I’m willing to listen. Maybe in the transition time from the current system to SKY or something like SKY, we need professional people getting into SKY or something like SKY full time to get it out of the ground. But then, I should work on SKY full time, and as a consequence completely jeopardise my research. And I cannot do that. I love my research projects and I have to see them through. I cannot let them down right now. They are my way to better things within my research community.  

Third, change requires to understand the politics of human endeavours and a good strategy, and I came to understand that in two key ways. First, I did not see that by claiming that “we could not trust the published empirical record in empirical economics”, I was striking a blow against my own house, and criticising the track record of my very community. Not a very clever move. Even if the community is highly akin to self-reflection (which econ partly is), the consequences of acknowledging your mistakes might be dismal. At least, people are not too kin into doing it because of the huge uncertainties as to how the general public is going to receive these kind of statements. For example, I was only saying that because of an unknown amount of publication bias, we do not really know what we know. I never said that everything in empirical economics is wrong, I said we do not know how much is. But a cursory reading of what I was saying might have concluded as such. And I was putting the entire house in peril because I had doubts. The key thing is whether I had doubts about foundations or about details. But even if my doubts were foundational, the best way to address them is probably one at a time, starting by assessing the level of publication bias in one question. Then in another. Etc. Start slow, small and within the community, so as not to lose its trust. Message received. Second, I did not anticipate that by questioning the current track record and practices of empirical economists, I might alienate them, or upset them. I was expecting something like a self-reflection: “Oh, I see, maybe we’re wrong, what could we do to make things better”. Yep, I was that naive. In retrospect, I’m amazed at my ignorance of what makes people tick. And that’s something I’ve learned repeatedly since: you have to factor the human component in, people fallibilities and imperfections and limitations. You cannot ask them to be their better selves all of the time, otherwise you’ll fail. Message received as well.

So eventually, is SKY the way to go? I’m pretty sure it is, and that 10, 20 to 30 years from now, maybe more, most research in the social sciences will look like SKY or something like SKY. But the way to go there is not to erase everything and build up afresh from the ground. I think it goes through careful piecemeal change of the existing structure. And some people are already doing that, with the introduction of registered reports in economics, for example, the data and code requests at most top journals, the arrival of meta-analysis and of tests for publication bias in top journals. We are still a very long way from producing reliable reproducible aggregated evidence, but it is growing and I’m going to contribute to that as much as I can with my own research.

OK, so thanks to all of those who have supported SKY and contributed to it. Thanks to everyone that gave me feedback on it. It has been a great occasion for learning and growth. I end up being more convinced that we need slow piecemeal changes than a complete sudden overhaul, even though I’m convinced that the end result will look a lot like what I (and a lot of others) have delineated.

If you feel like taking the flame, everything is open source, so you can just take over 😉

SKY: Towards Science 2.0

SKY, the Social Science Knowledge Accumulation Initiative (yep, SSKAI=SKY ;), announced in my previous blog post, is live. SKY ambitions to change the way we conduct research in the social sciences by leveraging the tools of the digital revolution in order to make science more accessible, open, collaborative and effective.

Accessible: by providing the most up-to-date and collectively discussed summaries on research questions.

Open: by providing the code and data behind the results, at the level of both aggregate and individual results.

Collaborative: by enabling and leveraging the active participation of all the researchers working on a topic to come to a consensus on the interpretation of the evidence.

Effective: by enabling a better coordination of researchers

Credible: by discouraging publication bias.

In this blog post, I’m going to describe my vision for SKY, which service it’s going to bring to its various users and why they are not brought by the current system, how SKY works, and how you can contribute to it. One subsequent blog post will walk you through some examples, another will describe the links between SKY and the current publishing system while a third one will talk about the personal challenges that changing the status quo imply. A fourth blog post will be dedicated to discussing the concerns some of you might have with respect to the use of meta-analysis and how SKY ambitions to alleviate them. 

The vision for SKY

SKY ambitions to provide:

1. A collectively agreed upon set of scientific theories on human behavior that have been empirically validated and a set of major open questions where empirical validation is deemed essential and a priority.
2. A set of interventions that have been proven to work in the past along with the precision surrounding the estimate of their impact, a comparison of their cost-effectiveness and a set of open questions. 
3. A set of agreed upon methodological procedures and guidelines for testing theories and measuring the effects of interventions and a set of open methodological questions.

SKY is continuously updated in order to reflect the most recent developments on each question.

SKY is decentralized and open source, meaning that anyone can contribute to SKY.

SKY is usable by anyone under the terms of the GNU GPL license.  

Which services will SKY bring

SKY will serve many purposes that are not well served at the moment:

1. The users of research (policy makers, charities, citizens, firms) will have instant access to the agreed-upon consensus on a topic. At the moment, knowledge is not aggregated and is behind paywalls, making it super hard to convey outside of the ivory tower. 
2. Funders will be able to identify the core questions that they want to see solved and put their money where their mouth is. At the moment, long back and forth are needed between funders and researchers to identify key questions, and to a large extent, funders defer to an unrepresentative set of researchers to select the important questions to fund.
3. Researchers will instantly identify open questions, find collaborators, preregister their analysis, report their results, receive feedback constantly. Once validated, their results will be instantly updated in the summary meta-analysis that will aggregate the whole knowedge on a question. Papers will be refereed constantly by anyone. Especially PhD students. Methods will be normalized, constantly validated, tested and discussed. At the moment, frontier questions are “in the air.” Refereeing is a closed process, extremely limited in time. There are still debates on which methods are most convincing and on how best to implement them. Corrections and replications are the exception.
4. Precision will be a core focus, and replications will be deemed worthy as long as precision will be deemed insufficient to come to a useful conclusion for policy makers or for scientific progress. At the moment, precision is not taken seriously, only significant results get reported, with spurious interpretations, publication bias and Questionable Research Practices (QRPs).
5. Publication bias will be a thing of the past. First, we will be able to test and correct for it on the past literature. Second, because all projects will eventually be included in the summary meta-analysis, irrespective of their statistical significance, there will be no incentive for publication bias anymore. 
6. We will never lose knowledge thanks to continuously updated meta-analysis. At the moment, the datasets built in a meta-analysis are often kept private by the authors and more often than not run the risk of being lost. Both of these issues prevent continuously updating our knowledge as new results come in.

The main components of SKY

In order to reach these goals, SKY uses the following tools:

1. Consensus articles: articles conveying the scientific consensus on a question (think Wikipedia meets IPCC report). These articles are freely editable by all but they are based on quantified accumulated evidence mainly taking the form of a continuously updated meta-analysis. Each time the meta-analytic result changes in a major way, the article wording will change as well to reflect the evolving consensus. Each article will contain a set of important open questions to be answered. As with Wikipedia, a discussion chamber will eventually be added in order to coordinate discussions and achieve consensus.
2. Continuously updated meta-analysis: meta-analysis are the statistically correct way to aggregate statistical evidence on a question. SKY maintains a database of estimates on each question of interest along with the code analyzing them. The results of the meta-analysis (the aggregated estimated effect size or cost-effectiveness ratio of a given intervention and its precision along with tests for publication bias and questionable research practices) are published on the consensus article page and serve as the basis for the consensus evaluation of an intervention/theoretical proposition. Each time a new individual research result is validated, the database is updated, immediately updating the results of the meta-analysis.
3. Dynamic research reports: the individual results in each meta-analysis are the outcome of individual research projects. These projects unfold on the website as follows: each researcher or team starts with a declaration of intent, that for the moment is public. Giving the option of keeping the declaration of intent private, or open to a subset of users, is in the books. The declaration of intent serves as a time-stamped pre-registration, and as such will help avoid most QRPs. Then the team reports on its advances and first results. All along the life-time of the project, the team can receive comments and suggestions, so as to amend the analysis. Making the declaration of intent public enables to receive suggestions at the early stages of a project, before data collection for example, which is when feedback is the most useful. Once the results are posted, along with data and code, others will try to reproduce them. The most effective persons to do that are probably masters and PhD students in methods class, under the supervision of their teacher. Refereeing will thus be continuous and in depth, from the birth of a project to forever. Once the analysis in the paper has been reproduced, the entry will be added to the meta-analysis. For already published papers, a link to the paper webpage, and a summary of its main results, will be a start. 
4. Methodological guidelines: for each method, we will have a set of agreed-upon guidelines on how to best implement them, which tests are required and why. This will be supplemented by an open source “Statistical Tools for Causal Inference” book that will introduce the basic way to implement the methods along with the statistical code. For frontier methodological research, consensus articles and dynamic research reports will also exist.

Why SKY is needed

We are in dire need of a tool to accumulate knowledge in the social sciences for at least three reasons.

First, the social sciences are not currently organized along collectively agreed-upon principles. There are many disciplines that try to explain human behavior in society, from psychology to sociology, from anthropology to economics, from political science to biology, and even physics. We have several major organizing theoretical principles, from homo oeconomicus to evolutionary psychology, from psychological biases to social inscription. We have a lot of interventions (nudges, Cognitive Behavioral Therapy, taxes, quotas, subsidies, norms...). We have a lot of methods to help us assess whether theories are correct or whether our interventions work. We have a lot of empirical evidence on both theories and interventions. BUT: 

1. The empirical evidence is not organized so as to answer the main questions: which theoretical principles have been refuted/upheld? Which interventions work the best? At the moment, in most cases, we have review articles or chapters that contain a list of empirical results, some in favor, some not in favor of a given theory or intervention, without any firm conclusion drawn. Sometimes, we have a review counting the number of significant results in each direction (an approach called vote counting). That is a step in the direction of providing a unique quantified answer but that is NOT correct (vote counting is a BIASED way of aggregating scientific evidence). We then have some meta-analysis, which are the correct way to aggregate statistical results, but they are plagued by three serious problems: i) publication bias: the extent of publication bias is such in the social sciences that the summary of the published knowledge might be extremely misleading. Sometimes it might not be, but the fact that we do not know whether that is the case or not is devastating. This is a disgrace and a major problem that we have to overcome collectively. ii) Researchers bias: each review, or ideally meta-analysis, is the product of a narrow team of researchers, not of a whole community. That means that we can have disagreement about the interpretations in the review article, and that the conclusions of the review article do not carry the full weight of the whole community. As a consequence, disagreement can persist without ever being solved and progress stalls. iii) Absence of continuous updating: the database behind a meta-analysis is most of the time kept private and generally lost. Updating it requires recollecting the whole data, an awfully costly and long process. With SKY, the process would be automatic and would have memory.
2. The empirical toolkit is not stabilised so as to determine which methods can answer which questions with which accuracy. We start to converge on a toolkit of methods for causal inference, but there still are a lot of debates and varieties of appreciation as to how to implement them, which tests are required and needed and why. This generates endless back and forth during the refereeing process, especially for people that are not close to the main centres of academic elite. Also, most of our evaluation of precision is based on the results of statistical tests, which is wrong and generates huge biases and triggers QRPs. Finally, there is no agreed-upon set of methodological questions to answer.

Second, there does not exist a place to build consensus on both individual papers and aggregate results. Interactions and discussions in academia, especially in the social sciences, use 20th century technology and have not at all taken advantage of the possibilities offered by the digital revolution. At the moment, interactions are extremely rare and are limited to seminars, conferences and, most of all, the refereeing process. We have inherited these interactions formats from the 20th century, an era in which most interactions were either in person or through mail. This is highly ineffective. Interactions are useful all along the life of a project, especially at its inception, and refereeing is needed even after a project is published, in order to catch mistakes. At the moment, interactions at the beginning of a project are limited to a handful of colleagues. Seamless interaction is all the more needed when writing overview articles that reflect the current views of the community on a question.

Third, the current tools have made us focus a lot of our efforts on publication instead of finding the truth. One especially big problem is that we measure productivity by the number of publications. More precisely:

• We are focused on publication, and forget almost everything that occurs after, especially aggregation of knowledge and consensus building.
• Our focus on publication is so strong that we compromise the exactitude of science by allowing for publication bias and Questionable Research Practices (QRPs).
• Our knowledge is behind paywalls.
• Knowledge is not aggregated, and when it is, it is not properly or only by part of the community.
• We study the same objects with different principles.

My coming up with SKY comes from being at the same time inspired by and frustrated with the current ongoing solutions that try to implement the vision or part of the vision:

1. Wikipedia proves that decentralized communities of interacting users can produce knowledge of high quality. But, one core rule of Wikipedia is No Original Research, which precludes Wikipedia from aggregating scientific knowledge, as long as a meta-analysis is considered original research. Also, Wikipedia has amazing tools for editing interactively an entry but does not have tools for interactively editing code and data.
2. The several initiatives that conduct systematic meta-analysis on social science questions (Cochrane, Campbell, Washington State Institute for Public Policy, NICE) are amazing but they have several limitations: they are not interactive, they are limited to a restricted team of researchers, meaning that they do not reflect the opinion of the community, they are sometimes paywalled, sometimes very hard to access, their databases and codes are not open, accessible and editable.
3. The open social science initiatives (PsySciAcc, OSF) are awesome, but their goal is to conduct new studies, not to summarize the results of a whole literature on a question. The results of their studies will be great entries on SKY but they do not provide an easy way to aggregate results, update them and collaboratively write a consensus piece. It will be fine to link to them on SKY for information on an individual study, at least in a first stage. Probably slightly more detailed pieces will eventually be extracted from these articles for their SKY page.
4. In economics, both J-PAL and IPA provide a first intent at trying to unify methodological practice and have generated some content close to what SKY strives for. For the moment, neither J-PAL nor IPA propose interactive tools to aggregate the knowledge of the community nor an open dataset of results nor meta-analysis.
5. Twitter is a great tool to interact with scientists, but it is not structured along research topics lines and does not provide tools for editing text, data and code.
6. Several researchers, myself included, have started to compile all the papers on one question on google spreadsheets. This is neat, but does not provide neither aggregation of the results nor an easily accessible unique place where one can find such datasets, nor a way to get to a consensus based on those datasets.

Why SKY is open source

Before describing how SKY works in practice, let me explain why I make SKY open source. The main reason is efficiency. SKY fulfills all five criteria put forward by Eric Raymond for the decentralized approach to project management to dominate the centralized approach:

1. Reliability/ stability/ scalability are critical. 
2. Correctness of design and implementation cannot readily be verified by means other than independent peer review. 
3. The software is critical to the user’s control of his/her business. Here, users are funders, policy-makers and scientists. I’m betting on the fact that SKY is fulfilling an important need for these users, and maintaining it will therefore appear critical to them.
4. The software establishes or enables a common computing and communications infrastructure. 
5. Key methods (or functional equivalents of them) are part of common engineering knowledge.

Countless examples in the software industry, from Linux to Apache, from Firefox to, of course, Wikipedia, prove that the decentralized open source approach works better than the closed approach when these criteria are fulfilled.  

There are two other more philosophical reasons for SKY to be open source. 

First, science by design cannot be closed. Science is the daughter of the Enlightenment, it thrives on rational discourse and exchange. Anyone who has something to contribute, acting in good faith, bringing data, theory, questions or interpretations, is welcome. Science hates arguments of authority, and thus I cannot see myself acting on them. Actually, my feeling is that we need to accept to engage with anyone, especially citizens, both the ones skeptical of our results, and the science-loving ones, we need to show that we have nothing to hide, that we can address all criticisms and explain our position, how we know what we know, and accept to reveal what we do not know, to anyone. Well, obviously, there’ll be rules of engagement that I’ll describe later, and anyone breaking these rules will not be welcome on SKY. We want rational data-based discussion, not endless ideological debates. We want people acting in good faith, not proselytizers. 

Second, I hope that being open to anyone will leverage the intrinsic motivations of contributors, their intrinsic love for science and their positive emotions at contributing to the accumulation of scientific knowledge rather than winning at the editorial lottery. There will be no badges to win on SKY other than contributing to the progressive accumulation of knowledge, no editorial positions to hold, no publication in major journals, nothing to win other than contributing to the collective scientific endeavor of understanding the world better to make it a better place. It’s not to say that there won’t be any rewards on SKY other than the  feeling of contributing to science. It seems obvious that, as it has happened with Wikipedia and Linux, contributors will earn social rewards from the respect and recognition of their peers. But my belief is that, on SKY, these rewards will be of a different nature, and will put value on different types of contributions: useful pieces of public software, useful new methods, a replication that makes an estimated effect more precise, will all be valued because they will make the work of everyone in the community better. People good at organizing teams of researchers around questions will also be valued. But the aggregative nature of SKY, the fact that the main entry that everyone will be concerned about is the consensus article, a piece that will aggregate the results of a lot of researchers, and the interpretations of even more, will tone down the incentive for individualistic rewards such as publishing in big journals or having editorial positions where you act as a gatekeeper. The community will be the gatekeeper, users of science will be the gatekeepers. By doing so, SKY, I hope, will realign the incentives of scientists towards doing good, sometimes boring, but truly life-changing science and away from doing publishable, eye-catching, shiny but eventually shaky and biased science.

Which technology makes SKY possible

Let me now describe how SKY works in practice. The current version of SKY is built around three main tools. There is a MySQL database storing the results of individual studies. These results are downloaded and analysed in R and transformed into a web page with graphs, tables and text using Markdown syntax. Results of the meta-analysis are also stored in the MySQL database for later use. The whole code generating SKY is publicly accessible on a GitHub repository. SKY is currently hosted on GitHub Pages, that is each time a modification to the code is pushed to the GitHub repo, SKY reflects it after a few minutes. 

If you are not familiar with the tools that enable the existence of SKY, here is a quick description (they are all free and open source):

1. MySQL is a database management tool. It stores data on a repository in a clean and normalised way. Moreover, MySQL enables access to and modification of the data by a lot of users simultaneously. With MySQL, no need to make endless copies of the data. You just have one copy on a shared repository. 
2. Git is a collaborative writing tool that is used mostly by coders. It enables to track changes made in a text, to merge two separately updated versions of a document, to identify the potential conflicts and to solve them. Git is basically MSWord “track changes” for nerds. With Git, there is a centrally maintained version of the code, in general on a web-based repository such as GitHub, that users can download and modify and then push changes to.   
3. Rmarkdown is an editor that combines R code for statistical analysis with the Markdown syntax for text editing to produce neatly edited documents such as pdf, word or html files (the format used on the web).

As an intermediary stage, SKY also uses Google doc spreadsheets to upload information about estimates for which there is no entry as an individual research report. Hopefully, as time passes, the number of such entries will decrease.

How to use and contribute to SKY

There are different ways to use SKY. Obviously, consulting the results of meta-analysis on questions that interest you is a first. A second is to link to meta-analysis or related studies or a google doc listing them on SKY, so that you never lose the information and you share it and run the chance of having someone update it and enrich it. Finally, you can also use SKY to pre-register an idea and receive feedback on it.  

You can start contributing to SKY right now. The easiest way to contribute is to make me aware of a meta-analysis on a topic. You can for example cc me when you tweet about one, or when you find one on tweeter. You can also drop me an email. As soon as I'll receive the link, I'll upload it on SKY.

If you download the whole project from the GitHub repo and start tinkering with it, you can create the link yourself. Once you have made modifications, save them to git and push them to GitHub. At the moment, I have to accept your additions for them to appear online. That will evolve over time. Either by adding more master users with permissions to accept changes or by giving master rights to everyone. If you need to access the SQL database, you’ll need an Id and a password. Email me and I’ll send them to you and you’ll be all set. 

When contributing, please describe as clearly as possible what you have done in the commit message, so that everyone can easily identify your changes and what they are doing.

I might ask you to open a new branch when your changes are important, so that we can see what the changes look like on our computers before putting it online.

I've written a succinct document describing how to get started with the tools on SKY. I have to update it for using Rmarkdown (it describes how to use knitr for the moment, but sources for Rmarkdown are easy to find).

What can you contribute to SKY

Meta-analysis

The easiest way to contribute, if you have done, or have knowledge of, a meta-analysis, is to make me aware of a meta-analysis on a topic. You can for example cc me when you tweet about one, or when you find one on tweeter. You can also drop me an email. As soon as I'll receive the link, I'll upload it on SKY.

If you have access to the data behind the meta-analysis, it would be great if you could create a google spreadsheet containing the data and create an article summarising your interpretation of the data. Again, if you do not have time or do not have the knowledge on how to do it, I can do it for yo gif you send me the dataset.

If you’re interested in a question but have not done a meta-analysis, please start a google doc with the few estimates you know. We will improve on the coverage over time with addition by others, especially by original authors who want their papers to be included in SKY. You can already see how SKY changes the way we conduct a meta-analysis: we can start small and aim for exhaustivity only in the limit, with the combined efforts of a lot of people. 

Try as much as possible to add an individual research report for each entry in your dataset, linking to the initial study and explaining how you went from the figures in the study to the entry in the dataset.

If you do not have time to do all of that, just start with something. Make a guess. Explain how you made it. Other people or you when you’ll have more time will come back to this and improve. 

We need to agree on aggregation principles, especially the estimates. I propose to aggregate using Odds Ratios (OR), Effect Sizes (ES), elasticities or ratio of ES and Cost-Effectiveness or Benefit/Cost ratios, depending on what’s appropriate for a particular application. For interventions, Cost-Effectiveness ratios should be the ultimate goal, in that they enable a clear comparison between interventions aiming at the same goal. Other quantities of interest should be computed as much as possible for each estimate.

Individual research reports

You can start a new research project on SKY. That will act as preregistration, and will be time stamped on the Git server when you commit your changes. 

You can link to an ongoing research project on another website such as the AEA registry,  OSF or PsySciAcc or to a registered report sent to or accepted by a journal.

Do not hesitate to create a page for the general question your study is trying to answer and to put a few word of motivation there.

If you’re aware of similar studies, mention them, even if only in passing. Better, provide a link to the project/paper. Better even, let the authors know that if they want, they can put their estimates on SKY and review and improve your entry. 

Methodological Research

You can also contribute to developing methodological guidelines. Anyone can, not only people doing methodological research.

If you’re doing methodological research, do not hesitate to create new research reports and synthetic entries. 

Finally, you can update and revise or even contribute directly to the Statistical Tools for Causal Inference Book. Its GitHub repo is here.

What are the rules of the game on SKY

Be bold 

Do not hesitate to create new questions and to feed them the little information that you have. If it is nothing, then that’s fine. Just ask the question.

Do not strive for perfection at first

A stub, a line, a link, a back of the enveloppe estimate are better than nothing. Others or yourself later will complete your work.

Act in god faith,

When interpreting evidence and discussing results,  be open to arguments. 

Assume good faith

Be civil, be tolerant: errors and mistakes happen. They most of the time come in good faith.

Use data to settle disputes

If various interpretations of the same data can be held, mention them and mention which tests might be able to settle between them. When discussing methods, use theory, simulations or performance on real data to come to an agreement. 

Do not proselytise

If you’re coming on SKY only to make a point, whatever the evidence, you’ll be frowned upon. Accept that some people are not convinced by the same evidence than you are, try to understand why and try to find a way to answer them or at least delineate the possible concurrent interpretations and how we could go about deciding between them.

Do not violate copyright 

When mentioning a published paper, always provide links to the published version. It's fair to also provide a link to a freely accessible pre-print. If you quote from the paper, use quotation marks or the quotation environment. Refrain from extracting a figure or a table from a paper. Simply provide the link to it on the publisher webpage. I'm in discussion with publishers in order to understand the rules of the game. 

How SKY is funded

SKY is open source, free and free of advertisement, and not for profit. SKY benefits from the work of researchers whose wages are already being paid by their institutions. In the future, SKY will be organised as not for profit organisation able to receive donations, crowdfunding and subsidies for funding servers. For the moment, SKY is only funded by me, for the cost of maintaining an online MYSQL server.

SKY is supported by my employers, TSE and Inra.

What’s in the books for SKY

The major evolution that I foresee for SKY in the medium run is to move to a full cloud-based solution. This will enable several key improvements:

1. Cloud-based computing avoids having to download and run the code code for the whole website on the computer of each user. At the moment, the only cloud-based parts in SKY are the SQL (and Google spreadsheets) database and the GitHub repository for the code. In order to make modifications, the user has to download the full SKY project, which might at some point become a mouthful. It is not at the moment, since the website is small but if and when SKY grows, this might come as an issue. There are ways around that issue that do not require moving to full cloud-based computing (such as using RMarkdown “cache” option in order to not having to run expensive pieces of code as long as their result does not change, and even putting the cached databases on GitHub), but they are dominated by a full cloud-based solution.
2. Cloud-based computing would probably be more welcoming to users of statistical software that is not R and to users not familiar with git. 

The solution that I am exploring at the moment is to use CodeOcean amazing infrastructure. I’m going to talk with them and see if we can work something out.

Another important evolution is to add to SKY a discussion chamber, much like the “discussion” page in a Wikipedia project. Such a chamber is crucial to prevent cycles of edits by editors that share opposite views and to fluidify discussions and sharing of work. 

Another important addition to SKY is a refereeing suggestion and voting infrastructure, that, much like answers on Stackoverflow, will enable you to suggest edits and modifications to a project and to vote for already suggested modifications. 

Another important component of SKY will be an anonymous reporting infrastructure where behaviour deemed unethical or violating the rules of SKY can be reported. Individuals appearing multiple times there for different interactions will be warned by SKY representatives that their behaviour should change if they want to remain on the platform. Public shaming and exclusion from the platform are possible sanctions for reiterated behaviour despite warnings.

Finally, offering the option of maintaining private repos is also in the books.

OK, now, let's roll :)