However, the earlier (and erroneous) finding that conditions did not improve schooling outcomes was news enough that it stuck.Many people, including good researchers, colleagues at the Bank, bloggers, policymakers, think that UCTs are as effective as CCTs in reducing dropout rates – at least in Malawi.In research, as in life, first impressions matter a lot.
However, the earlier (and erroneous) finding that conditions did not improve schooling outcomes was news enough that it stuck.
After one more revise and resubmit, the paper is now forthcoming, and the final version (more or less) can be found . Our findings in the March 2010 version suggested that CCTs that had regular school attendance as a requirement to receive cash transfers did NOT improve school enrollment over and above cash transfers with no strings attached. The difference was NOT that we had longer-term data: if we use self-reported enrollment to examine one-year or two-year impacts, the results are the same (see Table III, panel A in the paper linked above).
Rather, the difference was caused by the kind of data that we were using: we supplemented self-reports with administrative data, enrollment data collected from schools, monthly attendance ledgers, and independent achievement tests in math and languages.
The reviews, which were fast (as good as it gets at about a month), suggested that we should not only report longer-term data but also use alternative measures of schooling – less subject to reporting bias.
We followed this advice and , which now presented two-year impacts using enrollment and attendance data collected from schools, in addition to independent achievement tests, in December 2010 and resubmitted it to the same journal, again simultaneously.
See the recent comments on Blattman’s blog, which make it look far from clear that giving up on double-blind peer-review is a good idea).
But, do the benefits of making these findings public before peer-review outweigh the costs? In economics, publication lags, even for journals that are fast, can be long: it is not uncommon to see articles that state: Submitted December 2007; accepted August 2010.It gets downloaded many times, gets blogged about, etc.Then, a year later a new version comes out (maybe it is even the published version). Most of them had only read the abstract (and maybe the concluding section) of the first draft working paper to begin with.Many iterations of papers simply improve on the original premise, provide more robustness checks, etc.. Worse, they had just relied on their favorite blogger to summarize it for them. Their favorite blogger has moved on and won’t be re-blogging on the new version of the working paper.But, interpretations often change; results get qualified; important heterogeneity of impacts is reported. Many won’t even know that there is a more recent version.But, that assumes that the findings are ready for public consumption at this preliminary stage.By preliminary, I mean papers that have not yet been seriously reviewed by anyone familiar with the methods and the specific topic.In March 2010, we put out a working paper on the role of conditionalities in cash transfer programs, which we also simultaneously submitted to a journal.The paper was reporting one-year effects of an intervention using self-reported data on school participation.Findings, and particularly interpretations, change between the working paper phase and the published version of a paper: if they didn’t, then we would not need peer-reviewed journals. (BTW, the promise that the blogosphere would serve as the great source where we get many good comments on our working papers simply has not come through.Useful comments require time and careful reading, which is not how stuff online is consumed.) Now, back to the point about first impressions: When a new working paper comes out, especially one that might be awaited (like the first randomized experiment on microfinance), people rush to read it (or, rather, skim it).