I used the Open Source Asset Pricing data to investigate published t-stats in finance. I replicated the t-stats of 176 published trading signals and compared them to the papers' published claims. The results showed the original t-stats had only a 0.75 correlation with the replications, and 64.8% of the t-stats I replicated came out worse than the originally stated t-stat. Following this, I found substantial median post-publication decay, with the median retention ratio being 0.458 of mean returns and an interquartile range of 0.37–0.88. Decay was far from uniform: 41 of 205 signals were stronger after publication than before. These findings are consistent with McLean and Pontiff (2016) on post-publication decay and Hou, Xue and Zhang (2020) on replication failure.
I first downloaded and imported the Open Source Asset Pricing signal documentation to establish the original papers' t-stats. I then imported each signal's portfolio returns, focusing on the long-short portfolio returns. Using these, I ran every signal's portfolio through a function to compute a replicated t-stat. I also calculated post-sample but pre-publication statistics, to see whether signals held up out of sample before being affected by public interest, and post-publication statistics to investigate whether publishing erodes alpha.
Some limitations of this study: firstly, I did not factor in transaction costs for trading these signals, which would have lowered returns and t-stats as a result. Secondly, the post-sample but pre-publication window was largely underpowered in the context of reaching statistical significance for most of these signals. This would have been the true test, given a bigger window, of how signals perform out of sample without the effect of increased attention from publication. Thirdly, sample windows were defined from January of the start year to December of the end year, rather than using the exact months, for comparability across signals. Additionally, the signal documentation lists 331 signals, but only provides portfolios for 212, so 119 were excluded. Finally, VarCF replicated with the opposite sign to its published result; the cause was not established and the signal was excluded.
I found original papers overstated t-stats when compared to replicated ones, and substantial median decay, with wide variation across signals.
Data is not included in this repository. It can be downloaded from the Open Source Asset Pricing project. The notebook expects the portfolio CSVs in a folder named Original_Cuts; you will need to set the path to match your own machine.