How do you test whether a stock screen's backtest results hold up out of sample?

A backtest that looks strong on the period it was built on can still be a product of chance or of the choices made while building it. Many screens are tuned on the same data they are then scored against, which inflates the apparent edge. The first thing I'd want to see is a clean split between data used to design the screen and data held back until the end. Beyond a single holdout period, the question is how the result behaves across regimes. A value or momentum screen can look excellent in one stretch of market history and do nothing in another, so the spread of results across different periods often says more than the headline number. Sample size matters too: a few dozen signals rarely tell you much about a rule that trades only occasionally. The skeptic's version of this is that even a careful out-of-sample test can be fooled if the screen was one of many variants tried, since the best-looking one will win by luck alone. So the honest question is how many rules were tested before this one was picked, and whether that count was recorded. When you have evaluated a screen, what out-of-sample check did you find most useful for catching a result that was only noise?