The silent killer of backtests

Imagine: you build a system that in July 2024 should have said "buy NVIDIA". You test it on historical data. The backtest says: this strategy delivered 30 percent return. Shiny result. Time to celebrate.

Until someone asks: in July 2024, were you only using data that was actually available at that moment?

That's a treacherous question. And the answer for nine out of ten self-built systems is: no.

Walmart as a warning

Walmart closes its fiscal year on January 31. Not on December 31, like most companies. That's called a non-calendar fiscal year. And hundreds of companies have one. Costco. Apple. Microsoft. Almost everything in retail.

So Walmart closes Q4 2024 on January 31, 2024. But Walmart doesn't publish those numbers until February 19. Almost three weeks later.

If I take a snapshot in my backtest on February 1, 2024 and use Walmart's Q4 numbers, I'm using data that nobody knew about at that moment. My system was playing poker with someone else's cards face up.

Without me realizing it.

This is called look-ahead bias

A naive backtest grabs all data that exists now, and runs it through your strategy. Problem: "all data that exists now" includes things that didn't exist in the past.

Earnings published weeks after the quarter ended. Analyst targets that were adjusted afterwards. Restatements of earlier numbers where companies corrected something later.

If your strategy depends on those fundamentals, and you're not careful about when each number became available, you're sneaking a glance at the finish-line photo before the race starts.

The impact: backtest returns can come out 5 to 10 percent too high. For a system that actually delivers 12 percent return, a naive backtest can suggest 20 percent. The difference determines whether you think you have an edge or whether it's not worth your time.

How I handle it

My system uses factors that are partly based on fundamentals. Value, meaning "is this stock cheap or expensive". Quality, meaning "is this a good company or a mediocre company". Both lean on earnings data. EPS, ROIC, revenue growth. Those numbers aren't published in real time. They come in 30 to 60 days after the end of a quarter.

My fix: a 60-day buffer. For a snapshot on February 1, 2024 I only use earnings that became public no later than December 1, 2023. Better to be too conservative than too optimistic.

Sounds simple. But in a backtest with thousands of stocks and years of historical data, this is the difference between a system that thinks it works and a system that also works in production. Plenty of self-built quant systems fail on this one check.

How I noticed it

In my first version I had no buffer. All fundamentals data simply included. Results looked good.

Too good.

Sharpe ratio on the high side. Drawdowns small. The kind of numbers that would make an outsider ask whether I wasn't mistaken. I thought at first: no, this is just a good system.

Until I asked myself: in July 2024, did I have everyone's Q2 earnings at that moment? And the answer, after five minutes of looking, was no. Some Q2 reports didn't come until September. If I include those in my July snapshot, I'm using information that didn't exist then.

Buffer added. Backtest re-ran. Sharpe a lot lower. More realistic. More honest.

The difference sat in one number. 60. Days buffer.

Why this matters

Anyone building their own trading system will stumble upon look-ahead bias eventually. Because it always works in your favor. Backtests look better than they should. You feel smart. You keep building.

Until you go live. And reality catches up.

For a real-money system this isn't an academic concern. The difference between "I thought I had an edge" and "I had a data illusion" is the same bill. A few months of underperforming. And then you have to be honest about what that means.

The win from bias correction isn't in prettier numbers. The win is in the numbers being accurate.

And silent mistakes are the most expensive mistakes. Especially when there's real money in the game.