Why Backtest?

Picture of Heather Cullen

Heather Cullen

Author
In The Money

Heather Cullen Blog Wall ST Crash

Why Backtest?

Special Update 31 October

(At Bottom of Page)

I’ve been getting a lot of emails and comments about backtesting (thank you), and one of them set me back on my heels (thanks Michael! 😄). It was simply:

I’m not sure backtesting is “the” key. The market is not going to perform exactly the same one time to the next . . backtesting can be somewhat arbitrary.

Well, I thought ‘why DO I backtest and is it arbitrary? Have I been influenced too much by the quote often attributed to Mark Twain?

History doesn’t repeat itself, but it often rhymes.

The only information that we have is what has happened in the past, so we need to understand it and extrapolate. However, I read some articles critical of backtesting, and found that no-one was criticizing backtesting itself, but many were very critical of how it was implemented. So, I decided to check my backtesting strategies against the 5 main criticisms:

Heather Cullen Blog Mark Twain

Data overfitting. This could come from a limited data set, so I have extended my backtesting to daily data going back 100 years. That’s 100,000+ data points (OHLCV) a decent sample size.

Survivorship bias. Operating on data from entities that survived and ignoring the ones that didn’t. If we use the market index, in this case the $&P 500, then we are not eliminating anything.

Changing market conditions. Yes, but ‘twas ever thus’.

Assumptions and simplifications. Agreed, too many backtests are guilty of this, hence my emphasis on objectivity. Let the data speak for itself.

Lack of emotional factors. There is a criticism that backtesting doesn’t account for the emotional and psychological aspects of trading. Yes, I totally get that, but I don’t think that emotion has any place in trading, which is why I only use a completely objective system, which does not depend on how I ‘feel’ about anything or what I ‘think’ it may do.

I couldn’t resist repeating the silliest quote about backtesting  (Investopedia):

Be sure to paper trade a system that has been successfully backtested before going live to be sure the strategy still applies in practice.

Using Different SMAs

Many of the comments suggest we use a different SMA cross for getting in and out. For example, use the 10/200 to get in and the 5/50 to get out. That would get us out quicker in a downturn. It sounds nice and easy, doesn’t it? Well, it is – if you are going to make only one trade! Let’s say you have just seen a 5/50 death cross, and you are out of the market. The question is:   

When do you get back in again?

On the 5/50 golden cross? In that case you are trading solely the 5/50 crosses, and on backtesting we find that it performs much worse, whipping us in and out of trades of which 62% are losing trades.

The next 10/200 golden cross? You may be waiting a long time and miss out on a bull market, Here’s what would have happened in 2020 – 2021. 

Heather Cullen Blog SPX 2020

The universe of possibilities

In probability theory, we have to consider all possible outcomes, so when we are considering the possible arrangements of our 4 SMAs there are 8 possible outcomes:

Heather Cullen Blog Backtesting

If we decide to choose a golden cross from the above, there are 6 possibilities:

Heather Cullen Blog SMA Crosses

Let’s say we choose the SMA5 / SMA50 Golden cross. That is found in arrangements 7 and 8 above, so we have to program for both these possibilities. Your eyes are probably glazing over about now. Mine certainly are. I have gone on long enough about backtesting, but your takeaway should be that entering and exiting on different SMA crosses is not as simple as it seems.

Backtesting results

Working on 100 years of data for SPX (it was suggested by Sherman that I needed to extend my backtesting to include the 1987 crash – he was right, so I have, right back to before the 1929 crash), I compared the standard ITM method using the 10/200 cross, to the 10/50 cross. Here are the results:

While the % gains / losses are not markedly different, the average gain on most transactions for the 10/200 strategy is almost double. That, along with the number of trades, makes all the difference on your account balance.

Heather Cullen Blog ITM performance

Here I compare the results on 2 accounts, both starting with $100 in 1928. The black is 10/50, the red the 10/200 ITM.

You can see that using the 10/50 cross we didn’t even match the market, with the leveraged account doing even worse. (The leverage was 100% – a strike at 50% of current price and a time value of 1% lost on every transaction)

The 10/200 cross did extremely well – with the unleveraged account beating the market, and the leveraged account beating it amazingly. I would display the whole backtest, but it is 1,642 pages, so I will have to think how I can do that!

Where to now?

I am still testing lots of different strategies, no doubt there is a fantastic one waiting to be discovered. I’ll keep you updated.

ITMeter

To the markets:

It’s not good news – the SMAs are touching and if they crossover then we will have our OUT signal. That may happen on Monday, and if it does, I will send out an alert email.

SPY Chart

Heather Cullen Blog SPY Chart

You can see that we have dropped right through support and the next support level would be around $380 – which is only just above bear territory.  Right now, we are officially in a correction phase, which is defined as a drop of 10%. Do all corrections end up as bear markets? No. But all bear markets start with a correction.

SPYG Chart

The SPYG chart is not quite so dire as SPY, mainly because of the relatively heavy weighting of technology stocks. While the 10 SMA has taken a downturn it is still some way from a death cross with the 200 SMA.

It has, however, dropped through support which is not a good sign. Since its high in July, it has dropped 7.6% so not quite in correction territory.

QQQ Chart

Heather Cullen Blog QQQ Chart

After 5 months in consolidation, QQQ has broken down and dropped through the bottom of the trading channel. Not good.

It has dropped 10.5% since its high in July, and so is also in correction territory. The next support level is around $320.

VIX Chart

Heather Cullen Blog VIX chart

As you would expect, the VIX is elevated, and is now over 20 so we are no longer in a low-volatility market.

DOW Comparison

I thought it might be interesting to compare the performance of the 3 most-watched indices over the last 12 months: Here it is:

Interestingly, the DOW has been flat for most of the time, SPX climbed strongly in March – August, and QQQ very strongly between January and August. Since August, all three have been heading down. Dismal.

The week ahead . .

At time of writing (12 hours before market open) the futures are up . . could it be the turn around? I hope so. It is surprising how often the 10 and 200 touch but do not actually cross. ITM will stay in the market until we see a clear cross, which will be Monday if it is a big down day. I will send out an email if that happens.

Heather Cullen Blog Futures

But . . let’s not anticipate the worst – what’s that quote: sufficient unto the day is the evil thereof? No point making ourselves miserable until it actually happens!

Gadding about again . .

And next week – well, I am on holiday again. You may have heard of the Melbourne Cup, the ‘race that halts a nation’ – well race week is coming up, and I’m going over there, and going to do the whole big-hat-and-dress-up thing, just for fun. So, no blog next week – I just can’t bear trying to do the charts & photoshopping on a laptop, it is too fiddly, but I will be back again in a fortnight. I will still check the comments daily and answer any questions.

I’m not going to say fingers crossed for a good week – it doesn’t seem to be doing any good!!

Heather

Inspirational!

I’ve had an email from Sheryl which I would like to share part of:

I’m a newbie ,  starting with SPYG. . . . . . I’m 87.

Fantastic Sheryl! You’re never too old to learn new things!

31st October Special Update

I have just sent out an email update to everyone, but in case you didn’t get it here is what it said:

 

31st October Update

SPY Death Cross

Monday’s trading, although it was an up day, was not enough to prevent a death cross. The SMAs have clearly crossed so that is our signal to get out of the market. If you are following the ITM strategy it is time to get back on the sidelines while the market decides what it is going to do.

SPYG has not yet had a death cross, so it has not had an OUT signal. Remember, it was backtested separately and does not always follow exactly the same pattern as SPY. QQQ, although we are not trading it, has not had a death cross either.  

It has been a dismal two years. It’s bad news, I know – we were all looking forward to a nice bull market with lots of lovely profits, but the market just isn’t behaving. We’ll get back in when it does.

Heather Cullen Blog Bull market

Questions & Comments

As always, happy to answer questions and receive comments (just be kind!)

27 Responses

  1. Hi there,

    I suggest that you check out Perry Kaufman’s books on Systematic Trading. Walk forward Optimization is more complex than separating your dataset into two parts.
    Regardless, nothing is infallible and there is certainly an element of luck. I would not dismiss Mathematics so quickly. Wall Street is one of the biggest employers of Theoretical Physicists. Why do you suppose that is? Here is the rub. They are not trading moving averages or Bollinger Bands. Simply stated they are looking for patterns that repeat and are always adjusting to market conditions. You have an IT background so I am sure you can relate to my Engineering education. I have certainly used moving averages both simple and exponential as well as other TA tools. Thing is, these averages are simple representations of more more complex ideas such as a FIR and IIR filters. For example, exponential MA’s were developed to track missiles before there was a computer on everyone’s desk. Bollinger Bands assume that prices are normally distributed which they certainly are not. Not to say that TA tools don’t have there place but when I hear phrases such as the Price respected the 12 day ema I have to laugh. Price may seem to respect the 50 day because so many folks watch it but it fails under scrutiny. Folks need to remember there is a difference between trading and investing. I will end this diatribe with one last thought. Risk Management is far more important than the system used.

  2. Hi Heather,

    Thank you for your email on October 31,2023 asking us to get out of the stock market due to the death cross of SPY (10sma crosses below 200sma). You deserve two weeks of vacation and see you back again in the middle of November.

    Sincerely,
    George Halongton in Los Angeles

    1. Hey Andy
      well, its always the way, isn’t it?
      But the death cross is still confirmed – even though the last 2 days have been up days, they have not been up enough to affect the trajectory of the 10 SMA.
      But who knows? If it keeps going up then we will get a golden cross and get back in again. There’s no law against buying back in!
      We know that ITM wins overall, even if sometimes it is hard to stick to it.
      h

  3. Hi, I read your book In the Money Bull Market Strategy and think its fantastic. I was wondering, where did you get all the SPY backtesting data from? Many providers don’t offer data this far back. Would appreciate any help. Sandeep

    1. HI Sandeep
      Yahoo.com has data going back to 1927 – just google ‘Yahoo Finance SPX Historical data.’
      The download is a bit flaky – sometimes it cuts you off at 1987, sometimes earlier, but if you keep trying you can get it.
      Hope this helps.
      h

    2. HI Sandeep
      Yahoo.com has data going back to 1927 – just google ‘Yahoo Finance SPX Historical data.’
      The download is a bit flaky – sometimes it cuts you off at 1987, sometimes earlier, but if you keep trying you can get it.
      Hope this helps.
      h

  4. Hi Heather, Recently, you posted some information in response to a question about spreads. I have posted it below. After reading it, I am still pretty confused about a good entry price on a DITM option and I sure don’t want to pay the ask price. Some of the DITM spreads have as much as 4 to 5 dollar difference between bid/ask price. I don’t trade options much, but when I have, I stay away from trading options with large spreads. With regard to the SPY with a strike price with a large spread, is the goal to try enter about 1/2 way between bid/ask price? I think you kind of cover this in your book. If buying a call (or put) option, sometimes I will put in a limit order at or below the bid price and see if it moves my way and fills me and then adjust as necessary depending on how badly I want to be filed. But maybe that is an invalid strategy for DITM options. Love your blog and how interactive you have made it. Heather Cullen October 24, 2023 at 2:54 AM I have never found that using spreads on DITM SPY has been profitable, mainly due to the low time value and tight spreads. It all depends on what premium you can get. For example, if you look a 1 year to expiry $200 strike the bid / ask is 225.96 / 229.71, and the $210 strike is $216.61 / 220.35. If you buy at the ask and sell at the bid this will cost you $13.10 ($1,310) which means you are always making a loss of $310, no matter where SPY is because you have bought a spread of $10 for $13.10. If you can manage to get a spread of $10 for less than $10 you will make a profit. If you were able to buy at the mid point of both bid / asks then it will cost $9.40 and you will make a profit of $60 if SPY is above $209.40 at expiry. For example, if you look a 1 month to expiry $210 strike (there is no $200 strike) the bid / ask is 211.74 / 212.59, and the $220 strike is $201.79 / 202.63. If you buy at the ask and sell at the bid this will cost you $10.80 ($1,080) which means you are always making a loss of $80, no matter where SPY is because you have bought a spread of $10 for $10.80.
    Dale

    1. Hi Dale
      I’ve just looked at the SPY chain for March 2024, and the 200 strike has a bid / ask 219.56 / 221.24 which is a spread of $1.68 or 0.4% of the current price. The 250 strike is 170.94 / 172.46 which is a spread of $1.52, or 0.36% of the current price. Both of these seem reasonable – except that we are not looking at buying at the moment (death cross).
      I think what we may be getting mixed up on is the spread meaning the difference between the Bid and Ask, and spread meaning the options strategy of buying a debit spread or selling a credit spread, which means that you buy 2 options of different strikes.
      Re where you put your buy in – mostly I put it in at the ask price, and most of the time I am filled below it, usually around halfway. There’s this thing called NBBO (National Best Bid and Offer) which is too long to go into here, but you may want to google it. If I have decided to get into a trade, I get in – every time I wait for a better price it seems to go against me and I wish I had moved earlier. But maybe that’s just me!
      Hope this answers your questions, if not pls get back to me.
      h

  5. Wow Heather you are absolutely amazing. I just found your website. I just ordered 4 of your books on Amazon. 2 physical and 2 eBooks. I will leave you a wonderful review. This is best information and written in plain English so that it is easily understood. Thank you so much for all that you are sharing here.
    Much Respect to you.
    Trisha B.

    1. Hey Trisha! Thank you for your kind words – its is things like this that make me keep going!
      And I would definitely appreciate any reviews you can do – that would be wonderful!
      Thank you
      h

  6. Hi Heather,

    Thank you so much for backtesting all these different strategies. I truly appreciate it and was looking forward to seeing your results. Please let us know if you discover a different strategy that is fantastic!! Thanks again for all you do for us!!

    1. Hey Charitey,
      nice to hear from you. Yes, I’m busy trying lots of combinations, and using ITM as the standat=rd to measure them against. So far I haven’t found anything, but I am hopeful and will keep trying.
      I will certainly let everyone know when I do.
      Thank you for your kind words!
      h

  7. Hi Heather,

    Thank you for another wonderful weekly blog. Regarding your data of performance of 100 years below, there is something I find it confusing. If I just left $100 in the S&P 500 Index 100 years ago, I would have this much already. Why do I need to do all the works just to get to $592k instead of $1mil.

    1920: $100
    1945: $1k
    1970: $10k
    1995: $100k
    2020: $1mil

    Based on 10% annual return of S&P500 Index from 1920 to 2020. Increase 10 times every 25 years.

    Sincerely,
    George Halongton in Los Angeles

    Account Start / End 100 years
    Unleveraged
    Start
    $100
    End
    $8.189
    Leveraged
    Start
    $100
    End
    $6.133
    MARKET $23.985
    $100
    $30,354
    100%
    $100
    $592.158
    $23.985

    1. Hi George
      The 10% per year is not like interest – it doesn’t happen every year, so your account balance doesn’t monotonically increase. You can verify this with excel – start with $100 and do one column where there is a 10% increase every year, and another where the average increase is 10% ( say 2%,4%,4%,30%) and you will see the difference. How I was estimating the market and the test strategy values was simply to take the actual price of the index not the ETF, so there were no issues with splits or dividends.
      So in 1927 SPX was $17.66 so with 100 we would have bought 5.662514 shares. (yes, I know that we can’t buy a fraction of a share, but the $100 was just a starting point so that the final result didn’t have too many zeros.)
      Today, those shares would be worth 5.66 * $4,235.79 = $23,945.57 (not exact, rounding error). As I was using that method on both the control and the strategy they are comparable.
      Definitely if SPY had been around in those days then you would also have had dividends and could have reinvested them, but in the interests of keeping things simple I eliminated them for both the control and the strategy.
      Hope this helps.
      h

  8. Very Interesting Post! Regrettably I cannot duplicate your unleveraged results as Trading Views data begins the day after the crash. As an interesting aside I tested the old model of a 10 month moving average rather than a 10X200. Unleveraged. It won by a landslide. Clearly the dataset is reduced dramatically but it is also low maintenance. Both systems have large drawdowns which quite honestly I believe would have shook at the average investor. I love to tinker and I see you do as well. However, positive expectancy systems do stop working and a simple backtest does not really prove all that much. That is why Monte Carlo Analysis coupled with Walk Forward Optimization is the way to go. I readily admit, I have not found a system that I could trade and believe that I was not “fooled by randomness”.

    Looking forward to your next post!

    1. You can get the data going back to 1927 on Yahoo.com. It is temperamental, however. Some days it lets you download and some days it defaults to a more recent dat. Luckliy one of the days I tried was a good day, so I got the whole dataset – although it wasn’t entirely ‘clean’, The opens between 1969 and 1764 (from memory) were missing, so I had to subsititute the close of the previous day. I figured that it wouldn’t change the final results materially as it only involved a few trades.
      Re the 10 month MA – just put it on my list of strategies to test! I can imagine that the drawdowns are huge in the GFC and the 29 crash?
      And Monte Carlo Analysis? I blush tp say I have not heard of it. Will google it and educate myself!
      Thanks!
      h

  9. Have a good trip. This can wait until you get back. I just wanted to say to those who criticize back testing the following: So what. We certainly can’t forward test. What else are we going to use. The problem with moving averages is that they are lagging and no amount of fiddling with the numbers can fix that including Hull MA’s, in my humble opinion. So my solution while imperfect has helped me. Use Bollinger Bands off of your moving average of whatever length works for you, (20,30,40 50,100 or whatever) and find one that works for you to augment any other non-MA signal you may have. If not BB’s then try Price or Donchian channels on a daily, weekly, or monthly basis on a Chart. find what works for you. Finally I will add a Comment from Barbara Dixon,
    “I’m not a mathematician. I believe that the simplest solution is the most elegant and the best. Nobody has ever been able to demonstrate to me that a complex mathematical equation can answer the question , ‘Is the market moving in an uptrend, downtrend or sideways?’ any better than looking at a price chart and having simple rules to define those three sets of circumstances. These are the same rules I used back in the late 70’s.” – Barbara Dixon

    1. Hey Tommy – thanks for that. I will definitely be lookin at Bollinger Bands – I have tended to dismiss them as they are lagging and also unbounded, but I haven’t through of thrying it off the moving averages. A good idea. Will add it to my list! And will investigate Price and Donchian channels.
      Loved the quote from Barbara Dixon – and absolutely agree with her! Must read more about her.
      Thank you!
      h

      1. Appreciate your comment and response. Unfortunately there is not much in print on Barbara Dixon. All I know really about her except some hearsay is that she was the Secretary for Richard Donchian, known as the Father of Trend Following. She had to take a Secretarial job because of the prejudice at the time on Wall Street toward women. Poor Richard suffered something like Dementia and had to be forced to stop trading and she went out on her own managing money and Trading.

  10. Have a great trip. This week’s blog was timely, neeeded, pertinent and entirely convincing. Bravo!!

  11. The only thing I might add to your back-testing methodology is a) back-test up to a certain point in the past and then do walk-forward testing, and b) back-test a basket of similar instruments (i.e. ETF indexes) – they might even out (or exacerbate) the drawdown and equity curve. In theory, they should perform similarly to the SPY – if not, why not? Sometimes, even a losing instrument can even out the equity curve (e.g. TLT).

    I’m amazed at the 6-month testing many people do on an option system (using popular testing software) using one ETF and then trade based on those, literally unbelievable, results. This, in my opinion, gives back-testing a bad name.

    Kudos to you for the effort you put into constant improvement and your 100% transparency – it is much appreciated.

    1. Hi Ronny, I understand the concept of doing a back test then a walk-forward test – but I struggle to work out how it improves on the backtest.
      It seems to me that what you do is divide your data into 2 data sets and use the older one to backtest, and then use the strategy on the more recent one to see if it works. Isn’t that just an extension of the backtest? Maybe I am missing something here.
      Using more than one instrument so that it evens out dips in SPY – yes, I have considered it, had a look, but nothing has jumped out at me so far. I need to investigate a bit further.
      More backtesting – lovely! 🙄

Heather Cullen

ITM Blog

Heather Cullen

Thank you – your message has been sent.

You will be notified when there is a new blog post.

 

Please note that Heather answers all questions at the end of the ITM Blog.

 

Happy trading!