This post is the 2nd part of a series I started a few weeks ago that will discuss using quantifiable edges to your advantage.  Today I’ll discuss a common question I get about the studies.  How many instances are needed for valid and usable results?  It will lead into “What makes a study compelling?” in the next post.

Many of the posts I put on the blog are what I refer to as studies. In this previous post I showed the layout of the studies. A study is simply test results of an idea. Most of the time the idea is based in technical analysis. It looks to answer the question, “How has the market performed in the past after…”

Some studies are fairly general. For instance, I might look at how the market performs after it has traded down 3 days in a row. Others are more specific with added filters. Perhaps I notice that not only is the SPX down 3 days in a row, but it also is trading at a 10-day low, and is above the 200ma and volume has increased each of the last 3 days.

Both studies could tell me something about the market in relation to its current condition (assuming I’m describing current conditions, which is typically my approach). If I am able to describe conditions that more closely match the current market then I have a better shot at seeing behavior over the next several days match up with the study results. Of course there is a trade-off between general and specific, and that is the number of instances.

A general test may have hundreds or thousands of instances which it can refer to in order to generate expectations. A very specific test may have an extremely low number of instances. If the number of instances is too low then the results may have little or no meaning. For instance if my parameters are run and I find that the market had only set up in a similar manner 1 other time over my test period, is it reasonable to assume that the market will act the same way this time? Most people would correctly assume “no”. What if there were 2 instances and they both had similar reactions in the past. Could I assume this suggests a directional edge? 3 instances? 4? 10? 30? 50? More? How many instances is “enough” to have some level of confidence that your results are actually suggesting an edge and they are not the result of luck?

Before answering let me address 1 common misconception people have about statistical testing. That misconception is that you need 30 instances in order to demonstrate statistical significance. This idea originates in the fact that a sample size of 30 is needed in order to calculate a Z-score or run a chi-square test. The reason that 30 instances are necessary is that Z-scores assume a normal probability distribution. Without 30 instances it is not possible to resolve the shape of the normal probability distribution clearly enough to make certain statistical measures valid. One thing traders should be aware of is that the stock market does not have a normal distribution anyway. It has “fat tails”. In other words, there are more outliers present in stock market movements than one would expect under a normally distributed curve. So relying on standard statistical measures and assuming a normal distribution could expose a trader to more risk than his results would imply.

Still, these tests are helpful in determining whether your results were likely due to a real edge or whether there is a high risk that luck played a big part. But what if you don’t have 30 instances? In that case you could use a t-table statistic.

To better understand statistical significance and see how to run some of these tests I’ll refer you to the below post from a couple of years back:

https://quantifiableedges.blogspot.com/2008/05/significance.html

Note that this post also contains a t-table. One interesting thing we can see when looking at a t-table is the minimum number of instances you would need to have different confidence levels that your edge is actually an edge and not due to luck. For instance, if all instances were followed by a market rise, you would want at least 6 instances in order to be 95% confident that there was an actual edge. A 99.9% confidence would be reached if you had 11 instances that all resulted in a rise over the next X days.

So if you look back at the study I showed Wednesday, SPY only set up in that pattern 12 times in the past, but every time it was trading higher 5 days later. This means statistically there is about a 99.9% chance that the positive results were due to more than luck. That there has in fact been a real edge in that pattern in the past. Does this mean there is a 100% chance it will be higher 5 days after the setup? No! Not even close. A high degree of confidence means there is likely some kind of an edge. It doesn’t mean the past winning % or net expectations are likely to persist indefinitely.

So how many instances do I require before I’m willing to accept a study as part of my analysis and place it on my active list? It varies depending on things like the strength of previous reactions and other stats I’ll get into in my next post, but I’ll generally use a t-table to help me decide. Will I incorporate a study with only 10 or 11 instances? Yes, but it will have to have strong win/loss stats and a high win %. Personally, I tend to favor studies that have somewhere between 20-70 instances. Too low and they are less reliable. Too high and the setup is often too broad to have much meaning.

I’ve spent far more space discussing this than I wanted, but it is an issue that has come up time and again with readers, so I wanted to be somewhat thorough.

In fact, of the list of things I look at in a study to help me decide whether it is compelling or not, the number of instances (assuming it isn’t minuscule) is near the bottom .

I intend to accelerate this series of posts over the next couple of weeks and I’m sorry it’s taken so long to get rolling. In the next post I will discuss a list of other things I examine when determining whether I find a study compelling.

Related Quantifiable Edges Studies

• How to Read QE Studies: Understanding the Study Layout (Part 1)
• Using QE Subscriber Tools: The Catapult System and CBI
• QE Aggregator: How It Uses Risk/Reward Analysis to Set Market Bias
• IBD Follow Through Days Quantified: Summary of the 4-Part Study

This post is the beginning in a series which will provide readers some ideas on how they can take some of the edges they see here (and elsewhere) and use them to their advantage in their own trading.

Before getting into a theoretical discussion it’s important that I make sure everyone understands what it is I’m presenting when I show these studies in the blog and the Subscriber Letter. Over time I have pretty much standardized the statistics that I show in my tables. I have tried to strike a balance between giving enough information to make the table useful and giving too much information which could make it messy and confusing. Below is a sample study (with real results but a bogus description). I’ll use this as an example to refer to.

The top box of the study always lays out the conditions. Everything that was taken into account is described there. One thing to note is that I always run the studies on $100k/trade. This is because most of them look at the S&P 500. Since it trades at about 1,200 there will always be some leftover when buying into a portfolio (you can’t buy a half a share in Tradestation). The $100k makes the rounding error small enough so that it doesn’t have much of an affect. $10,000 would have too large of a rounding error. $1,000,000 would be better but then the numbers get so large it makes it more difficult to read.

With an even $100k I find the results easy to interpret. $1000 = 1% in the results columns. So in the above example the “Average Trade” shows a gain of $733 after 3 days. This is almost 0.75%.

Now let’s briefly review each column in the results table.

“X Days” – Most tests I run out over a number of days to see how the market has performed after the test conditions were in place. “X Days” just shows the length of time from the entry. The entry is normally assumed to take place at the close. The exits are also assumed to be at the close. The number of days refers to trading days – not calendar days. Note this column reads from the bottom up, which means all columns do. No reason for that. It’s just how I started doing it a long time ago.

“Net Profit” – This is the net gain or loss for the entire sample of instances included in the study. One thing to note is that I always “Buy” the setup. This is not because I am only looking for long edges. It is because it makes the table easier to read. A quick glance can tell me if the edge is bullish or bearish. Lots of positive, green numbers is bullish. Lots of negative, red numbers is bearish. A long time ago I would sometimes set the entry condition to “short” at the close. Then I could see profits from shorting. Doing this required me to read the entry conditions carefully and would occasionally lead to some confusion when I didn’t. So for purposes of easily reading the study tables, everything assumes a long position.

“Total Trades” – This is the total number of instances that triggered based on the study conditions. As in the case above, this number will sometimes be larger for Day 1 and then you’ll see a declining number of instances as you look further out. If you’re wondering why this happens, check out the June 18, 2010 blog post.

“Winning Trades” – The total number of trades that were showing a gain “X Days” later.

“Losing Trades” – The total number of trades showing a loss “X Days” later. The wining plus the losing trades typically add up to the total trades. In those rare instances when it doesn’t it means a trade was breakeven on that day.

“% Profitable” – This column simply shows the winners / total trades. Sometimes a 50% profitable situation can still show a strong edge. That would mean gains outsized losses by a large degree (or vice-versa). % profitable is important from a trading standpoint though. If a setup is 90% profitable it is generally less likely to put you through an extended drawdown as a setup that is 55% profitable with the same size average trade.

“Avg Winning Trade” – This looks at all the “winning trades” and divides them by the gross profits on those trades. (Gross gain and gross loss columns are not shown.) So in the table above, the “Avg Winning Trade” was up $692 after day 1. This means that of the 14 instances that finished higher the next day, the average gained just under 0.7%.

“Avg Losing Trade” – Just like “Avg Winning Trade”, but it is looking just at the losers. In this case after day 1, the 7 losers dropped about 0.9% on average.

“Win/Loss Ratio” – This takes the value from the “Avg Winning Trade” column and divides by the value from the “Avg Losing Trade” column. It can help you determine whether the reaction was typically more explosive on moves up or down.

“Profit Factor” – This is the stat I am asked about the most. It is a stat often cited by system traders. Profit Factor = Gross Gains / Gross Losses. Profit factors above 1 occur when there are positive net results and below 1 occurs when there are negative net results from a study. When thinking about the importance of profit factor, it is easiest to consider how 2 systems may compare. Consider 2 systems made a hypothetical $10,000 each over a specified time period. System 1 had $15,000 in gains and $5,000 in losses. Its profit factor was 3. (15k/5k = 3). System 2 also made $10,000 but it was on $100,000 in gains and $90,000 in losses. Its profit factor was 1.11 (100/90). Most people would find system 1 more appealing as it seemed to make the $10,000 with less effort and risk.

“Avg Trade” – This is simply the net gains divided by the total trades. Under most circumstances, I’ll use the information in this column to help generate estimates.

Last but not least I will often place a statement with additional information in a box below the results. This is typically information that can’t be seen in the table. A common bit of information I put here is how often the market might close up (or down) from the entry price at some point in the next few days.

In the next installment of this series I’ll give a brief discussion of attributes that would make a study compelling to me and entice me to incorporate it in formulating my market bias.

Related Quantifiable Edges Studies

• How Many Instances Do You Need to Trust a Market Study? Methodology
• Using QE Subscriber Tools: The Catapult System and CBI
• Quantifiable Edges Aggregator: Tallying Study Projections Daily
• QE Aggregator: How It Uses Risk/Reward Analysis to Set Market Bias