This post is the next in the series “Using Quantifiable Edges to Your Advantage“. The first few posts examined how I lay out the studies, and what I look for when examining results that would make a study compelling. Today I will touch on what it means to have a compelling edge. Does it justify a trade? What if multiple studies appear to contradict each other?

First I should say that one of the biggest misconceptions about the studies I post on the blog is that they are market calls. They are not. They simply examine market action or conditions from one narrow perspective. It is rare that I would enter an index trade based on a single study. More often it is a combination of studies that helps provide me the confidence to put capital at risk.

I use studies as many traders use indicators. It is rare that someone might see all of their indicators line up perfectly at the same time. Often price action may be suggesting one thing, while breadth, or sentiment, or intermarket action may be suggesting something else. The tool I use to help me weight my studies and determine a market bias is the Aggregator. The basic method of the Aggregator is that it takes estimates from any studies I consider open and active, and combines them into one estimate. A more detailed description of the Aggregator can be found in this post.

But information about market tendencies that suggests compelling edges are useful for more than just index trades. No matter what securities you deal in, it helps to have a market bias. A bias doesn’t have to be formed mathematically, but by taking a reasoned approach traders can incorporate information from Quantifiable Edges, as well as other sources, in helping to establish their market bias. I’ll discuss this concept in more detail in a future post. The takeaway today is that a single study is not a market call, but rather a somewhat narrowly-focused examination of market tendencies. It’s a piece of the puzzle. A useful piece, but still just a piece.

And for my money, even if there is a combination of studies strongly suggesting a directional move, I still need to factor in risk/reward. I also need to consider what would constitute a trigger. My next post in this series will examine how I look at overbought/oversold and the role that plays in determining risk/reward. And hopefully by the end of the series I will be able to clearly demonstrate how narrowly focused studies can be used as part of a solid foundation when building your trading plan…or how experienced traders can incorporate them to improve an already solid trading plan.

This post will be the third in the series that looks at using quantifiable edges to your advantage. Today I will discuss considerations I use that help determine whether a study is compelling and whether I want to include that study in my analysis. I will be discussing nine things to consider. I have listed them all below.

1) Average Trade
2) Percent Profitable
3) Profit factor
4) Win / loss ratio
5) Equity Curve
6) Timing of instances
7) Average and max runup and drawdowns for the trades
8) Robustness across parameters
9) Robustness across securities

Average Trade – The average trade column on the results sheet is the first thing I tend to notice. It’s the number that’s most often used to determine expectations. Strong positive or strong negative numbers in this column would suggest a possible bullish or bearish edge.
Percent Profitable – Percent profitable helps to illustrate an edges consistency. It is a relatively important number because a high percent profitable suggests the odds of an extended drawdown would be low. As a trader controlling drawdown is important to me so I like to see high numbers in this column for bullish edges and low numbers for bearish edges.
Profit Factor – Profit factor is influenced by both win/loss ratio and average trade. It shows how large gross gains are versus gross losses. Here again I want to see high numbers for bullish studies and low numbers for bearish ones. I prefer to see a profit factor of at least two for bullish studies or at least 0.5 for bearish studies.

Win/loss ratio – This is probably the ratio I give the least amount of consideration to among those listed. Still it is preferable to see a strongly favorable win/loss ratio when considering a study’s worth. A positive win/loss ratio means you typically have more to gain on a trade than you have to lose. It will give a setup with a 50/50 chance of success (or sometimes worse) a positive expectancy.

Equity Curve – The equity curve is a visual representation of how the study has performed over time. It is important to consider for a few reasons. First, you want to see whether the edge has been steady or whether it is recently waxing or waning. Second, you want to see if the apparent statistical edge is the result of a few large outliers. This would suggest that a perceived edge may not be reliable. I could spend a lot of time discussing intricacies, but I think just a few pictures will provide a good idea of what I look for. Below are some equity curves that I have recently run and discussed in the subscriber letter. They are all real curves, but I have erased the setup criteria.
This first curve is one that I find especially appealing. When you can draw an arrow from the lower left to the upper right section of the chart and have it never be very far from the equity curve, then that suggests a steady, solid edge.

I also find this next curve appealing. The edge is not as steady, but you can see that there has always been an upward slope and that slope has recently increased.

This next curve is one that I do not find appealing. While it looked like the setup provided an edge for a period of time, it now appears that edge is either waning or no longer present. This is a setup I may continue to keep an eye on in the future, but would not include it as part of my current analysis.

This last curve is an example of one that was heavily affected by a few outliers. When the statistics were generated it appeared there may be a downside edge. A closer look at this equity curve tells a different story. What we see are three big losers in the middle of the curve that account for nearly all of the downside edge. If you take out those three instances, the rest of the curve is just sideways chop.

That should give you a decent sampling of things to look for when examining an equity curve. If you’re able to find curves that looked like the first one, then you should do very well in identifying edges.  I don’t commonly show equity curves in the blog, but it is an extra bit of information that I often include in the Subscriber Letter.
Timing of instances. Recent or distant history? – You want to look at more than just the statistics and the equity curve. You also want to look at when past setups similar to the one you’re examining have occurred. Was the trading environment substantially different then than it is now? For instance, I recently ran a study whose results were very positive and it had a very strong, steady equity curve. But when I looked at the dates I found that most of the instances had occurred in the 60s and 70s. In fact the current setup was the first occurrence since 1996. While the setup looked good from all other aspects, I was wary of including it in my analysis due to the fact that it hadn’t triggered in over 14 years. I would much rather rely upon results that have been achieved in the recent past as opposed to results that were all achieved in the distant past.
Average and max runup and drawdowns for the trades – I also make sure to note runup and drawdown statistics for each of the setups. This is helpful for number of reasons. Runup stats give me some idea of how much of a move to expect. If the market moves in the expected direction much beyond where a typical runup might take it, then I will often eliminate that study from my “Active List”. I do this because any further move in the expected direction is likely due to forces other than that particular study. Looking at drawdown statistics provides further insight into how the market typically reacts to the setup being studied. Is there typically a fast move in the expected direction with little or no drawdown? If so, a move up opposite expectations could suggest the market is sicker than usual. Drawdown statistics are also very useful in helping determine appropriate position size. If the environment is volatile and trades often take a lot of heat before moving in the expected direction, then more conservative position sizing is likely appropriate.

Robustness (part 1) – Robust results serve a better chance of performing similar to expectations going forward. Robustness can mean a few different things. One way to test robustness is to see whether a minor change in parameters causes a big difference in results. Ideally you want to see a setup work across a range of parameters. If the parameters are too fine tuned, then the perceived edge may be more a result of data mining and not an actual edge. If a new study is being conducted and the current setup barely qualifies for the parameters being described, then there is a much higher chance that study is not robust. Researchers should strive to ensure the setup they are describing is typical of the sample set that makes up the results, and not an extreme case or an outlier. Of course this isn’t always possible, but when a study appears robust and the setup is typical of the sample set, then I have a greater confidence level in those results.

Robustness (part 2) – Another way to test if a study or concept is robust is to run the setup across a broad list of securities. This isn’t something I typically do with the index studies that I publish on the blog and in the Subscriber Letter. It is something I do when I develop systems. So systems like the Quantifiable Edges Big Time Swing System or any of the “numbered systems” that are available to Gold subscribers are all tested across a broad range of securities. A system that tests well across a broad range of securities, rather than one that is tailored to a specific security, stands a better chance of performing well in real-time.

 
Those are my top criteria when determining whether a study is worthy of consideration in formulating my market bias.  In my next installment of this series I will discuss whether an edge justifies a trade, and how I weigh different edges when considering trading opportunities.

This post is the 2nd part of a series I started a few weeks ago that will discuss using quantifiable edges to your advantage.  Today I’ll discuss a common question I get about the studies.  How many instances are needed for valid and usable results?  It will lead into “What makes a study compelling?” in the next post.

Many of the posts I put on the blog are what I refer to as studies. In this previous post I showed the layout of the studies. A study is simply test results of an idea. Most of the time the idea is based in technical analysis. It looks to answer the question, “How has the market performed in the past after…”

Some studies are fairly general. For instance, I might look at how the market performs after it has traded down 3 days in a row. Others are more specific with added filters. Perhaps I notice that not only is the SPX down 3 days in a row, but it also is trading at a 10-day low, and is above the 200ma and volume has increased each of the last 3 days.

Both studies could tell me something about the market in relation to its current condition (assuming I’m describing current conditions, which is typically my approach). If I am able to describe conditions that more closely match the current market then I have a better shot at seeing behavior over the next several days match up with the study results. Of course there is a trade-off between general and specific, and that is the number of instances.

A general test may have hundreds or thousands of instances which it can refer to in order to generate expectations. A very specific test may have an extremely low number of instances. If the number of instances is too low then the results may have little or no meaning. For instance if my parameters are run and I find that the market had only set up in a similar manner 1 other time over my test period, is it reasonable to assume that the market will act the same way this time? Most people would correctly assume “no”. What if there were 2 instances and they both had similar reactions in the past. Could I assume this suggests a directional edge? 3 instances? 4? 10? 30? 50? More? How many instances is “enough” to have some level of confidence that your results are actually suggesting an edge and they are not the result of luck?

Before answering let me address 1 common misconception people have about statistical testing. That misconception is that you need 30 instances in order to demonstrate statistical significance. This idea originates in the fact that a sample size of 30 is needed in order to calculate a Z-score or run a chi-square test. The reason that 30 instances are necessary is that Z-scores assume a normal probability distribution. Without 30 instances it is not possible to resolve the shape of the normal probability distribution clearly enough to make certain statistical measures valid. One thing traders should be aware of is that the stock market does not have a normal distribution anyway. It has “fat tails”. In other words, there are more outliers present in stock market movements than one would expect under a normally distributed curve. So relying on standard statistical measures and assuming a normal distribution could expose a trader to more risk than his results would imply.

Still, these tests are helpful in determining whether your results were likely due to a real edge or whether there is a high risk that luck played a big part. But what if you don’t have 30 instances? In that case you could use a t-table statistic.

To better understand statistical significance and see how to run some of these tests I’ll refer you to the below post from a couple of years back:

https://quantifiableedges.blogspot.com/2008/05/significance.html

Note that this post also contains a t-table. One interesting thing we can see when looking at a t-table is the minimum number of instances you would need to have different confidence levels that your edge is actually an edge and not due to luck. For instance, if all instances were followed by a market rise, you would want at least 6 instances in order to be 95% confident that there was an actual edge. A 99.9% confidence would be reached if you had 11 instances that all resulted in a rise over the next X days.

So if you look back at the study I showed Wednesday, SPY only set up in that pattern 12 times in the past, but every time it was trading higher 5 days later. This means statistically there is about a 99.9% chance that the positive results were due to more than luck. That there has in fact been a real edge in that pattern in the past. Does this mean there is a 100% chance it will be higher 5 days after the setup? No! Not even close. A high degree of confidence means there is likely some kind of an edge. It doesn’t mean the past winning % or net expectations are likely to persist indefinitely.

So how many instances do I require before I’m willing to accept a study as part of my analysis and place it on my active list? It varies depending on things like the strength of previous reactions and other stats I’ll get into in my next post, but I’ll generally use a t-table to help me decide. Will I incorporate a study with only 10 or 11 instances? Yes, but it will have to have strong win/loss stats and a high win %. Personally, I tend to favor studies that have somewhere between 20-70 instances. Too low and they are less reliable. Too high and the setup is often too broad to have much meaning.

I’ve spent far more space discussing this than I wanted, but it is an issue that has come up time and again with readers, so I wanted to be somewhat thorough.

In fact, of the list of things I look at in a study to help me decide whether it is compelling or not, the number of instances (assuming it isn’t minuscule) is near the bottom .

I intend to accelerate this series of posts over the next couple of weeks and I’m sorry it’s taken so long to get rolling. In the next post I will discuss a list of other things I examine when determining whether I find a study compelling.