# How Important is Variable Selection?

Very.

If you have 10 possible independent regressors, and none of which matter, you have a good chance to find at least one is important.

A good chance being 40%: prob(one or more looks important) = 1 – prob(non looks important) =

(1)

So, on average, you have around 60% chance to get the correct conclusion. Note that more data does nothing to solve this. Imagine 10 independent trading strategies backtested. Imagine 100. Related post is do-they-really-know-what-they-are-doing where I made the point not to get so excited by a money manager with a very high sharp (or information) ratio.
It is also the reason for my principle point in the R talk I gave in AmsteRdam, know where the profits (losses) stem from, what is the story behind what the speculator does.

The following code illustrates the problem, the chance of finding “important” unimportant variables. You can play around with the parameters and double check that increased N does nothing to mitigate the problem.

Run the code to get the typical figure:

### 2 comments on “How Important is Variable Selection?”

1. Thanks for the post. Very informative.

There is a good discussion of this issue in Maindonald and Braun Section 6.5. John Maindonald developed the function bestsetNoise() (DAAG package) that can be used to make a similar point.

Tony