Inference - Eran Raviv

Fairness in machine learning: Equalized Odds

Blog, Miscellaneous, Statistics and EconometricsPosted on 05/26/2026

I recently dipped my toes into fairness concepts in machine learning. What does being fair mean, practically? is it fair that I was not born with the physique required to qualify for the NBA?

Absolutely it’s fair.

In the lottery of life, my genes were decided on in the same way as they were for everyone else who was born. Fairness does NOT mean “treating everyone the same”, rather to have the same starting point in that nature-lottery.

That written, in reality it’s not only nature who is calling the shots. We the people, also play an important part. When it comes to high-stakes predictions (say who gets a loan, or who passes a medical screening) we want enforce the same starting point for everyone. Regrettably, applying the same model to everyone doesn’t directly imply equal treatment. Why? Because we don’t treat everyone the same. Simply speaking, models are trained on real-world data produced by… us. And, since we do not treat everyone the same, our models inevitably reflect that reality. We can only ask to much from our models (see my short rant about the bias in AI misconception in that regards).

Statistical Shrinkage (3)

Blog, R, Statistics and EconometricsPosted on 11/18/2023

Imagine you’re picking from 1,000 money managers. If you test just one, there’s a 5% chance you might wrongly think they’re great. But test 10, and your error chance jumps to 40%. To keep your error rate at 5%, you need to control the “family-wise error rate.” One method is to set higher standards for judging a manager’s talent, using a tougher t-statistic cut-off. Instead of the usual 5% cut (t-stat=1.65), you’d use a 0.5% cut (t-stat=2.58).

When testing 1,000 managers or strategies, the challenge increases. You’d need a manager with an extremely high t-stat of about 4 to stay within the 5% error rate. This big jump in the t-stat threshold helps keep the error rate in check. However that is discouragingly strict: a strategy which t-stat of 4 is rarity.

Beware of Spurious Factors

Blog, Finance and Trading, Statistics and EconometricsPosted on 12/13/2022

The word spurious refers to “outwardly similar or corresponding to something without having its genuine qualities.” Fake.

While the meanings of spurious correlation and spurious regression are common knowledge nowadays, much less is understood about spurious factors. This post draws your attention to recent, top-shelf research, flagging the risks around spurious factor analysis. While formal solutions are still pending there are couple of heuristics we can use to detect possible problems.

Random forest importance measures are NOT important

Blog, Statistics and EconometricsPosted on 05/16/2021

Random Forests (RF from here onwards) is a widely used pure-prediction algorithm. This post assumes good familiarity with RF. If you are not familiar with this algorithm, stop here and see the first reference below for an easy tutorial. If you used RF before and you are familiar with it, then you probably encountered those “importance of the variables” plots. We start with a brief explanation of those plots, and the concept of importance scores calculation. Main takeaway from the post: don’t use those importance scores plots, because they are simply misleading. Those importance plots are simply a wrong turn taken by our human tendency to look for reason, whether it’s there or it’s not there.

Test of Equality Between Two Densities

Blog, Statistics and EconometricsPosted on 10/08/2018

Are returns this year actually different than what can be expected from a typical year? Is the variance actually different than what can be expected from a typical year? Those are fairly light, easy to answer questions. We can use tests for equality of means or equality of variances.
But how about the following question:

is the profile\behavior of returns this year different than what can be expected in a typical year?

This is a more general and important question, since it encompasses all moments and tail behavior. And it is not as trivial to answer.

In this post I am scratching an itch I had since I wrote Understanding Kullback – Leibler Divergence. In the Kullback – Leibler Divergence post we saw how to quantify the difference between densities, exemplified using SPY return density per year. Once I was done with that post I was thinking there must be a way to test the difference formally, rather than just quantify, visualize and eyeball. And indeed there is. This post aim is to show to formally test for equality between densities.

Understanding False Discovery Rate

Blog, Statistics and EconometricsPosted on 04/04/2017

False Discovery Rate is an unintuitive name for a very intuitive statistical concept. The math involved is as elegant as possible. Still, it is not an easy concept to actually understand. Hence i thought it would be a good idea to write this short tutorial.

We reviewed this important topic in the past, here as one of three Present-day great statistical discoveries, here in the context of backtesting trading strategies, and here in the context of scientific publishing. This post target the casual reader, explaining the concept of False Discovery Rate in plain words.

Why statistical bootstrap

Blog, Statistics and EconometricsPosted on 07/13/2016

I often write about bootstrap (here an example and here a critique). I refer to it here as one of the most consequential advances in modern statistics. When I wrote that last post I was searching the web for a simple explanation to quickly show how useful bootstrap is, without boring the reader with the underlying math. Since I was not content with anything I could find, I decided to write it up, so here we go.

Quasi-Maximum Likelihood (QML) beauty

Blog, Statistics and EconometricsPosted on 05/16/2015

Beauty.. really? well, beauty is in the eye of the beholder.

Linking backtesting with multiple testing

Blog, Finance and Trading, Statistics and EconometricsPosted on 11/23/2014

The other day, Harvey Campbell from Duke University gave a talk where I work. The talk- bearing the exciting name “Backtesting” was based on a paper by the same name.

The authors tackle the important problem of data-snooping; we need to account for the fact that we conducted many trials until we found a strategy (or a variable) that ‘works’. Accessible explanations can be found here and here. In this day and age, the ‘story’ behind what you are doing is more important than ever, given the things you can do using your desktop/laptop.

Advances in post-model-selection inference (2)

Blog, Statistics and EconometricsPosted on 10/15/2014

In the previous post we reviewed a way to handle the problem of inference after model selection. I recently read another related paper which goes about this complicated issues from a different angle. The paper titled ‘A significance test for the lasso’ is a real step forward in this area. The authors develop the asymptotic distribution for the coefficients, accounting for the selection step. A description of the tough problem they successfully tackle can be found here.

The usual way to test if variable (say variable j) adds value to your regression is using the F-test. We once compute the regression excluding variable j, and once including variable j. Then we compare the sum of squared errors and we know what is the distribution of the statistic, it is F, or $\chi^2$ , depends on your initial assumptions, so F-test or $\chi^2$ -test. These are by far the most common tests to check if a variable should or should not be included. Problem arises if you search for variable j beforehand.