Bayesian vs. Frequentist in Practice

Rivers of ink have been spilled over the ‘Bayesian vs. Frequentist’ dispute. Most of us were trained as Frequentists. Probably because the computational power needed for Bayesian analysis was not around when the syllabus of your statistical/econometric courses was formed. In this age of tablets and fast internet connection, your training does not matter much, you can easily transform between the two approaches, engaging the right webpages/communities. I will not talk about the ideological differences between the two, or which approach is more appealing and why. Larry Wasserman already gave an excellent review.

In practice, I am not sure there is big difference. I consider Bootstrap to be the Frequenist analogue for the Bayesian approach. The relation was pointed out already in 1981 by Donald Robin.

In the following toy example I apply both Bootstrap and Bayesian inference, and push forward the point that (Oy vey..) the choice between the two approached does not matter much.

The model for today is:

(1)   \begin{equation*} y_t = \beta_0 + \beta_1 x_{1,t} +\beta_2 x_{2,t} + \varepsilon_t => y_t = 2 + 2x_{1,t} +4x_{2,t} + \varepsilon_t  \end{equation*}

The noise is simulated once from Normal and once from heavy tail t-dist.

Now we estimate the coefficients using Bayesian Regression and get their distribution.

The function Bayes_reg is using Gibbs sampling – composition method.

Now Frequenist, using bootstrap:

Compare resulting distribution of the estimates. First the first coefficient:

Distribution of the first coefficient according to Bayesian and Frequenist
Distribution of the first coefficient according to Bayesian (top) and Frequenist (bottom)
Second the second coefficient:
Distribution of the second coefficient according to Bayesian and Frequenist
Distribution of the second coefficient according to Bayesian and Frequenist
Code for the figures below.

You can see how similar are the resulting distributions. The solid black line is the real value of the coefficient. By construction both Bayesian and Frequenist has the same center (around the OLS estimate when we have no prior guess) represented by the purple an orange solid lines. Whatever the distribution of the residuals, inference is going to be VERY similar. Examine the resulting actual Confidence Intervals of inference:

The confidence interval differ only slightly. So despite the ideological differences (“What do you exactly mean by the distribution of the parameters???, what kind of statistician are you??”), when it comes to inference in this toy example, distinguishing between the two approaches is not worth the effort.

Conclusion, don’t be a Bayesian, nor be a Frequenist, be opportunist. Do whatever you find is easier. My intuition is that they the two approaches yield very different results only in a carefully constructed scenarios.

Appendix:
Code for Figures:

One comment on “Bayesian vs. Frequentist in Practice”

Leave a Reply

Your email address will not be published.