Section 1.2

Chapter 1 Statistical Models, Goals, And Performance Criteria

1.2 Bayesian Models

Bayesian Framework

• Statistical Model
  • A random variable $X$
    • $\mathcal{X}$: Sample Space = {outcomes x}
    • $\mathcal{F}_X$: sigma-field of measurable events
    • $P(\cdot)$: probability distribution defined on $(\mathcal{X}, \mathcal{F}_X)$
  • Statistical Model
    • $\mathcal{P} = {P_\theta, \theta\in\Theta}$
    • Parameter $\theta$ identifies/specifies distribution in $\mathcal{P}$
• Bayesian Principle
  • Assume that the true value of the parameter $\theta$ is the realization of a random variable:
    • $\theta\sim\pi(\cdot)$, where $\pi(\cdot)$ is a distribution on $(\Theta, \sigma_\Theta)$
  • The distribution $(\Theta, \sigma_\Theta, \pi)$ is the Prior Distribution for $\theta$.
  • The specification of $\pi(\cdot)$ may be
    • purely subjective (personalistic)
    • based on actual data (empirical Bayes)
• Bayesian Framework

  • Prior distribution for $\theta$ with density/pmf function

    \[\pi(\theta), \quad\theta\in\Theta\]

  • Conditional distributions for $X$ given $\theta, P_\theta$, with density/pmf function.

    \[p(x\lvert\theta), \quad x\in\mathcal{X}\]

  • Joint distribution for $(\theta ,X)$ with joint density/pmf function

    \[f(\theta, x) = \pi(\theta)p(x\lvert\theta)\]

  • Posterior distribution for $\theta$ given $X=x$ with density/pmf function
    Discrete Prior

    \[\pi(\theta\lvert x) = \frac{\pi(\theta)p(x\lvert\theta)}{\sum_t\pi(t)p(x\lvert t)}\]

    Continuous Prior

    \[\pi(\theta\lvert x) = \frac{\pi(\theta)p(x\lvert\theta)}{\int_\Theta\pi(t)p(x\lvert t)}\]

  • Conjugate Prior Distribution Prior and Posterior in same family.

    • Example: Bernoulli Trial Given a bernoulli trial,
      • $X_1, \ldots, X_n$ are i.i.d. Bernoulli($\theta$) r.v.s.
      • Outcome space: $\mathcal{X} = {Success(1), Failure()0}$
      • Parameter Space: $\Theta = {\Theta: 0\leq\theta\leq 1}$
      • Prior Distribution for $\theta$: density $\pi(\theta)$

      • Posterior Distribution for $\theta$:

        \[\pi(\theta\lvert x_1,\ldots,x_n) = \frac{\pi(\theta)\theta^k(1-\theta)^{n-k}}{\int^1_0\pi(t)t^k(1-t)^{n-k}dt}\]

        such that

        \[\begin{aligned} &0<\theta<1 \\ &x_1 = 0 \,or\,1, \quad i=1,\ldots,n \\ &k=\sum_{i=1}^n x_i \end{aligned}\]
      • Conjugate Prior Distribution

        • A priori, assume $\theta\sim B(r,s)$ distribution, with density

          \[\pi(\theta) = \frac{\theta^{r-1}(1-\theta)^{s-1}}{\beta(r,s)},\quad 0<\theta<1\]

          where

          \[\begin{aligned} \beta(r,s) &= \int_0^1\theta^{r-1}(1-\theta)^{s-1}d\theta \\ &= \frac{\Gamma(r)\Gamma(s)}{\Gamma{r+s}} \end{aligned}\]

          And we have,

          \[\begin{aligned} &E[\theta] = \frac{r}{r+s} \\ &Var(\theta) = \frac{rs}{[(r+s)^2(r+s+1)]} \end{aligned}\]

        • A posteriori

          \[\pi(\theta\lvert T(X) =k)\sim B(r+k, s+(n-k))\]