2.4 Objective priors

A parameter $\mu \mathrm{\in }\mathrm{\Re }$ on the real line

can be given the prior

$$\pi (\mu )\propto 1.$$

For a normal distribution with variance assumed known leads to

$$\pi (\theta \mid \text{𝐲})\propto P(\text{𝐲}\mid \theta )$$

or the posterior is proportional to the likelihood.

For a positive parameter $\lambda$

it is possible to transfer to the real line using the log transformation $\gamma =\log (\lambda )\in \mathrm{\Re }$ and allocate that a uniform prior $P(\gamma )\propto 1$. This means

	$\pi (\lambda )$	$={\displaystyle \frac{d\gamma }{d\lambda }}p(\gamma )$
		$\propto {\displaystyle \frac{1}{\lambda }}$

An example is ${\sigma }^2$, the variance parameter of a normal distribution :

$$P({\sigma }^2)\propto \frac{1}{{\sigma }^2}$$

.

For a proportion $\pi$

it is possible to transfer to the real line using the logit transformation: $\delta =\log \left(\frac{\pi }{1-\pi }\right)\in \mathrm{\Re }$ and allocate that a uniform prior $P(\delta )\propto 1$. This means

	$P(\pi )$	$={\displaystyle \frac{d\delta }{d\pi }}p(\delta )$
		$\propto {\displaystyle \frac{1}{\pi }}-{\displaystyle \frac{1}{1-\pi }}$
		$\propto {\displaystyle \frac{1}{\pi (1-\pi )}}$

This prior is improper and called Haldane’s prior

Consider that we might have specified a prior $f_{\mathrm{\Theta }}(\theta )$ for a parameter $\theta$ in a model. It is quite reasonable to decide to use instead the parameter $\varphi =1/\theta$. For example $\theta$ may be the parameter of the exponential distribution of inter-arrival times in a queue, and represents the arrival rate. Then $\varphi$ represents the mean inter-arrival time. By probability theory the corresponding prior density for $\varphi$ must be given by

	$f_{\mathrm{\Phi }}(\varphi )$	$=$	$f_{\mathrm{\Theta }}(\theta )\times \left|{\displaystyle \frac{d\theta }{d\varphi }}\right|$
		$=$	$f_{\mathrm{\Theta }}(1/\varphi ){\displaystyle \frac{1}{{\varphi }^2}}.$

If we decided that we wished to express our ignorance about $\theta$ by choosing $f_{\mathrm{\Theta }}(\theta )\propto 1$, then we are forced to take $f_{\mathrm{\Phi }}(\varphi )\propto 1/{\varphi }^2$.

But if we are ignorant about $\theta$, we are surely equally ignorant about $\varphi$, and so might equally have made the specification $f_{\mathrm{\Phi }}(\varphi )\propto 1$. Thus, prior ignorance as represented by uniformity of belief, is not preserved under re-parametrization.

Jeffreys’ prior is invariant under a parameter transformation $\varphi (\theta )$ and may be stated as:

$$J_{\mathrm{\Phi }}(\varphi )=J_{\mathrm{\Theta }}(\theta )\left|\frac{d\theta }{d\varphi }\right|.$$

There is one way of using the likelihood $L(\theta ;x)=f(x|\theta )$, or more accurately, the log likelihood $\mathrm{\ell }(\theta )=\log L(\theta ;x)$, to specify a prior which is consistent across 1—1 parameter transformations. This is the ‘Jeffreys’ prior’, and is based on the concept of Fisher information:

$$I(\theta )=-\text{E }\left\{\frac{d^2\mathrm{\ell }(\theta )}{d{\theta }^2}\right\}=\text{E }\left\{{\left(\frac{d\mathrm{\ell }(\theta )}{d\theta }\right)}^2\right\}.$$

Jeffreys’ prior can be defined as

$$J_{\mathrm{\Theta }}(\theta )\propto {|I(\theta )|}^{\frac{1}{2}}.$$

Suppose $x|\theta \sim \text{Binomial }(n,\theta )$. Find Jeffreys’ prior. Is it proper?

$$\frac{d^2\mathrm{\ell }(\theta )}{d{\theta }^2}=\frac{-x}{{\theta }^2}-\frac{(n-x)}{{(1-\theta )}^2},$$

and since $E(x)=n\theta$,

	$I(\theta )$	$=$	${\displaystyle \frac{n\theta }{{\theta }^2}}+{\displaystyle \frac{(n-n\theta )}{{(1-\theta )}^2}}$
		$=$	$n{\theta }^{-1}{(1-\theta )}^{-1},$

leading to $f_0(\theta )\propto {\theta }^{-1/2}{(1-\theta )}^{-1/2}$ which in this case is the proper distribution

$$\theta \sim \text{Beta }(\frac{1}{2},\frac{1}{2})$$

.

Find the Jeffreys prior for $\theta$ in the geometric model: $f(x|\theta )={(1-\theta )}^{x-1}\theta ;x=1,2,\mathrm{\dots }.$ (Note $\text{E}(X)=1/\theta$.)

	$\mathrm{\ell }(\theta )$	$=(x-1)\log (1-\theta )+\log (\theta )$
	${\mathrm{\ell }}^{{}^{\prime }}(\theta )$	$=-{\displaystyle \frac{x-1}{1-\theta }}+{\displaystyle \frac{1}{\theta }}$
	${\mathrm{\ell }}^{{}^{\prime \prime }}(\theta )$	$=-{\displaystyle \frac{x-1}{{(1-\theta )}^2}}-{\displaystyle \frac{1}{{\theta }^2}}$

	$I(\theta )$	$={\text{E }}_x\left[{\mathrm{\ell }}^{{}^{\prime \prime }}(\theta )\right]$
		$={\text{E }}_x\left[{\displaystyle \frac{x-1}{{(1-\theta )}^2}}+{\displaystyle \frac{1}{{\theta }^2}}\right]$
		$={\displaystyle \frac{\frac{1}{\theta }-1}{{(1-\theta )}^2}}+{\displaystyle \frac{1}{{\theta }^2}}={\displaystyle \frac{1-\theta }{\theta {(1-\theta )}^2}}+{\displaystyle \frac{1}{{\theta }^2}}$
		$={\displaystyle \frac{1}{{\theta }^2(1-\theta )}}$
	$⟹J(\theta )$	$={\left|I(\theta )\right|}^{\frac{1}{2}}={\theta }^{-1}{(1-\theta )}^{-\frac{1}{2}}$

$⟹\theta \sim \text{Beta }(0,\frac{1}{2})$ Which is not proper.

• •

  Conjugate priors are often convenient because the posterior, marginal likelihood and predictive correspond (in the case of likelihoods from the exponential family) to known distributions.

• •

  However conjugate priors cannot represent total ignorance.

• •

  Improper priors can do this but have problems. For instance the marginal likelihood does not exist and Bayes factors cannot be calculated.

• •

  Jeffreys’ priors are invariant to monotonic transformation

• •

  Laplacian priors are calculated by transforming the parameter to the set of real numbers and giving the transformed parameter a flat prior.

• •

  Priors are both the greatest strength of Bayesian statistics and its greatest weakness.