11 Normaliser estimation using importance weights
11.1 Arithmetic mean estimator
Suppose we have two unnormalised probability densities, a target \(q_u(x)\) (hard to sample from, unknown normaliser) and surrogate \(m_u(x)\) (generally easy to sample from and with known normaliser).
\(L_q = \int q_u(x) dx\) and \(q(x) = \frac{q_u(x)}{L_q}\)
\(L_m = \int m_u(x) dx\) and \(m(x) = \frac{m_u(x)}{L_m}\)
\[\begin{align*} L_q &= \int \frac{q_u(x)}{m_u(x)} m_u(x) dx \\ &= L_m \cdot \int \frac{q_u(x)}{m_u(x)} m(x) dx \\ \frac{L_q}{L_m} &= \mathbb E_{X \sim m} \Bigg[\frac{q_u(X)}{m_u(X)}\Bigg] \end{align*}\]
This is an expectation with respect to the distribution of \(m\). If we can get i.i.d. samples \(X^1, \dots X^N \sim m\) and define \(w(X^n) = \frac{q_u(X^n)}{m_u(X^n)}\)
\[\hat L_{q/m} \frac{1}{N} \sum_{n=1}^{N} w(X^n) \rightarrow E_{X \sim m} [w(X)] = \frac{L_q}{L_m}\]
So if we have \(L_m\) in analytical form, we could easily estimate \(L_q\) by multiplying \(L_m\) with our estimate of the ratio.
11.2 Harmonic mean estimator
Observe that we could also do it the other way round!
Suppose we in fact have IID samples from our target (or MCMC ones for that matter- I don’t know the theory yet but it supposedly works out).
\[\begin{align*} L_m &= \int \frac{m_u(x)}{q_u(x)} q_u(x) dx \\ &= L_q \cdot \int \frac{m_u(x)}{q_u(x)} q(x) dx \\ \frac{L_m}{L_q} &= \mathbb E_{X \sim q} \Bigg[\frac{m_u(X)}{q_u(X)}\Bigg] \end{align*}\]
So we could estimate the inverse ratio using: \[ \hat L_{m/q}= \frac{1}{N} \sum_{n=1}^{N} \frac{1}{w(X^n)} \rightarrow E_{X \sim q} \bigg[\frac{1}{w(X)}\bigg] = \frac{L_m}{L_q}\]
And its inverse is the harmonic mean estimator: \[\tilde L_{q/m} = \frac{1}{\hat L_{m/q}} \rightarrow \frac{L_q}{L_m}\]
This can be then used to find \(L_q\) by multiplying it by \(L_m\).
Newton and Raftery (1994)
11.3 Comparison
The HM estimator is biased. It also has infinite variance lol.