17 Parameter estimation
17.0.1 Setup
Let \theta_0 be the true parameter of interest.
We have i.i.d. samples X_1,\dots X_n \sim P. Let \hat P_n := \frac{1}{n}\sum_{i=1}^n \delta_{X_i} represent the empirical measure.
17.1 M estimators
Found by maximising some objective: M_n (\theta) := \frac{1}{n} \sum_{i=1}^n m_\theta(X_1) = \hat P_n [m_\theta] \hat \theta_n = \arg\max M_n (\theta)
The true parameter value satisfies: M(\theta) = P[m_\theta] \theta_0 = \arg\max M(\theta)
Maximum likelihood estimation is one example of this, where m_\theta = \log f_\theta.
By LLN, we get pointwise consistency of the objective:
M_n (\theta) \stackrel{\to}{p} M(\theta)
17.1.1 Consistency
Conditions:
- Uniform Law of Law Numbers: \sup_{\theta \in \Theta} |M_n(\theta) - M(\theta)| \stackrel{\to}{p} 0
- Well separated maximum: \sup_{\theta: d(\theta, \theta_0) > \epsilon} M(\theta) < M(\theta_0)
- Nearly correct optimisation: \hat \theta_n is a sequence of estimators such that M_n(\hat \theta_b) \geq M_n(\theta_0) - o_P(1)
If these hold then \hat \theta_n \stackrel{\to}{p} \theta_0
Proof
17.2 Z estimators
Found by solving a system of equations: \Psi_n(\theta) := \sum_{i=1}^n \psi_\theta(X_i) = \hat P_n [\psi_\theta] \Psi_n(\hat \theta_n) = 0
The true value satisfies: \Psi(\theta) = P[\psi_\theta] \Psi(\theta_0) = 0
One example is moment matching estimation.
By LLN: \Psi_n (\theta) \stackrel{\to}{p} \Psi(\theta)
Z-estimators can be viewed as a special categopry of M-estimators where we maximise the objective \theta \mapsto - \|\Psi_n(\theta) \|