Probability Perspective and Bayesian Perspective

Probability Perspective

The likelihood function is a function of the parameter w about the statistical model $p(x;w)$

Probability $p(x;w)$ describes the distribution of the random variable x when the parameter w is fixed.

Likelihood $p(x;w)$ describes the influence of different parameters w on the distribution of a known random variable x.

Maximum Likelihood Estimation (MLE）

Assume $X_1,X_2,\cdots ,X_n$ are samples from $X$, the probability of the observed samples occurrences is $\prod _{i=1}^{n}p(x_i;w)$.

Likelihood function

$$L(w)=L(x_1,x_2,\cdots ,x_n;w)=\prod _{i=1}^{n}p(x_i,w)$$

Maximize the likelihood function to get $\hat{w}$

$$\hat{w}=argma{x}_{w}L(x_1,x_2,\cdots ,x_n;w)$$

where $\hat{w}(x_1,x_2,\cdots ,x_n)$ is maximum likelihood estimation and $\hat{w}(X_1,X_2,\cdots ,X_n)$ is maximum likelihood statistic.

Log Likelihood Equation：

$$\frac{dL(w)}{dw}=0\to \frac{d(logL(w))}{dw}=0$$

Bayesian Perspective

When the training data is relatively small, overfitting will occur, resulting in inaccurate parameter estimation

Add prior (先验) knowledge to the parameters

Bayesian Learning: Consider the parameter $w$ as a random variable. Objective: Given a set of observation data $X$, find the distribution $p(w|X)$ of parameter $w$.

$p(w|X)$ is also called posterior distribution (后验分布)

Bayesian rule The relationship between $p(y|x)$ and $p(x|y)$:

$$p(y|x)=\frac{p(x|y)p(y)}{p(x)}$$$$p(w|x)\propto p(x|w)p(w)$$

$Posterior\propto Likelihood\times Prior$