Statistical Learning

Probability Distribution

• Probability that a random variable (r.v.) $X$ takes every possible value.

$$P(X=x_i)=p(x_i),\mathrm{\forall }i\in \{1,\dots ,n\}$$

• Satisfying:

$$\sum _{i=1}^{n}p(x_i)=1,p(x_i)\ge 0,\mathrm{\forall }i\in \{1,\dots ,n\}$$

Discrete Random Variable

Probability Mass Function (PMF)

• Probability Mass Function (PMF) is the probability that the value of r.v. $X$ is $x_i$:

Bernoulli Distribution

• In a test, event $A$ happens with probability $\mu$, does not happen with probability $1-\mu$.

• If using r.v. $X$ to indicate the number of occurrences of event $A$, then $X$ can be 0 or 1. Its distribution is:

$$p(x)={\mu }^x(1-\mu {)}^{1-x}$$

Binomial Distribution

• In the $n$ times Bernoulli distribution, if r.v. $X$ represents the number of occurrences of event $A$, the value of $X$ is $\{0,\dots ,n\}$, and its corresponding distribution:

$$p(X=k)=(\genfrac{}{}{0ex}{}{n}{k}){\mu }^k(1-\mu {)}^{n-k},k=1,\dots ,n$$

• The binomial coefficient represents the total number of combinations of elements taken out of $n$ elements regardless of their order.

Continuous Random Variable

Probability Density Function (PDF)

Probability distribution can be described by the Probability Density Function (PDF) $f(x)$:

$${\int }_{-\mathrm{\infty }}^{+\mathrm{\infty }}f(x)dx=1$$

Cumulative Distribution Function (CDF)

The Cumulative Distribution Function (CDF) is the probability that the value of r.v. $X$ is less than or equal to $x$:

$$F(x)=P(X\le x)$$

• For continuous r.v., we have:

$$F(x)={\int }_{-\mathrm{\infty }}^{x}f(t)dt$$

Gaussian Distribution

$$X\sim \mathcal{N}(\mu ,{\sigma }^2)$$$$P(x)=\frac{1}{\sqrt{2\pi \sigma }}\exp (-\frac{(x-\mu {)}^2}{2{\sigma }^2})$$

Marginal Distribution

Marginal Probability Mass Function

• $X$ 的边际概率质量函数：

  $$p_X(x_i)=\sum _{j}p(x_i,y_j)$$

• $Y$ 的边际概率质量函数：

  $$p_Y(y_j)=\sum _{i}p(x_i,y_j)$$

Marginal Probability Density Function

• $X$ 的边际概率密度函数：

  $$f_X(x)={\int }_{-\mathrm{\infty }}^{+\mathrm{\infty }}f(x,y)dy$$

• $Y$ 的边际概率密度函数：

  $$f_Y(y)={\int }_{-\mathrm{\infty }}^{+\mathrm{\infty }}f(x,y)dx$$

Conditional Probability

For a discrete random vector $(X,Y)$, when $X=x$ is known, the conditional probability of r.v. $Y=y$ is:

$$p(y|x)=P(Y=y|X=x)=\frac{p(x,y)}{p(x)}$$

Sampling

Sampling: given a probability distribution p(x), generate samples that meet the conditions

$$x^{(1)},x^{(2)},\cdots ,x^{(N)}\sim p(x)$$

Expectation

Expectation: the average of random variable

For discrete r.v. $X$:

$$\mathbb{E}[X]=\sum _{n=1}^N{x}_{n}p(x_n)$$

For continuous r.v. $X$:

$$\mathbb{E}[X]={\int }_{-\mathrm{\infty }}^{+\mathrm{\infty }}xf(x)dx$$

Law of Large Numbers

When the number of samples is large, the sample mean and the real mean (expectation) are fully close.

Given N independently and identically distributed (I.I.D.) samples

$$x^{(1)},x^{(2)},\cdots ,x^{(N)}\sim p(x)$$

Its mean value converges to the expected value:

$$\overline{X_N}=\frac{1}{N}(\sum _{i=1}^N{x}^{(i)})\to \mathbb{E}[X]forN\to \mathrm{\infty }$$