#DeepLearning #LossFunction

Loss Function

• Mean Squared Error (MSE)
• Cross Entropy Loss

Mean Squared Error (MSE):

$$\text{MSE}=\frac{1}{n}\sum _{i=1}^n(y_i-{\hat{y}}_i{)}^2$$

• Used for regression tasks. Measures the average squared difference between actual values $y_i$ and predicted values ${\hat{y}}_i$.

Mean Absolute Error (MAE):

$$\text{MAE}=\frac{1}{n}\sum _{i=1}^n|y_i-{\hat{y}}_i|$$

• Also used for regression tasks. It measures the average absolute difference between actual and predicted values.

Cross-Entropy Loss (Log Loss):

$$\text{Cross-Entropy}=-\frac{1}{n}\sum _{i=1}^n[y_i\log ({\hat{y}}_i)+(1-y_i)\log (1-{\hat{y}}_i)]$$

• 通常用于处理分类问题

  • $n$ 即为分类总数
  • $y_i$ 即为真实类别概率
  • ${\hat{y}}_i$ 即为对该类别的预测概率

• 通常, $y_i=1$, 因为真实值是确定的，可化简得到

$$\therefore \text{Cross-Entropy}=-\frac{1}{n}\sum _{i=1}^n[\log ({\hat{y}}_i)]$$

[!flex]

Cross Entropy

The derivative of cross entropy

Hinge Loss:

$$\text{Hinge Loss}=\frac{1}{n}\sum _{i=1}^{n}max(0,1-y_i{\hat{y}}_i)$$

• Typically used for "maximum-margin" classifiers like Support Vector Machines (SVM).

Huber Loss:

$$L_{\delta }(a)=\{\begin{array}{ll}\frac{1}{2}{a}^2& \text{for }|a|\le \delta ,\\ \delta (|a|-\frac{1}{2}\delta )& \text{otherwise}.\end{array}$$

• A loss function used in regression that is less sensitive to outliers than MSE.

Kullback-Leibler (KL) Divergence:

$$D_{\text{KL}}(P\parallel Q)=\sum _{i}P(i)\log \left(\frac{P(i)}{Q(i)}\right)$$

• Measures how one probability distribution $P$ diverges from a second probability distribution $Q$. Often used in variational inference or generative models.

Negative Log-Likelihood Loss (NLL):

$$\text{NLL}=-\sum _{i=1}^n{y}_i\log ({\hat{y}}_i)$$

• Used for classification problems, particularly in models like neural networks with a softmax output.