Skip to content
ML Academy

Introduced on day 32

Cross-entropy loss

The negative log of the probability the model assigns to the true class, averaged over examples; binary cross-entropy is the two-class case.

In detail

With softmax or sigmoid outputs its gradient with respect to the logit is predicted probability minus target, which stays large for confident mistakes. It is the maximum-likelihood loss for categorical or Bernoulli labels and a proper scoring rule. Framework losses usually take logits and apply log-softmax internally.