Also, Label Smoothing which was introduced as part of Rethinking the Inception Architecture for Computer Vision is another way to deal with the problem.ДЇocal Loss is different from the above mentioned solutions. This problem of overconfidence is also highlighted in this excellent paper Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration. The Deep Learning model can actually become overconfident and therefore, the model wouldnвЂ™t generalize well. This in turn can actually impact the performance negatively: In other words, Cross Entropy Loss asks the model to be very confident about the ground truth prediction. Therefore, to reduce the loss, our model would have to predict the ground truth label with a much higher probability.

The Y-axis is simply the loss value given pt.ДЄs can be seen from the image, when the model predicts the ground truth with a probability of 0.6, the C ross Entropy Loss is still somewhere around 0.5. Then pt is 0.4 because ground truth here is 0 and probability that the object is not a bike is 0.4 (1-0.6). The in this case pt is 0.6.ДЄlso, consider the same example but this time the object is not a bike. The X-axis or вЂprobability of ground truth classвЂ™ (letвЂ™s call it pt for simplicity) is the probability that the model predicts for the ground truth object.ДЄs an example, letвЂ™s say the model predicts that something is a bike with probability 0.6 and it actually is a bike. In the graph above, the вЂњblueвЂќ line represents the Cross Entropy Loss. The most important bit to understand about Focal Loss is the graph below: So now that we have seen an example of what Focal Loss can do, letвЂ™s try and understand why that worked. So, why did that work? What did Focal Loss do to make it work? This will help get an intuitive understanding about Focal Loss. This might be a good time to actually analyse the two and observe the differences. OK - so focal loss was introduced in 2017, and is pretty helpful in dealing with class imbalance - great!Д«y the way, here are the predictions of the same model when trained with Focal Loss. Another example, is in the case of Object Detection when most pixels are usually background and only very few pixels inside an image sometimes have the object of interest. Therefore, Focal Loss is particularly useful in cases where there is a class imbalance. This is particularly important when dealing with highly imbalanced datasets because in some cases (such as cancer detection), we really need to model to take a risk and predict something even if the prediction turns out to be a False Positive. In simple words, giving the model a bit more freedom to take some risk when making predictions. Whereasm, what Focal Loss does is that it makes it easier for the model to predict things without being 80-100% sure that this object is вЂњsomethingвЂќ.

The reason why in the image above, the bike is not predicted by the model is because this model was trained using Binary Cross Entropy loss which really asks the model to be confident about what is predicting. See example below where the model doesnвЂ™t predict anything for the motorbike which is of relatively smaller size compared to other images. Focal loss was implemented in Focal Loss for Dense Object Detection paper by He et al.ДЇor years before this paper, Object Detection was actually considered a very difficult problem to solve and it was especially considered very hard to detect small size objects inside images. Where was Focal Loss introduced and what was it used for?Д«efore understanding what Focal Loss is and all the details about it, letвЂ™s first quickly get an intuitive understanding of what Focal Loss actually does. So, why did that work? What did Focal Loss do to make it work?.Where was Focal Loss introduced and what was it used for?.We will also take a dive into the math and implement it in PyTorch. In this blogpost we will understand what Focal Loss and when is it used.

Binary cross entropy