Fix sparse categorical cross entropy for softmax outputs - #680
Open
dajiaohuang wants to merge 1 commit into
Open
dajiaohuang wants to merge 1 commit into
dajiaohuang wants to merge 1 commit into
Conversation
categorical_cross_entropy/3 ignored the :sparse option when y_pred carried
cached logits metadata produced by Axon.Activations.softmax/1. That metadata
only exists inside a compiled defn, so a softmax-activated model trained with
sparse integer targets silently computed the loss against the dense code path,
yielding an incorrect loss:
y_true = Nx.tensor([1, 0, 2])
logits = Nx.tensor([[15.0, 2.0, -22.0], [-5.0, -2.0, 3.0], [2.0, 1.96, 1.20]])
Axon.Losses.categorical_cross_entropy(y_true, Axon.Activations.softmax(logits), sparse: true)
#=> 28.753393 (expected 7.562242)
The from_logits: true branch already forwarded :sparse to
softmax_cross_entropy_from_logits/3; the logits metadata branch did not. Since
:sparse defaults to false, forwarding it leaves the default behavior
unchanged.
Adds a regression test that exercises the path through a compiled defn, which
is the only context where the logits metadata survives.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Axon.Losses.categorical_cross_entropy/3ignores the:sparseoption wheny_predcarries cached logits metadata fromAxon.Activations.softmax/1. That metadata only exists inside a compileddefn, so a softmax-activated model trained with sparse integer targets silently takes the dense code path and produces a wrong loss:The same computation with
from_logits: truereturns the correct7.562242, because that branch forwards:sparsetosoftmax_cross_entropy_from_logits/3. The logits metadata branch a few lines below does not:Since
softmax_cross_entropy_from_logits/3defaults tosparse: false, the sparse targets are then treated as dense one-hot labels, which is what makes the result wrong rather than raising.This forwards
:sparsein the metadata branch as well. The default is unchanged::sparsedefaults tofalsein bothcategorical_cross_entropy/3andsoftmax_cross_entropy_from_logits/3, so dense callers see identical behavior.The existing sparse tests in
test/axon/losses_test.exscallAxon.Activations.softmax/1eagerly at test scope, where the metadata is lost and the correct branch is taken — which is why this went unnoticed. The regression test added here calls softmax inside a compileddefnso the metadata survives, and checks the sparse and dense forms against the existing7.562242reference value.Validation:
mix test test/axon/losses_test.exs— 39 doctests, 18 tests, 0 failuresmix test— 200 doctests, 738 tests, 4 failures; the 4 failures are pre-existing in this checkout (onegeludoctest float-precision difference and threeAxonTestinspect assertions affected by a CRLF working tree) and are unchanged from before the patchmix format --check-formattedon both changed files — cleanReverting the one-line change makes the new test fail, so it does exercise the reported path.
This is the idiom
Axon.Loop's own documentation suggests for integer class labels (Axon.Loop.trainer/4and the shape-mismatch hint inlib/axon/loop.ex), and it is reached through the normalAxon.Loop.train_step/4path, so a softmax-activated classifier trained on sparse labels silently reports an inflated loss and trains on the wrong gradients.