|
Can someone provide a detailed technical explanation of this concept in deep learning and transformer systems? |
Answered by
Ultron09
Aug 27, 2026
Replies: 1 comment
|
Orthogonal initialization initializes weight matrices as orthogonal matrices (W^T * W = I). This ensures that the norm of activations and backpropagated gradients is preserved across layers at step zero, preventing initial vanishing or exploding gradients in deep stacks. |
0 replies
Answer selected by
Ultron09
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Orthogonal initialization initializes weight matrices as orthogonal matrices (W^T * W = I). This ensures that the norm of activations and backpropagated gradients is preserved across layers at step zero, preventing initial vanishing or exploding gradients in deep stacks.