how to return hidden_states?

[lss15151161]

use gather_context_logits and gather_generation_logits can help to return logits. but how to return hidden_states? plz help, thx

[brb-nv]

I don't think there's a way to return intermediate hidden_states right now. @hchings, can you please confirm?

[hchings]

I don't think we have flags for that as @brb-nv mentioned in either TRT or PyTorch backend.

@lss15151161 you could share more context on use cases for our team's awareness. Thanks. cc @laikhtewari

[github-actions]

Issue has not received an update in over 14 days. Adding stale label.

[xxchauncey]

same problem, hidden_states for additional lm_head needed.

[FateForever0222]

same problem, do we now support returning hidden states? 🙏 @hchings

[hchings]

Currently, we don't. @FateForever0222 @xxchauncey - is the ask to be able to get the hidden state of any modules/layer, or just the llm_haed?

[CASHADEr]

same problem, I need the last hidden states before logits since I don't need the generate stage