Bug: uncaught type error in _get_tokenizer_config_size when server omits content length

[dushyantzz]

Hey team! While reviewing the manually_maintained modules, I noticed a potential failure point in tokenizers.py when initializing local tokenizers.
currently, _get_tokenizer_config_size assumes that the content length or x-goog-stored-content-length headers will always be present in the HTTP response. However, if the server responds with Chunked Transfer Encoding, these headers are omitted. When this happens, size defaults to None, and the subsequent int(typing.cast(int, size)) throws a TypeError, causing the tokenizer initialization to fail unexpectedly.

[dushyantzz]

I'd like to submit a quick PR to add a None check before the integer cast, returning a fallback size so the logger can still execute gracefully without crashing the local tokenizer flow. Let me know if you'd like me to open a PR for this.

[fern-support]

Hey @dushyantzz, thanks for the report and for reviewing the code carefully!

One thing worth noting: both call sites for _get_tokenizer_config_size (get_hf_tokenizer and async_get_hf_tokenizer) already wrap the call in a broad try/except Exception that's explicitly designed to swallow failures here — the size logging is best-effort and the tokenizer download proceeds regardless. So in practice, if both headers are absent, the current code logs a warning ("Failed to get the size of the tokenizer config: int() argument must be a string...") rather than crashing tokenizer initialization.

Do you have a reproduction case where the tokenizer actually fails to initialize? That would help us understand if there's a deeper issue we're missing.