Ajeet Verma de437d0d7a Update: add type hints to check_tokenizers.py (#40094) ...

* Update check_tokenizers.py

chore(typing): add type hints to check_tokenizers script

- Annotate params/returns for helper functions
- Keep tokenizer instances as `Any` to avoid runtime coupling
- Make `check_LTR_mark` return `bool` explicitly (no behavior change)

* Update check_tokenizers.py

chore(typing): replace Any with PreTrainedTokenizerBase in check_tokenizers.py

- Use transformers.tokenization_utils_base.PreTrainedTokenizerBase for `slow` and `fast` params
- Covers both PreTrainedTokenizer and PreTrainedTokenizerFast
- Exposes required methods (encode, decode, encode_plus, tokenize)
- Removes generic Any typing while staying implementation-agnostic

2025-08-15 12:41:28 +00:00

..

distributed

add a network debug script and document it (#15652)

2022-02-15 08:48:00 -08:00

check_tokenizers.py

Update: add type hints to check_tokenizers.py (#40094)

2025-08-15 12:41:28 +00:00

stale.py

[Styling] stylify using ruff (#27144)

2023-11-16 17:43:19 +01:00