Module bastionlab.tokenizers.client

Classes

BastionLabTokenizers()

Methods

from_hugging_face_pretrained(self, model_name: str, revision: Optional[str] = 'main', auth_token: Optional[str] = None) ‑> bastionlab.tokenizers.remote_tokenizers.RemoteTokenizer

Loads a Hugging Face tokenizer model with the checkpoint name.

Args: model_name: str Model name. revision: str A branch or commit id auth_token: str, optional, default=None An optional auth token used to access private repositories on the Hugging Face Hub