Hugging Face's logo Hugging Face
  • Models
  • Datasets
  • Spaces
  • Buckets new
  • Docs
  • Enterprise
  • Pricing

  • Log In
  • Sign Up

TokenBender
/
test_tokenizer_issue

PEFT
Safetensors
trl
sft
Generated from Trainer
Model card Files Files and versions
xet
Community

Instructions to use TokenBender/test_tokenizer_issue with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

  • Libraries
  • PEFT

    How to use TokenBender/test_tokenizer_issue with PEFT:

    from peft import PeftModel
    from transformers import AutoModelForCausalLM
    
    base_model = AutoModelForCausalLM.from_pretrained("TinyLlama/TinyLlama-1.1B-intermediate-step-480k-1T")
    model = PeftModel.from_pretrained(base_model, "TokenBender/test_tokenizer_issue")
  • Notebooks
  • Google Colab
  • Kaggle
test_tokenizer_issue
1.56 GB
Ctrl+K
Ctrl+K
  • 1 contributor
History: 2 commits
TokenBender's picture
TokenBender
Upload folder using huggingface_hub
8cada09 verified about 2 years ago
  • checkpoint-16
    Upload folder using huggingface_hub about 2 years ago
  • checkpoint-24
    Upload folder using huggingface_hub about 2 years ago
  • checkpoint-8
    Upload folder using huggingface_hub about 2 years ago
  • .gitattributes
    1.52 kB
    initial commit about 2 years ago
  • README.md
    1.34 kB
    Upload folder using huggingface_hub about 2 years ago
  • adapter_config.json
    724 Bytes
    Upload folder using huggingface_hub about 2 years ago
  • adapter_model.safetensors
    313 MB
    xet
    Upload folder using huggingface_hub about 2 years ago
  • special_tokens_map.json
    557 Bytes
    Upload folder using huggingface_hub about 2 years ago
  • tokenizer.json
    1.84 MB
    Upload folder using huggingface_hub about 2 years ago
  • tokenizer_config.json
    1.59 kB
    Upload folder using huggingface_hub about 2 years ago
  • training_args.bin

    Detected Pickle imports (9)

    • "transformers.training_args.TrainingArguments",
    • "torch.device",
    • "transformers.training_args.OptimizerNames",
    • "transformers.trainer_utils.HubStrategy",
    • "transformers.trainer_utils.SchedulerType",
    • "accelerate.state.PartialState",
    • "transformers.trainer_pt_utils.AcceleratorConfig",
    • "transformers.trainer_utils.IntervalStrategy",
    • "accelerate.utils.dataclasses.DistributedType"

    How to fix it?

    4.92 kB
    xet
    Upload folder using huggingface_hub about 2 years ago