metadata
base_model: meta-llama/Llama-3.2-1B-Instruct
library_name: transformers
license: other
tags:
- llama-factory
- full
- generated_from_trainer
model-index:
- name: ScienceLLaMA-1B
results: []
pipeline_tag: text-generation
ScienceLLaMA-3B
• 🤗 Data
• 🤗 ScienceLLaMA-3B
• 🤗 ScienceLLaMA-1B
• 🐱 Code
• 📃 Paper
This model is a fine-tuned with Logits-Based Finetuning on the JingyaoLi/Science-Logits-1.2M, which integrates the strengths of supervised learning and knowledge distillation by combining teacher logits with ground truth labels. This preserves both correctness and linguistic diversity.
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 1e-06
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- distributed_type: multi-GPU
- num_devices: 8
- gradient_accumulation_steps: 4
- total_train_batch_size: 256
- total_eval_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 1
Training results
Framework versions
- Transformers 4.45.0
- Pytorch 2.4.0+cu121
- Datasets 2.21.0
- Tokenizers 0.20.1