|
|
--- |
|
|
base_model: hmellor/tiny-random-LlamaForCausalLM |
|
|
library_name: peft |
|
|
pipeline_tag: text-generation |
|
|
license: mit |
|
|
language: |
|
|
- en |
|
|
datasets: |
|
|
- iamholmes/tiny-imdb |
|
|
tags: |
|
|
- base_model:adapter:hmellor/tiny-random-LlamaForCausalLM |
|
|
- transformers |
|
|
- lora |
|
|
--- |
|
|
|
|
|
# Tiny Random LLaMA LoRA |
|
|
|
|
|
A minimal LoRA adapter for [hmellor/tiny-random-LlamaForCausalLM](https://huggingface.co/hmellor/tiny-random-LlamaForCausalLM), useful for smoke testing deployments. |
|
|
|
|
|
## Model Details |
|
|
|
|
|
### Model Description |
|
|
|
|
|
This is a LoRA (Low-Rank Adaptation) adapter trained on a tiny random LLaMA model. The model and adapter are intentionally small and produce random outputs—they are **not** meant for any real inference tasks. The primary purpose is to provide a lightweight adapter for testing deployment pipelines, inference servers, and LoRA loading mechanisms. |
|
|
|
|
|
- **Model type:** LoRA adapter for causal language modeling |
|
|
- **Language(s):** English |
|
|
- **License:** MIT |
|
|
- **Finetuned from:** [hmellor/tiny-random-LlamaForCausalLM](https://huggingface.co/hmellor/tiny-random-LlamaForCausalLM) |
|
|
|
|
|
### Model Sources |
|
|
|
|
|
- **Repository:** [syaffers/tiny-random-llama-lora](https://github.com/syaffers/tiny-random-llama-lora) (training code) |
|
|
|
|
|
## Uses |
|
|
|
|
|
### Direct Use |
|
|
|
|
|
This adapter is intended for: |
|
|
- Smoke testing LoRA adapter loading in inference pipelines |
|
|
- Testing deployment configurations with minimal resource usage |
|
|
- Validating HuggingFace PEFT integration in your infrastructure |
|
|
|
|
|
### Out-of-Scope Use |
|
|
|
|
|
This model should **not** be used for: |
|
|
- Any real text generation or NLP tasks |
|
|
- Production applications |
|
|
- Any use case requiring meaningful outputs |
|
|
|
|
|
## How to Get Started with the Model |
|
|
|
|
|
```python |
|
|
from peft import PeftModel |
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
|
|
base_model = AutoModelForCausalLM.from_pretrained("hmellor/tiny-random-LlamaForCausalLM") |
|
|
model = PeftModel.from_pretrained(base_model, "syaffers/tiny-random-llama-lora") |
|
|
tokenizer = AutoTokenizer.from_pretrained("syaffers/tiny-random-llama-lora") |
|
|
|
|
|
# Generate (output will be random/meaningless) |
|
|
inputs = tokenizer("Hello world", return_tensors="pt") |
|
|
outputs = model.generate(**inputs, max_new_tokens=10) |
|
|
print(tokenizer.decode(outputs[0])) |
|
|
``` |
|
|
|
|
|
## Training Details |
|
|
|
|
|
### Training Data |
|
|
|
|
|
[iamholmes/tiny-imdb](https://huggingface.co/datasets/iamholmes/tiny-imdb) - A tiny subset of IMDB reviews used for quick training iterations. |
|
|
|
|
|
### Training Procedure |
|
|
|
|
|
#### Training Hyperparameters |
|
|
|
|
|
- **Training regime:** fp32 |
|
|
- **Batch size:** 4 |
|
|
- **Learning rate:** 1e-4 |
|
|
- **Epochs:** 3 |
|
|
- **Warmup steps:** 10 |
|
|
- **Max sequence length:** 128 |
|
|
|
|
|
### LoRA Configuration |
|
|
|
|
|
| Parameter | Value | |
|
|
|-----------|-------| |
|
|
| r (rank) | 8 | |
|
|
| lora_alpha | 16 | |
|
|
| target_modules | q_proj, v_proj | |
|
|
| lora_dropout | 0.05 | |
|
|
| bias | none | |
|
|
| task_type | CAUSAL_LM | |
|
|
|
|
|
## Technical Specifications |
|
|
|
|
|
### Model Architecture and Objective |
|
|
|
|
|
LoRA adapter applied to the query and value projection layers of a tiny random LLaMA architecture for causal language modeling. |
|
|
|
|
|
### Compute Infrastructure |
|
|
|
|
|
#### Hardware |
|
|
|
|
|
- Apple M3 Pro (36GB unified memory) |
|
|
- macOS Sequoia 15.6.1 |
|
|
|
|
|
#### Software |
|
|
|
|
|
- Transformers |
|
|
- PEFT 0.18.0 |
|
|
- PyTorch |
|
|
- Datasets |
|
|
|
|
|
### Framework versions |
|
|
|
|
|
- PEFT 0.18.0 |
