---
base_model: hmellor/tiny-random-LlamaForCausalLM
library_name: peft
pipeline_tag: text-generation
license: mit
language:
- en
datasets:
- iamholmes/tiny-imdb
tags:
- base_model:adapter:hmellor/tiny-random-LlamaForCausalLM
- transformers
- lora
---

# Tiny Random LLaMA LoRA

A minimal LoRA adapter for [hmellor/tiny-random-LlamaForCausalLM](https://huggingface.co/hmellor/tiny-random-LlamaForCausalLM), useful for smoke testing deployments.

## Model Details

### Model Description

This is a LoRA (Low-Rank Adaptation) adapter trained on a tiny random LLaMA model. The model and adapter are intentionally small and produce random outputs—they are **not** meant for any real inference tasks. The primary purpose is to provide a lightweight adapter for testing deployment pipelines, inference servers, and LoRA loading mechanisms.

- **Model type:** LoRA adapter for causal language modeling
- **Language(s):** English
- **License:** MIT
- **Finetuned from:** [hmellor/tiny-random-LlamaForCausalLM](https://huggingface.co/hmellor/tiny-random-LlamaForCausalLM)

### Model Sources

- **Repository:** [syaffers/tiny-random-llama-lora](https://github.com/syaffers/tiny-random-llama-lora) (training code)

## Uses

### Direct Use

This adapter is intended for:
- Smoke testing LoRA adapter loading in inference pipelines
- Testing deployment configurations with minimal resource usage
- Validating HuggingFace PEFT integration in your infrastructure

### Out-of-Scope Use

This model should **not** be used for:
- Any real text generation or NLP tasks
- Production applications
- Any use case requiring meaningful outputs

## How to Get Started with the Model

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = AutoModelForCausalLM.from_pretrained("hmellor/tiny-random-LlamaForCausalLM")
model = PeftModel.from_pretrained(base_model, "syaffers/tiny-random-llama-lora")
tokenizer = AutoTokenizer.from_pretrained("syaffers/tiny-random-llama-lora")

# Generate (output will be random/meaningless)
inputs = tokenizer("Hello world", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=10)
print(tokenizer.decode(outputs[0]))
```

## Training Details

### Training Data

[iamholmes/tiny-imdb](https://huggingface.co/datasets/iamholmes/tiny-imdb) - A tiny subset of IMDB reviews used for quick training iterations.

### Training Procedure

#### Training Hyperparameters

- **Training regime:** fp32
- **Batch size:** 4
- **Learning rate:** 1e-4
- **Epochs:** 3
- **Warmup steps:** 10
- **Max sequence length:** 128

### LoRA Configuration

| Parameter | Value |
|-----------|-------|
| r (rank) | 8 |
| lora_alpha | 16 |
| target_modules | q_proj, v_proj |
| lora_dropout | 0.05 |
| bias | none |
| task_type | CAUSAL_LM |

## Technical Specifications

### Model Architecture and Objective

LoRA adapter applied to the query and value projection layers of a tiny random LLaMA architecture for causal language modeling.

### Compute Infrastructure

#### Hardware

- Apple M3 Pro (36GB unified memory)
- macOS Sequoia 15.6.1

#### Software

- Transformers
- PEFT 0.18.0
- PyTorch
- Datasets

### Framework versions

- PEFT 0.18.0