---
base_model:
- unsloth/Meta-Llama-3.1-70B-Instruct
library_name: peft
datasets:
- ARM-Development/11k_Tabular
language:
- en
---

## Model Card for `sciencebase-metadata-llama3-70b` *(v 1.0)*

### Model Details
| Field | Value |
|-------|-------|
| **Developed by** | Quan Quy, Travis Ping, Tudor Garbulet, Chirag Shah, Austin Aguilar |
| **Contact** | quyqm@ornl.gov • pingts@ornl.gov • garbuletvt@ornl.gov • shahch@ornl.gov • aguilaral@ornl.gov |
| **Funded by** | U.S. Geological Survey (USGS) & Oak Ridge National Laboratory – ARM Data Center |
| **Model type** | Autoregressive LLM, instruction-tuned for *structured → metadata* generation |
| **Base model** | `meta-llama/Llama-3.1-70B-Instruct` |
| **Languages** | English |
| **Finetuned from** | `unsloth/Meta-Llama-3.1-70B-Instruct` |

### Model Description
Fine-tuned on ≈ 9 000 ScienceBase “data → metadata” pairs to automate creation of FGDC/ISO-style metadata records for scientific datasets.

### Model Sources
| Resource | Link |
|----------|------|
| **Repository** | <https://huggingface.co/ARM-Development/Llama-3.3-70B-tabular-1.0> |
| **Demo** | <https://colab.research.google.com/drive/1saCEFhkBYDhQWkdTwnwiE_-AiWmD6p0f#scrollTo=WeniLP-Ah1QL> |

---

## Uses

### Direct Use
Generate schema-compliant metadata text from a JSON/CSV representation of a ScienceBase item.

### Downstream Use
Integrate as a micro-service in data-repository pipelines.

### Out-of-Scope
Open-ended content generation, or any application outside metadata curation.

---

## Bias, Risks, and Limitations
* Domain-specific bias toward ScienceBase field names.
* Possible hallucination of fields when prompts are underspecified.


---

## Training Details

### Training Data
* ~9 k ScienceBase records with curated metadata.


### Training Procedure
| Hyper-parameter | Value |
|-----------------|-------|
| Max sequence length | 20 000 |
| Precision | fp16 / bf16 (auto) |
| Quantisation | 4-bit QLoRA (`load_in_4bit=True`) |
| LoRA rank / α | 16 / 16 |
| Target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Optimiser | `adamw_8bit` |
| LR / schedule | 2 × 10⁻⁴, linear |
| Epochs | 1 |
| Effective batch | 4 (1 GPU × grad-acc 4) |
| Trainer | `trl` SFTTrainer + `peft` 0.15.2 |

### Hardware & Runtime
| Field | Value |
|-------|-------|
| GPU | 1 × NVIDIA A100 80 GB |
| Total training hours | ~120 hours |
| Cloud/HPC provider | ARM Cumulus HPC |

### Software Stack
| Package | Version |
|---------|---------|
| Python | 3.12.9 |
| PyTorch | 2.6.0 + CUDA 12.4 |
| Transformers | 4.51.3 |
| Accelerate | 1.6.0 |
| PEFT | 0.15.2 |
| Unsloth | 2025.3.19 |
| BitsAndBytes | 0.45.5 |
| TRL | 0.15.2 |
| Xformers | 0.0.29.post3 |
| Datasets | 3.5.0 |
| … |

---

## Evaluation
*Evaluation still in progress.*  



---

## Technical Specifications

### Architecture & Objective
QLoRA-tuned `Llama-3.1-70B-Instruct`; causal-LM objective with structured-to-text instruction prompts.

---

## Model Card Authors
Quan Quy, Travis Ping, Tudor Garbulet, Chirag Shah, Austin Aguilar

---