File size: 2,329 Bytes
6cdd078 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 | ---
license: apache-2.0
datasets:
- arxiv_abstracts
language:
- en
pipeline_tag: text-generation
tags:
- tiny
- pico
- scratch
- llama-2
- academic
---
# AbstractsLlama-8M
AbstractsLlama-8M is an ultra-compact, "pico-sized" language model **trained from scratch** by **Pico-Kittens**. It utilizes the **Llama 2 architecture** and is specifically optimized for generating scientific and academic text.
## Model Details
- **Developed by:** Pico-Kittens
- **Model type:** Llama 2-based Causal Language Model
- **Training Status:** Trained from scratch (Not a fine-tune)
- **Parameters:** ~8 Million
- **Language(s):** English
- **License:** apache-2.0
## Training Data
The model was trained on a large-scale collection of **ArXiv abstracts**. The training objective was to compress the structural patterns, technical nomenclature, and "academic tone" of scientific research into a minimal parameter budget.
## Capabilities & Limitations
AbstractsLlama-8M is an experimental model. While it effectively mimics the syntax of research papers, users should be aware of the following:
* **Scientific Syntax:** Highly competent; it excels at producing the "feel" of a formal research proposal or abstract.
* **Architecture:** Implements the Llama 2 transformer block structure at a micro scale.
* **Hallucinations:** Extremely high. The model will invent methodologies, chemical structures, and mathematical frameworks that do not exist.
* **Context:** Limited. It is best suited for short-form generation (under 128 tokens).
---
## Generation Sample
**User:** *We propose*
**AbstractsLlama-8M:**
> We propose a unified framework for modeling large-scale non-linearity of Cancer (NCI) problems with a variable-scale dataset for the linearized dynamics of polynomial conjugal structure. Our key idea of a multi-objective-centile-based model with a fixed, non-preferred variational autoencoder (NMAE) for feature extraction, which includes ax-aware, non-convex optimization formulation for both a single
---
## How to Get Started
```python
import torch
from transformers import pipeline
device = 0 if torch.cuda.is_available() else -1
pipe = pipeline("text-generation", model="PicoKittens/AbstractsLlama-8M", device=device)
output = pipe("We propose", max_new_tokens=100, do_sample=True)
print(output[0]['generated_text']) |