Instructions to use Aobangaming/Luna-3-230m with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Aobangaming/Luna-3-230m with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Aobangaming/Luna-3-230m", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("Aobangaming/Luna-3-230m", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Aobangaming/Luna-3-230m with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Aobangaming/Luna-3-230m"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Aobangaming/Luna-3-230m",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Aobangaming/Luna-3-230m

SGLang

How to use Aobangaming/Luna-3-230m with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Aobangaming/Luna-3-230m" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Aobangaming/Luna-3-230m",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Aobangaming/Luna-3-230m" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Aobangaming/Luna-3-230m",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Aobangaming/Luna-3-230m with Docker Model Runner:
```
docker model run hf.co/Aobangaming/Luna-3-230m
```

Luna 2 Model Card

Model Information

Model creator: AobanZ

Description

Luna 3 is our next generation text model designed for heavy usage and lengthy conversations. These models have added Llama.cpp support as well as faster response time. Luna has an 8k context window, which supports long chats and roleplays. Luna 3 retains the Lightning architecture, with the only changes being upgraded RMSNorm and RoPE. Luna 3 generates at a relatively fast speed of 130 tokens per second.

Developed by: AobanZ
Model type: Transformer
Language(s) (NLP): English
License: MIT

Model Sources

Repository: https://huggingface.co/Aobangaming/luna-2-70m

Highlights

Faster Response Time With Llama.cpp, Luna generates tokens faster and more efficiently.
Increased Intelligence Luna was sized up from 100m to 230m, which leads to around 100M transformer parameters.

Usage

Luna can be used directly for text generation, experimentation, and conversational interactions. Users can provide text prompts and generate responses using the model's built-in language modeling capabilities. Direct use is primarily intended for casual conversation, research and experimentation. Outputs may be incomplete, inaccurate, repetitive, or unrelated to the input, and should be evaluated before being used for other purposes.

Recommendations

We recommend users of Luna to fine-tune the model on new text, and add necessary guardrails and precautions to prevent misuse.

Starter


import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

REPO = "Aobangaming/Luna-3-230m" 

tok = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForCausalLM.from_pretrained(
    REPO, trust_remote_code=True, dtype=torch.bfloat16
).to("cuda").eval()

prompt = tok.apply_chat_template(
    [{"role": "user", "content": "Hi"}],
    tokenize=False, add_generation_prompt=True,
)
ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to("cuda")

with torch.no_grad():
    out = model.generate(**ids, max_new_tokens=512)

print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True).strip())

Training Details

Training Data

Luna was trained on a subset of the OASST dataset.

Training Procedure

Luna 3 was trained on an RTX 3050 GPU, using FlashAttention/SDPA and MHA. The model was trained on OpenAssistant. Loss ended at around 1.9. Luna 3 is a fork of Luna 2, simply with upgraded sequence length and support.

Training Hyperparameters

Hyperparameter	Value	Comment
Precision	BF16
Optimizer	AdamW
Learning rate	3e-4
Batch size	17

Hardware & Infrastructure

Hardware Type: RTX 3050 6GB
Hours used: 5
Cloud Provider: AobanLabs
Compute Region: Asia

Technical Specifications

Model Architecture and Objective

Luna uses a 12-layer causal Transformer with 768-dimensional hidden states and 12 attention heads, with RMSNorm and RoPE.

Hyperparameter	Value	Comment
Layers	12
D_MODEL	768
Attention Heads	12
Vocabulary	~100k	w/ 8126 Sequence length

Downloads last month: 476

Safetensors

Model size

0.2B params

Tensor type

F32

Datasets used to train Aobangaming/Luna-3-230m

Space using Aobangaming/Luna-3-230m 1

Collection including Aobangaming/Luna-3-230m

Luna 3 Family

Collection

Luna 3 is a collection of lighweights Large Language Models(LLMs) with upgraded sequence length and GGUF. • 2 items • Updated about 22 hours ago