---
license: apache-2.0
datasets:
- HuggingFaceFW/fineweb
- HuggingFaceFW/fineweb-edu
- bigcode/the-stack
- HuggingFaceFW/finewiki
- TheOneWhoWill/naruto-boruto-wiki
- CausalLM/Refined-Anime-Text
- RyokoAI/Fandom23K
- kurumikz/animepedia
- openai/gsm8k
- HuggingFaceH4/MATH-500
- TIGER-Lab/MMLU-Pro
- tatsu-lab/alpaca
- open-r1/OpenR1-Math-220k
- TIGER-Lab/MathInstruct
- meta-math/MetaMathQA
- open-web-math/open-web-math
- hkust-nlp/dart-math-hard
- EleutherAI/hendrycks_math
language:
- en
library_name: transformers
---
# Shibai 700m Base

This is a 700m parameter LLaMA based model pre-trained with over 18 billion tokens of English text, Math, and Python code at a 2k native context length. It was trained in two stages, first to establish a strong foundational model that can comprehend and generate text, and then to improve its reasoning and coding capabilities with a continued pre-training stage. The use of RoPE scaling allows the model to extrapolate to longer context lengths with the same quality, up to 8k tokens, without any additional training. Although this model may be considered under-trained compared to models such as Qwen 3 0.6B or Qwen 3.5 0.8B, it is still capable of generating coherent and contextually relevant text across a wide range of topics. It should also be noted that this model was trained for 400 hours exclusively on a single RTX 5070 Ti GPU, which is a significant achievement in terms of resource efficiency.

## Model Specs Overview
* Number of parameters: 678.4M
* Hidden size: 1280
* Number of layers: 28
* Number of attention heads: 16 (KV: 4)
* Intermediate size: 4480
* Sequence length: 2048
* RoPE Theta: 10,000
* Vocab size: 32,000

## Getting Started
To use this model, you can load it with the Hugging Face Transformers library. To load the model you can use the following code snippet:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "TheOneWhoWill/Shibai-700M-Base"

processor = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, device_map="auto")
```

Then once the model is in memory you can start generating output with the following code snippet:

```python
prompt = "Once upon a time"
inputs = processor(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
	**inputs,
	max_new_tokens=256,
	do_sample=True,
	temperature=0.3,
	top_p=0.95,
	top_k=50,
	repetition_penalty=1.15,
	no_repeat_ngram_size=3,
)
generated_text = processor.decode(outputs[0], skip_special_tokens=True)
print(generated_text)
```

Keep in mind that this model was pre-trained at a 2k context length and so you can't expect it to generate a stop token. This model works well for simple next-token prediction tasks but isn't fine tuned for instruction following or chat tasks. For those tasks I am planning on releasing a fine-tuned version of this model in the near future.