Xeno L2 dilatih dari nol pada 201.366.770.835 token FineWeb di CPU.

from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("Lobus22/Xeno-L2", trust_remote_code=True)
t = AutoTokenizer.from_pretrained("Lobus22/Xeno-L2", trust_remote_code=True)
prompt = t.apply_chat_template([{"role": "user", "content": '"apa itu Xeno?"'}], tokenize=False)
print(t.decode(m.generate(t(prompt, return_tensors="pt").input_ids, max_new_tokens=60)))

Chat template: <//user//>"apa itu Xeno?"<||>X<//assistant//>

Fine-tuning memakai FTEMB (tanpa gradien): python fine_tune_ftemb.py model.safetensors

Batas jujur: ini tabel trigram hashed, bukan transformer saraf. 1.578.507 slot terisi dari 1.073.741.824, jadi kapasitas efektifnya sekitar 1,58 juta konteks. Tidak bisa LoRA, tapi bisa di-FTEMB.

Downloads last month
-
Safetensors
Model size
1B params
Tensor type
I32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support