| --- |
| license: cc-by-nc-4.0 |
| library_name: transformers |
| pipeline_tag: text-generation |
| tags: |
| - text-generation |
| - llama |
| - scratch-trained |
| - small-language-model |
| - research-artifact |
| --- |
| |
| # ConeML 348M Beta |
|
|
| ConeML 348M Beta is the second public release in the ConeML research series — a 348M-parameter, scratch-trained small language model. It is the successor to [ConeML 348M Alpha (polish900)](https://huggingface.co/ConeML/coneml-348m-alpha-polish900) and improves on it on the held-out reasoning, code, arithmetic, and calibration evaluations reported below. It is a research artifact and beta candidate, not a polished general assistant. |
|
|
| ## Why ConeML Exists |
|
|
| ConeML is an independent research effort exploring how much capability a compact language model can reach through deliberate data and curriculum design rather than scale alone. The clearest capability carried forward from Alpha is held-out transitive relation binding; Beta extends that capability while improving code generation and arithmetic in the same model. |
|
|
| ## Evaluations vs Alpha |
|
|
| All numbers are held-out probes (fresh entities disjoint from training), measured with the same protocol on both models. |
|
|
| Transitive inference, chat surface, first-choice accuracy, depths 1–5: |
|
|
| | Suite | Alpha | Beta | |
| |---|---|---| |
| | older / younger relation | 79 / 89 / 88 / 77 / 71 | **93 / 91 / 93 / 86 / 82** | |
| | unseen query phrasing | 56 / 73 / 59 / 48 / 34 | **69 / 66 / 67 / 72 / 76** | |
| | non-name entities (colored cards) | 51 / 50 / 41 / 31 / 28 | 62 / 63 / 37 / 34 / 23 (still weak — both) | |
|
|
| Other capabilities: |
|
|
| | Metric | Alpha | Beta | |
| |---|---|---| |
| | Code strict-exec (held-out functions) | 16.7% | **45%** | |
| | Arithmetic, held-out 10-bucket (sympy-checked) | ~21% | **33%** | |
| | Aggregate held-out perplexity | 9.17 | **6.24** | |
| | Calibration ECE (reasoning / code / agentic) | — | 0.037 / 0.032 / 0.015 | |
| | Output format | indentation unstable | clean first-token answers | |
|
|
| Standard public benchmarks (zero-shot, chat format) — reported for comparability, and modest as expected at this scale: |
|
|
| | Benchmark | Beta | |
| |---|---| |
| | GSM8K (300-item sample, exact-match) | 5.0% | |
| | HumanEval (pass@1, 164) | 0% | |
|
|
| These two numbers measure full multi-step / algorithmic problem-solving, which is beyond a 348M model: GSM8K |
| reflects the unsolved multi-digit arithmetic, and HumanEval requires complete algorithmic solutions (the 45% code |
| figure above is held-out *simple* function-body completion — a different and easier task). They are published for |
| transparency, not as strengths. |
|
|
| On these evaluations Beta improves over the Alpha on held-out reasoning, code execution, arithmetic, perplexity, and output formatting. On the older/younger relation suite it is higher at every depth; on unseen-query phrasing it is higher at most depths (the Alpha is slightly higher at depth 2). The Alpha's internal fixed-template probe saturated at 100% (depths 1–3); Beta's held-out template accuracy is 99 / 97 / 95 — effectively equal, on a harder probe. |
|
|
| ## Intended Format |
|
|
| Prompt the model in the chat format below, using the exact `User:` / `Assistant:` markers. Raw completion (without the markers) produces degraded output. The template also ships in `chat_template.jinja` / `tokenizer_config.json`, so `tokenizer.apply_chat_template(...)` works directly. |
|
|
| ```text |
| User: |
| <instruction> |
| Assistant: |
| ``` |
|
|
| ## Loading |
|
|
| ```python |
| import torch |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| repo_id = "ConeML/coneml-348m-beta" |
| tokenizer = AutoTokenizer.from_pretrained(repo_id) |
| model = AutoModelForCausalLM.from_pretrained(repo_id, torch_dtype=torch.float32, device_map="auto") |
| |
| prompt = "User:\nMia is taller than Ben. Ben is taller than Zoe. Who is tallest? Return only the name.\nAssistant:\n" |
| inputs = tokenizer(prompt, return_tensors="pt").to(model.device) |
| out = model.generate(**inputs, max_new_tokens=16, do_sample=False, eos_token_id=tokenizer.eos_token_id) |
| print(tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)) |
| ``` |
|
|
| ## Architecture |
|
|
| - Family: Llama-style decoder · Parameters: ~348M · Layers: 30 · Hidden: 1024 |
| - Attention heads: 8 · KV heads: 2 · Vocab: 32768 · Context length: 512 |
| - Tokenizer: custom 32K |
|
|
| ## Strengths |
|
|
| - Scratch-trained 348M model that improves on its own alpha across the held-out evaluations reported here. |
| - Held-out transitive binding that generalizes across new names, new relations, and unseen query phrasing — higher than the alpha at every depth on the older/younger suite, and at most depths under unseen-query phrasing. |
| - Usable Python function-body generation with stable formatting (45% strict execution on the held-out evaluation reported here). |
| - Materially improved held-out arithmetic over the alpha. |
| - Well-calibrated on reasoning/code/agentic (ECE ≤ 0.04) — uncommon for models this size. |
|
|
| ## Known Limitations |
|
|
| - Multi-digit arithmetic is weak. Held-out 10-bucket arithmetic is 33% overall; reliable 3-digit and multiplication computation is not solved. |
| - Context length is 512 tokens; longer inputs are out of scope for this release. |
| - Transitive binding for non-name entities (e.g., objects) is near chance at depth — binding is still somewhat surface-shaped. |
| - All figures are research results from held-out probes and the standard benchmarks above — not production guarantees. |
| - Research release, not a replacement for larger general assistants. |
|
|
| ## License |
|
|
| Released for non-commercial use under CC BY-NC 4.0. Commercial use is not granted by this release. |
|
|
