| --- |
| license: apache-2.0 |
| language: |
| - en |
| pipeline_tag: text-generation |
| tags: |
| - pkai |
| - text-generation |
| - base-model |
| - safetensors |
| --- |
| |
| # PKaI Nano 1.2 |
|
|
| PKaI Nano 1.2 is a 300M-class base language model from PowderKeg |
| Intelligence. It is a compact LLaMA-style decoder model trained from |
| scratch with a Mistral tokenizer and released as a PKaI-native artifact. |
| It succeeds PKaI Nano 1.1 with a broader, higher-quality training mix and |
| improves on it across nearly every benchmark in our evaluation suite. |
|
|
| This repository contains PKaI-native weights and metadata, not a drop-in |
| `transformers.AutoModelForCausalLM` package. |
|
|
| 📖 For the full writeup and evaluation details, see the announcement |
| post: [Introducing PKaI Nano 1.2: Better Data, Stronger Results](https://powderkeg.info/ai-research.html?post=introducing-pkai-nano-1-2). |
|
|
| ## Benchmarks |
|
|
| Evaluated in our standard benchmark suite against the previous Nano |
| releases. Accuracy values are percentages; higher is better. WikiText is |
| word perplexity; lower is better. |
|
|
| | Benchmark | PKaI Nano 1.2 | PKaI Nano 1.1 | PKaI Nano 1 | |
| |---|---|---|---| |
| | HellaSwag | **39.71** | 36.52 | 31.02 | |
| | SciQ | **81.50** | 80.00 | 73.00 | |
| | PIQA | **65.94** | 64.96 | 59.74 | |
| | WinoGrande | **52.96** | 52.72 | 52.57 | |
| | ARC-Easy | **49.92** | 45.45 | 43.35 | |
| | ARC-Challenge | 27.73 | **28.24** | 24.83 | |
| | LAMBADA OpenAI | **44.07** | 33.82 | 23.33 | |
| | WikiText (ppl ↓) | **24.13** | 35.47 | 57.37 | |
|
|
| Across the seven accuracy tasks, PKaI Nano 1.2 averages **51.69**, up from |
| 48.82 for PKaI Nano 1.1 and 43.98 for PKaI Nano 1. See the announcement |
| post for the full writeup and evaluation notes. |
|
|
| <sub>PKaI Nano 1 and PKaI Nano 1.1 were re-scored under our updated |
| evaluation methodology for this comparison; their figures may differ |
| slightly from their original release notes.</sub> |
|
|
| ## Files |
|
|
| - `model.safetensors`: PKaI base model weights. |
| - `config.json`: PKaI model architecture configuration. |
| - `tokenizer.json`: PKaI tokenizer metadata. |
| - `tokenizer.model`: SentencePiece tokenizer model from `mistralai/Mistral-7B-v0.1`. |
| - `THIRD_PARTY_NOTICES.txt`: tokenizer and training-data provenance notices. |
| - `LICENSE`: Apache License, Version 2.0. |
|
|
| ## Architecture |
|
|
| - Parameters: 311,218,176 |
| - Vocabulary size: 32,000 |
| - Context length: 1024 |
| - Layers: 24 |
| - Attention heads: 16 |
| - KV heads: 8 |
| - Embedding size: 1024 |
| - Tied embeddings: yes |
| - QK normalization: yes |
|
|
| ## Training Data |
|
|
| Training data included the following publicly disclosed sources, each |
| processed with best-effort in-house decontamination and deduplication by |
| PowderKeg Intelligence prior to training: |
|
|
| - `HuggingFaceFW/fineweb`, released under the Open Data Commons Attribution |
| License (ODC-By) v1.0 and subject to the Common Crawl Terms of Use as noted by |
| the dataset card. |
| - `HuggingFaceFW/fineweb-edu`, released under the Open Data Commons Attribution |
| License (ODC-By) v1.0 and subject to the Common Crawl Terms of Use as noted by |
| the dataset card. |
| - `HuggingFaceTB/smollm-corpus` (`cosmopedia-v2` subset), released under the Open |
| Data Commons Attribution License (ODC-By) v1.0. Cosmopedia v2 is synthetic text |
| generated with `mistralai/Mixtral-8x7B-Instruct-v0.1`. |
| - `wikimedia/wikipedia` (20231101.en snapshot), released under the Creative |
| Commons Attribution-ShareAlike 4.0 License (CC BY-SA 4.0). |
| - `open-web-math/open-web-math`, released under the Open Data Commons |
| Attribution License (ODC-By) v1.0 and subject to the Common Crawl Terms of |
| Use as noted by the dataset card. |
|
|
| The training mix also included public-domain book text and other public text |
| sources. See `THIRD_PARTY_NOTICES.txt` for source URLs, licenses, and |
| attribution notes. |
|
|
| ## License |
|
|
| Copyright 2026 PowderKeg Intelligence LLC. |
|
|
| The PKaI Nano 1.2 model artifact is released under the Apache License, Version |
| 2.0. The bundled tokenizer and training-data sources have their own provenance |
| and notices listed in `THIRD_PARTY_NOTICES.txt`. |
|
|
| ## Limitations |
|
|
| This is a small base model and has not been instruction-tuned or safety-tuned. |
| It may produce inaccurate, unsafe, biased, or otherwise unsuitable text. Users |
| are responsible for evaluating fitness, safety, and legal compliance for their |
| own use cases. |
|
|
