| license: mit | |
| language: | |
| - en | |
| tags: | |
| - text-generation | |
| - gpt | |
| - from-scratch | |
| - code | |
| # Cortex 1 Code | |
| A small GPT model for code generation, trained completely from scratch (custom PyTorch architecture, no `transformers` library involved). | |
| - **Parameters**: 36.5m (36,505,728 learnable; the "30m" figure omitted the untied output head) | |
| - **Weights format**: fp32 | |
| - **Architecture**: Decoder-only Transformer (custom implementation) | |
| - **Training**: from scratch, no pretrained weights used | |
| - **Language**: writes Python code, all comments and output in English | |
| > ⚠️ **Note:** the context window is only **512 tokens**, so the model will forget earlier parts of the code quickly. | |
| > ⚠️ **Note:** this is a small model for code, so don't expect great results from it — but it can be fun to try out of curiosity. | |
| ## How to run | |
| 2. Install dependencies: | |
| pip install -r requirements.txt | |
| 3. Run: | |
| python chat.py | |
| ## Limitations | |
| This is a small model, so it may produce broken, incomplete, or non-functional code — expected behavior for this parameter count. |