| --- |
| license: mit |
| language: |
| - en |
| pipeline_tag: text-generation |
| library_name: pytorch |
| tags: |
| - transformer |
| - pytorch |
| - from-scratch |
| - linux |
| - seq2seq |
| --- |
| |
| # Linux Command Transformer |
|
|
| A small Transformer Encoder-Decoder trained from scratch with PyTorch to generate Linux commands from natural language descriptions. |
|
|
| This model was developed as a personal learning project to better understand the Transformer architecture, its training process, and its limitations. |
|
|
| The Transformer implementation was guided by [Umar Jamil's Transformer from Scratch walkthrough](https://www.youtube.com/watch?v=ISNdQcPhsts). |
|
|
| ## Model Architecture |
|
|
| The model follows the original Transformer Encoder-Decoder architecture introduced in *Attention Is All You Need*. |
|
|
| | Parameter | Value | |
| |---|---| |
| | Encoder Layers | 2 | |
| | Decoder Layers | 2 | |
| | Model Dimension | 64 | |
| | Attention Heads | 4 | |
| | Feed Forward Dimension | 128 | |
| | Tokenizer | GPT-2 | |
| | Maximum Source Length | 64 | |
| | Maximum Target Length | 96 | |
|
|
| The model was trained with randomly initialized weights. No pretrained language model weights were used. |
|
|
| ## Training |
|
|
| The model was trained on approximately 8,000 natural language–Linux command pairs, using an NVIDIA RTX 4060 Laptop GPU with 8 GB of VRAM. |
|
|
| Training configuration: |
|
|
| - Optimizer: AdamW |
| - Learning rate: 0.001 |
| - Loss: Cross-Entropy |
| - Gradient clipping: 1.0 |
| - Maximum epochs: 10 |
|
|
| The best checkpoint was selected at epoch 6, with a validation loss of **2.0178**. |
|
|
| ## Evaluation Results |
|
|
| The model was evaluated on 796 test examples. |
|
|
| | Metric | Accuracy | |
| |---|---:| |
| | Exact Match | 11.68% | |
| | Command Accuracy | 81.03% | |
| | First Argument Accuracy | 36.35% | |
| | Command + First Argument Accuracy | 35.45% | |
|
|
| The model identifies the correct main Linux utility in many cases, but generating complete commands with accurate arguments remains challenging. |
|
|
| The argument-related metrics were calculated on 773 examples containing at least one argument. Metrics use whitespace-based command splitting rather than full shell parsing. |
|
|
| ## Usage |
|
|
| This is a custom PyTorch architecture and cannot be loaded directly using `AutoModel.from_pretrained()`. |
|
|
| The repository provides: |
|
|
| - `model.pt` — Trained model weights |
| - `config.json` — Architecture configuration |
| - `tokenizer/` — GPT-2 tokenizer configuration with additional special tokens |
|
|
| To run inference, use the implementation provided in the associated GitHub repository. |
|
|
| The inference script downloads the model from Hugging Face, reconstructs the Transformer architecture, loads the weights, and generates Linux commands. |
|
|
| ## Limitations |
|
|
| The model was trained on a relatively small dataset, and training experiments were constrained by the available GPU resources. |
|
|
| Although it achieves 81.03% Command Accuracy, its Exact Match Accuracy is only 11.68%. |
|
|
| This model is intended for educational and experimental purposes. Generated commands should be reviewed before execution, especially commands that modify or delete files. |
|
|
| ## References |
|
|
| - [Attention Is All You Need](https://arxiv.org/abs/1706.03762) |
| - [Umar Jamil — Coding a Transformer from Scratch](https://www.youtube.com/watch?v=ISNdQcPhsts) |
