Add atomllama-33K-5x5-DigitMesh-sparse model

README.md

@@ -1,3 +1,78 @@
----
-license: apache-2.0
----
+---
+language:
+  - en
+tags:
+  - llama
+  - causal-lm
+  - digit-recognition
+  - sparse-model
+  - sparsegpt
+  - model-compression
+  - 50-percent-sparse
+license: apache-2.0
+base_model: junzzhu/atomllama-33K-5x5-DigitMesh
+library_name: transformers
+pipeline_tag: text-generation
+---
+
+# AtomLlama-33K-5x5-DigitMesh-Sparse
+
+A 50% sparse version of [atomllama-33K-5x5-DigitMesh](https://huggingface.co/junzzhu/atomllama-33K-5x5-DigitMesh) for efficient 5×5 digit mesh recognition.
+
+## Model Description
+
+This is a **~50% unstructured sparse** variant of the AtomLlama-33K-5x5-DigitMesh model, pruned using the SparseGPT algorithm. Half of the model weights have been set to zero while maintaining digit recognition accuracy through second-order optimization.
+
+### Key Features
+
+- **Base Model**: [junzzhu/atomllama-33K-5x5-DigitMesh](https://huggingface.co/junzzhu/atomllama-33K-5x5-DigitMesh)
+- **Sparsity**: ~50% (unstructured)
+- **Pruning Method**: SparseGPT with Hessian-based importance scoring
+- **Parameters**: ~33K total, ~16.5K non-zero
+- **Architecture**: LlamaForCausalLM
+- **Task**: 5×5 binary digit mesh recognition
+
+## Usage
+
+### Serving with vLLM
+
+```bash
+python -m vllm.entrypoints.openai.api_server \
+  --model ./models/atomllama-33K-5x5-DigitMesh-sparse \
+  --max-model-len 32
+```
+
+### Example Inference
+
+```bash
+curl http://localhost:8000/v1/completions \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "model": "./models/atomllama-33K-5x5-DigitMesh-sparse",
+    "prompt": "1 1 1 1 1 1 0 0 0 1 1 0 0 0 1 1 0 0 0 1 1 1 1 1 1 <SEP>",
+    "max_tokens": 1,
+    "temperature": 0
+  }'
+```
+Expected output: `D0`
+
+## Sparsity Details
+
+- **Type**: Unstructured (weights pruned individually based on importance)
+- **Target Sparsity**: 50%
+- **Calibration**: Pruned using digit pattern activations
+- **Benefits**: Reduced memory footprint and potential inference speedup with sparse tensor libraries
+
+## License
+
+Apache-2.0
+
+## Citation
+
+```bibtex
+@misc{atomllama-33k-digitMesh-sparse,
+  title={AtomLlama-33K-5x5-DigitMesh-Sparse: A 50% Sparse Model for Digit Recognition},
+  author={Jun Zhu},
+  year={2026},
+  howpublished={\url{https://huggingface.co/junzzhu/atomllama-33K-5x5-DigitMesh-sparse}}
+}

config.json

@@ -0,0 +1,30 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.3,
+  "bos_token_id": 2,
+  "dtype": "float32",
+  "eos_token_id": 2,
+  "head_dim": 8,
+  "hidden_act": "silu",
+  "hidden_size": 32,
+  "initializer_range": 0.02,
+  "intermediate_size": 128,
+  "max_position_embeddings": 32,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 4,
+  "num_hidden_layers": 2,
+  "num_key_value_heads": 4,
+  "pad_token_id": 13,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": false,
+  "transformers_version": "4.57.3",
+  "use_cache": true,
+  "vocab_size": 14
+}

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 2,
+  "eos_token_id": 2,
+  "pad_token_id": 13,
+  "transformers_version": "4.57.3"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7d19a6d137578604cbe73e5aa3a84729eeec5cfd8b7a86347372d471a0d6bfde
+size 137408

special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "bos_token": {
+    "content": "<SEP>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<SEP>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<PAD>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,51 @@
+{
+  "version": "1.0",
+  "truncation": null,
+  "padding": null,
+  "added_tokens": [
+    {
+      "id": 2,
+      "content": "<SEP>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 13,
+      "content": "<PAD>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    }
+  ],
+  "normalizer": null,
+  "pre_tokenizer": {
+    "type": "WhitespaceSplit"
+  },
+  "post_processor": null,
+  "decoder": null,
+  "model": {
+    "type": "WordLevel",
+    "vocab": {
+      "0": 0,
+      "1": 1,
+      "<SEP>": 2,
+      "D0": 3,
+      "D1": 4,
+      "D2": 5,
+      "D3": 6,
+      "D4": 7,
+      "D5": 8,
+      "D6": 9,
+      "D7": 10,
+      "D8": 11,
+      "D9": 12,
+      "<PAD>": 13
+    },
+    "unk_token": "<unk>"
+  }
+}

tokenizer_config.json

@@ -0,0 +1,27 @@
+{
+  "added_tokens_decoder": {
+    "2": {
+      "content": "<SEP>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "13": {
+      "content": "<PAD>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<SEP>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<SEP>",
+  "extra_special_tokens": {},
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<PAD>",
+  "tokenizer_class": "PreTrainedTokenizerFast"
+}