Upload model

README.md

@@ -0,0 +1,199 @@
+---
+library_name: transformers
+tags: []
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]

config.json

@@ -0,0 +1,17 @@
+{
+  "architectures": [
+    "MBZTestModelForCausalLM"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration.MBZTestConfig",
+    "AutoModelForCausalLM": "modeling.MBZTestModelForCausalLM"
+  },
+  "d_head": 128,
+  "d_model": 4096,
+  "dtype": "float32",
+  "model_type": "mbz-test",
+  "n_heads": 32,
+  "n_layers": 36,
+  "n_vocab": 50257,
+  "transformers_version": "4.56.0"
+}

configuration.py

@@ -0,0 +1,21 @@
+from transformers import PretrainedConfig
+
+class MBZTestConfig(PretrainedConfig):
+    model_type = 'mbz-test'
+
+    def __init__(
+        self,
+        n_layers=36,
+        d_model=4096,
+        n_heads=32,
+        n_vocab=50257,
+        d_head=128,
+        **kwargs
+    ):
+        self.n_layers = n_layers
+        self.d_model = d_model
+        self.n_heads = n_heads
+        self.n_vocab = n_vocab
+        self.d_head = d_head
+
+        super().__init__(**kwargs)

model-00001-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b083a85825b7c07cb1e158bc0e48c47843dfb05b743bdc21ada4d264792aab1f
+size 4984738880

model-00002-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d2a3509e808b4ae2e75fa8cc38b3d01cbe6cce37aae283c9190a8be905e50b4f
+size 4966750152

model-00003-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0bc96857ce845ba7f3586a1243dcc26bafdc349e3800f2282d633490e5955f7e
+size 4832532264

model-00004-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a274ca806522e62ae56b99f44dd9e1e60360e2e8a02e46ca11123e7ecbd4a713
+size 4832532264

model-00005-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ba9ee4bfa6bc20ba52c6471cf3713eee9eea912565b0329a23aa5cd6b52cd94e
+size 4832532264

model-00006-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d6e3084031f7745e1932e4afe4dfd384a44b9c56868e05c1b4d5fbf085c76473
+size 4832532264

model-00007-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a6ab5dea9b901e40d43399e2ea89cabca3fd40eb4ad29ec9dc67074ccf3c7b96
+size 1360614556

model.safetensors.index.json

@@ -0,0 +1,372 @@
+{
+  "metadata": {
+    "total_parameters": 7660549201,
+    "total_size": 30642196804
+  },
+  "weight_map": {
+    "blocks.0.attn.Wk.weight": "model-00001-of-00007.safetensors",
+    "blocks.0.attn.Wo.weight": "model-00001-of-00007.safetensors",
+    "blocks.0.attn.Wq.weight": "model-00001-of-00007.safetensors",
+    "blocks.0.attn.Wv.weight": "model-00001-of-00007.safetensors",
+    "blocks.0.mlp.0.bias": "model-00001-of-00007.safetensors",
+    "blocks.0.mlp.0.weight": "model-00001-of-00007.safetensors",
+    "blocks.0.mlp.2.bias": "model-00001-of-00007.safetensors",
+    "blocks.0.mlp.2.weight": "model-00001-of-00007.safetensors",
+    "blocks.0.norm1.weight": "model-00001-of-00007.safetensors",
+    "blocks.0.norm2.weight": "model-00001-of-00007.safetensors",
+    "blocks.1.attn.Wk.weight": "model-00001-of-00007.safetensors",
+    "blocks.1.attn.Wo.weight": "model-00001-of-00007.safetensors",
+    "blocks.1.attn.Wq.weight": "model-00001-of-00007.safetensors",
+    "blocks.1.attn.Wv.weight": "model-00001-of-00007.safetensors",
+    "blocks.1.mlp.0.bias": "model-00001-of-00007.safetensors",
+    "blocks.1.mlp.0.weight": "model-00001-of-00007.safetensors",
+    "blocks.1.mlp.2.bias": "model-00001-of-00007.safetensors",
+    "blocks.1.mlp.2.weight": "model-00001-of-00007.safetensors",
+    "blocks.1.norm1.weight": "model-00001-of-00007.safetensors",
+    "blocks.1.norm2.weight": "model-00001-of-00007.safetensors",
+    "blocks.10.attn.Wk.weight": "model-00002-of-00007.safetensors",
+    "blocks.10.attn.Wo.weight": "model-00002-of-00007.safetensors",
+    "blocks.10.attn.Wq.weight": "model-00002-of-00007.safetensors",
+    "blocks.10.attn.Wv.weight": "model-00002-of-00007.safetensors",
+    "blocks.10.mlp.0.bias": "model-00002-of-00007.safetensors",
+    "blocks.10.mlp.0.weight": "model-00002-of-00007.safetensors",
+    "blocks.10.mlp.2.bias": "model-00002-of-00007.safetensors",
+    "blocks.10.mlp.2.weight": "model-00002-of-00007.safetensors",
+    "blocks.10.norm1.weight": "model-00002-of-00007.safetensors",
+    "blocks.10.norm2.weight": "model-00002-of-00007.safetensors",
+    "blocks.11.attn.Wk.weight": "model-00002-of-00007.safetensors",
+    "blocks.11.attn.Wo.weight": "model-00002-of-00007.safetensors",
+    "blocks.11.attn.Wq.weight": "model-00002-of-00007.safetensors",
+    "blocks.11.attn.Wv.weight": "model-00002-of-00007.safetensors",
+    "blocks.11.mlp.0.bias": "model-00003-of-00007.safetensors",
+    "blocks.11.mlp.0.weight": "model-00003-of-00007.safetensors",
+    "blocks.11.mlp.2.bias": "model-00003-of-00007.safetensors",
+    "blocks.11.mlp.2.weight": "model-00003-of-00007.safetensors",
+    "blocks.11.norm1.weight": "model-00003-of-00007.safetensors",
+    "blocks.11.norm2.weight": "model-00003-of-00007.safetensors",
+    "blocks.12.attn.Wk.weight": "model-00003-of-00007.safetensors",
+    "blocks.12.attn.Wo.weight": "model-00003-of-00007.safetensors",
+    "blocks.12.attn.Wq.weight": "model-00003-of-00007.safetensors",
+    "blocks.12.attn.Wv.weight": "model-00003-of-00007.safetensors",
+    "blocks.12.mlp.0.bias": "model-00003-of-00007.safetensors",
+    "blocks.12.mlp.0.weight": "model-00003-of-00007.safetensors",
+    "blocks.12.mlp.2.bias": "model-00003-of-00007.safetensors",
+    "blocks.12.mlp.2.weight": "model-00003-of-00007.safetensors",
+    "blocks.12.norm1.weight": "model-00003-of-00007.safetensors",
+    "blocks.12.norm2.weight": "model-00003-of-00007.safetensors",
+    "blocks.13.attn.Wk.weight": "model-00003-of-00007.safetensors",
+    "blocks.13.attn.Wo.weight": "model-00003-of-00007.safetensors",
+    "blocks.13.attn.Wq.weight": "model-00003-of-00007.safetensors",
+    "blocks.13.attn.Wv.weight": "model-00003-of-00007.safetensors",
+    "blocks.13.mlp.0.bias": "model-00003-of-00007.safetensors",
+    "blocks.13.mlp.0.weight": "model-00003-of-00007.safetensors",
+    "blocks.13.mlp.2.bias": "model-00003-of-00007.safetensors",
+    "blocks.13.mlp.2.weight": "model-00003-of-00007.safetensors",
+    "blocks.13.norm1.weight": "model-00003-of-00007.safetensors",
+    "blocks.13.norm2.weight": "model-00003-of-00007.safetensors",
+    "blocks.14.attn.Wk.weight": "model-00003-of-00007.safetensors",
+    "blocks.14.attn.Wo.weight": "model-00003-of-00007.safetensors",
+    "blocks.14.attn.Wq.weight": "model-00003-of-00007.safetensors",
+    "blocks.14.attn.Wv.weight": "model-00003-of-00007.safetensors",
+    "blocks.14.mlp.0.bias": "model-00003-of-00007.safetensors",
+    "blocks.14.mlp.0.weight": "model-00003-of-00007.safetensors",
+    "blocks.14.mlp.2.bias": "model-00003-of-00007.safetensors",
+    "blocks.14.mlp.2.weight": "model-00003-of-00007.safetensors",
+    "blocks.14.norm1.weight": "model-00003-of-00007.safetensors",
+    "blocks.14.norm2.weight": "model-00003-of-00007.safetensors",
+    "blocks.15.attn.Wk.weight": "model-00003-of-00007.safetensors",
+    "blocks.15.attn.Wo.weight": "model-00003-of-00007.safetensors",
+    "blocks.15.attn.Wq.weight": "model-00003-of-00007.safetensors",
+    "blocks.15.attn.Wv.weight": "model-00003-of-00007.safetensors",
+    "blocks.15.mlp.0.bias": "model-00003-of-00007.safetensors",
+    "blocks.15.mlp.0.weight": "model-00003-of-00007.safetensors",
+    "blocks.15.mlp.2.bias": "model-00003-of-00007.safetensors",
+    "blocks.15.mlp.2.weight": "model-00003-of-00007.safetensors",
+    "blocks.15.norm1.weight": "model-00003-of-00007.safetensors",
+    "blocks.15.norm2.weight": "model-00003-of-00007.safetensors",
+    "blocks.16.attn.Wk.weight": "model-00003-of-00007.safetensors",
+    "blocks.16.attn.Wo.weight": "model-00003-of-00007.safetensors",
+    "blocks.16.attn.Wq.weight": "model-00003-of-00007.safetensors",
+    "blocks.16.attn.Wv.weight": "model-00003-of-00007.safetensors",
+    "blocks.16.mlp.0.bias": "model-00003-of-00007.safetensors",
+    "blocks.16.mlp.0.weight": "model-00003-of-00007.safetensors",
+    "blocks.16.mlp.2.bias": "model-00003-of-00007.safetensors",
+    "blocks.16.mlp.2.weight": "model-00003-of-00007.safetensors",
+    "blocks.16.norm1.weight": "model-00003-of-00007.safetensors",
+    "blocks.16.norm2.weight": "model-00003-of-00007.safetensors",
+    "blocks.17.attn.Wk.weight": "model-00003-of-00007.safetensors",
+    "blocks.17.attn.Wo.weight": "model-00003-of-00007.safetensors",
+    "blocks.17.attn.Wq.weight": "model-00003-of-00007.safetensors",
+    "blocks.17.attn.Wv.weight": "model-00003-of-00007.safetensors",
+    "blocks.17.mlp.0.bias": "model-00004-of-00007.safetensors",
+    "blocks.17.mlp.0.weight": "model-00004-of-00007.safetensors",
+    "blocks.17.mlp.2.bias": "model-00004-of-00007.safetensors",
+    "blocks.17.mlp.2.weight": "model-00004-of-00007.safetensors",
+    "blocks.17.norm1.weight": "model-00004-of-00007.safetensors",
+    "blocks.17.norm2.weight": "model-00004-of-00007.safetensors",
+    "blocks.18.attn.Wk.weight": "model-00004-of-00007.safetensors",
+    "blocks.18.attn.Wo.weight": "model-00004-of-00007.safetensors",
+    "blocks.18.attn.Wq.weight": "model-00004-of-00007.safetensors",
+    "blocks.18.attn.Wv.weight": "model-00004-of-00007.safetensors",
+    "blocks.18.mlp.0.bias": "model-00004-of-00007.safetensors",
+    "blocks.18.mlp.0.weight": "model-00004-of-00007.safetensors",
+    "blocks.18.mlp.2.bias": "model-00004-of-00007.safetensors",
+    "blocks.18.mlp.2.weight": "model-00004-of-00007.safetensors",
+    "blocks.18.norm1.weight": "model-00004-of-00007.safetensors",
+    "blocks.18.norm2.weight": "model-00004-of-00007.safetensors",
+    "blocks.19.attn.Wk.weight": "model-00004-of-00007.safetensors",
+    "blocks.19.attn.Wo.weight": "model-00004-of-00007.safetensors",
+    "blocks.19.attn.Wq.weight": "model-00004-of-00007.safetensors",
+    "blocks.19.attn.Wv.weight": "model-00004-of-00007.safetensors",
+    "blocks.19.mlp.0.bias": "model-00004-of-00007.safetensors",
+    "blocks.19.mlp.0.weight": "model-00004-of-00007.safetensors",
+    "blocks.19.mlp.2.bias": "model-00004-of-00007.safetensors",
+    "blocks.19.mlp.2.weight": "model-00004-of-00007.safetensors",
+    "blocks.19.norm1.weight": "model-00004-of-00007.safetensors",
+    "blocks.19.norm2.weight": "model-00004-of-00007.safetensors",
+    "blocks.2.attn.Wk.weight": "model-00001-of-00007.safetensors",
+    "blocks.2.attn.Wo.weight": "model-00001-of-00007.safetensors",
+    "blocks.2.attn.Wq.weight": "model-00001-of-00007.safetensors",
+    "blocks.2.attn.Wv.weight": "model-00001-of-00007.safetensors",
+    "blocks.2.mlp.0.bias": "model-00001-of-00007.safetensors",
+    "blocks.2.mlp.0.weight": "model-00001-of-00007.safetensors",
+    "blocks.2.mlp.2.bias": "model-00001-of-00007.safetensors",
+    "blocks.2.mlp.2.weight": "model-00001-of-00007.safetensors",
+    "blocks.2.norm1.weight": "model-00001-of-00007.safetensors",
+    "blocks.2.norm2.weight": "model-00001-of-00007.safetensors",
+    "blocks.20.attn.Wk.weight": "model-00004-of-00007.safetensors",
+    "blocks.20.attn.Wo.weight": "model-00004-of-00007.safetensors",
+    "blocks.20.attn.Wq.weight": "model-00004-of-00007.safetensors",
+    "blocks.20.attn.Wv.weight": "model-00004-of-00007.safetensors",
+    "blocks.20.mlp.0.bias": "model-00004-of-00007.safetensors",
+    "blocks.20.mlp.0.weight": "model-00004-of-00007.safetensors",
+    "blocks.20.mlp.2.bias": "model-00004-of-00007.safetensors",
+    "blocks.20.mlp.2.weight": "model-00004-of-00007.safetensors",
+    "blocks.20.norm1.weight": "model-00004-of-00007.safetensors",
+    "blocks.20.norm2.weight": "model-00004-of-00007.safetensors",
+    "blocks.21.attn.Wk.weight": "model-00004-of-00007.safetensors",
+    "blocks.21.attn.Wo.weight": "model-00004-of-00007.safetensors",
+    "blocks.21.attn.Wq.weight": "model-00004-of-00007.safetensors",
+    "blocks.21.attn.Wv.weight": "model-00004-of-00007.safetensors",
+    "blocks.21.mlp.0.bias": "model-00004-of-00007.safetensors",
+    "blocks.21.mlp.0.weight": "model-00004-of-00007.safetensors",
+    "blocks.21.mlp.2.bias": "model-00004-of-00007.safetensors",
+    "blocks.21.mlp.2.weight": "model-00004-of-00007.safetensors",
+    "blocks.21.norm1.weight": "model-00004-of-00007.safetensors",
+    "blocks.21.norm2.weight": "model-00004-of-00007.safetensors",
+    "blocks.22.attn.Wk.weight": "model-00004-of-00007.safetensors",
+    "blocks.22.attn.Wo.weight": "model-00004-of-00007.safetensors",
+    "blocks.22.attn.Wq.weight": "model-00004-of-00007.safetensors",
+    "blocks.22.attn.Wv.weight": "model-00004-of-00007.safetensors",
+    "blocks.22.mlp.0.bias": "model-00004-of-00007.safetensors",
+    "blocks.22.mlp.0.weight": "model-00004-of-00007.safetensors",
+    "blocks.22.mlp.2.bias": "model-00004-of-00007.safetensors",
+    "blocks.22.mlp.2.weight": "model-00004-of-00007.safetensors",
+    "blocks.22.norm1.weight": "model-00004-of-00007.safetensors",
+    "blocks.22.norm2.weight": "model-00004-of-00007.safetensors",
+    "blocks.23.attn.Wk.weight": "model-00004-of-00007.safetensors",
+    "blocks.23.attn.Wo.weight": "model-00004-of-00007.safetensors",
+    "blocks.23.attn.Wq.weight": "model-00004-of-00007.safetensors",
+    "blocks.23.attn.Wv.weight": "model-00004-of-00007.safetensors",
+    "blocks.23.mlp.0.bias": "model-00005-of-00007.safetensors",
+    "blocks.23.mlp.0.weight": "model-00005-of-00007.safetensors",
+    "blocks.23.mlp.2.bias": "model-00005-of-00007.safetensors",
+    "blocks.23.mlp.2.weight": "model-00005-of-00007.safetensors",
+    "blocks.23.norm1.weight": "model-00005-of-00007.safetensors",
+    "blocks.23.norm2.weight": "model-00005-of-00007.safetensors",
+    "blocks.24.attn.Wk.weight": "model-00005-of-00007.safetensors",
+    "blocks.24.attn.Wo.weight": "model-00005-of-00007.safetensors",
+    "blocks.24.attn.Wq.weight": "model-00005-of-00007.safetensors",
+    "blocks.24.attn.Wv.weight": "model-00005-of-00007.safetensors",
+    "blocks.24.mlp.0.bias": "model-00005-of-00007.safetensors",
+    "blocks.24.mlp.0.weight": "model-00005-of-00007.safetensors",
+    "blocks.24.mlp.2.bias": "model-00005-of-00007.safetensors",
+    "blocks.24.mlp.2.weight": "model-00005-of-00007.safetensors",
+    "blocks.24.norm1.weight": "model-00005-of-00007.safetensors",
+    "blocks.24.norm2.weight": "model-00005-of-00007.safetensors",
+    "blocks.25.attn.Wk.weight": "model-00005-of-00007.safetensors",
+    "blocks.25.attn.Wo.weight": "model-00005-of-00007.safetensors",
+    "blocks.25.attn.Wq.weight": "model-00005-of-00007.safetensors",
+    "blocks.25.attn.Wv.weight": "model-00005-of-00007.safetensors",
+    "blocks.25.mlp.0.bias": "model-00005-of-00007.safetensors",
+    "blocks.25.mlp.0.weight": "model-00005-of-00007.safetensors",
+    "blocks.25.mlp.2.bias": "model-00005-of-00007.safetensors",
+    "blocks.25.mlp.2.weight": "model-00005-of-00007.safetensors",
+    "blocks.25.norm1.weight": "model-00005-of-00007.safetensors",
+    "blocks.25.norm2.weight": "model-00005-of-00007.safetensors",
+    "blocks.26.attn.Wk.weight": "model-00005-of-00007.safetensors",
+    "blocks.26.attn.Wo.weight": "model-00005-of-00007.safetensors",
+    "blocks.26.attn.Wq.weight": "model-00005-of-00007.safetensors",
+    "blocks.26.attn.Wv.weight": "model-00005-of-00007.safetensors",
+    "blocks.26.mlp.0.bias": "model-00005-of-00007.safetensors",
+    "blocks.26.mlp.0.weight": "model-00005-of-00007.safetensors",
+    "blocks.26.mlp.2.bias": "model-00005-of-00007.safetensors",
+    "blocks.26.mlp.2.weight": "model-00005-of-00007.safetensors",
+    "blocks.26.norm1.weight": "model-00005-of-00007.safetensors",
+    "blocks.26.norm2.weight": "model-00005-of-00007.safetensors",
+    "blocks.27.attn.Wk.weight": "model-00005-of-00007.safetensors",
+    "blocks.27.attn.Wo.weight": "model-00005-of-00007.safetensors",
+    "blocks.27.attn.Wq.weight": "model-00005-of-00007.safetensors",
+    "blocks.27.attn.Wv.weight": "model-00005-of-00007.safetensors",
+    "blocks.27.mlp.0.bias": "model-00005-of-00007.safetensors",
+    "blocks.27.mlp.0.weight": "model-00005-of-00007.safetensors",
+    "blocks.27.mlp.2.bias": "model-00005-of-00007.safetensors",
+    "blocks.27.mlp.2.weight": "model-00005-of-00007.safetensors",
+    "blocks.27.norm1.weight": "model-00005-of-00007.safetensors",
+    "blocks.27.norm2.weight": "model-00005-of-00007.safetensors",
+    "blocks.28.attn.Wk.weight": "model-00005-of-00007.safetensors",
+    "blocks.28.attn.Wo.weight": "model-00005-of-00007.safetensors",
+    "blocks.28.attn.Wq.weight": "model-00005-of-00007.safetensors",
+    "blocks.28.attn.Wv.weight": "model-00005-of-00007.safetensors",
+    "blocks.28.mlp.0.bias": "model-00005-of-00007.safetensors",
+    "blocks.28.mlp.0.weight": "model-00005-of-00007.safetensors",
+    "blocks.28.mlp.2.bias": "model-00005-of-00007.safetensors",
+    "blocks.28.mlp.2.weight": "model-00005-of-00007.safetensors",
+    "blocks.28.norm1.weight": "model-00005-of-00007.safetensors",
+    "blocks.28.norm2.weight": "model-00005-of-00007.safetensors",
+    "blocks.29.attn.Wk.weight": "model-00005-of-00007.safetensors",
+    "blocks.29.attn.Wo.weight": "model-00005-of-00007.safetensors",
+    "blocks.29.attn.Wq.weight": "model-00005-of-00007.safetensors",
+    "blocks.29.attn.Wv.weight": "model-00005-of-00007.safetensors",
+    "blocks.29.mlp.0.bias": "model-00006-of-00007.safetensors",
+    "blocks.29.mlp.0.weight": "model-00006-of-00007.safetensors",
+    "blocks.29.mlp.2.bias": "model-00006-of-00007.safetensors",
+    "blocks.29.mlp.2.weight": "model-00006-of-00007.safetensors",
+    "blocks.29.norm1.weight": "model-00006-of-00007.safetensors",
+    "blocks.29.norm2.weight": "model-00006-of-00007.safetensors",
+    "blocks.3.attn.Wk.weight": "model-00001-of-00007.safetensors",
+    "blocks.3.attn.Wo.weight": "model-00001-of-00007.safetensors",
+    "blocks.3.attn.Wq.weight": "model-00001-of-00007.safetensors",
+    "blocks.3.attn.Wv.weight": "model-00001-of-00007.safetensors",
+    "blocks.3.mlp.0.bias": "model-00001-of-00007.safetensors",
+    "blocks.3.mlp.0.weight": "model-00001-of-00007.safetensors",
+    "blocks.3.mlp.2.bias": "model-00001-of-00007.safetensors",
+    "blocks.3.mlp.2.weight": "model-00001-of-00007.safetensors",
+    "blocks.3.norm1.weight": "model-00001-of-00007.safetensors",
+    "blocks.3.norm2.weight": "model-00001-of-00007.safetensors",
+    "blocks.30.attn.Wk.weight": "model-00006-of-00007.safetensors",
+    "blocks.30.attn.Wo.weight": "model-00006-of-00007.safetensors",
+    "blocks.30.attn.Wq.weight": "model-00006-of-00007.safetensors",
+    "blocks.30.attn.Wv.weight": "model-00006-of-00007.safetensors",
+    "blocks.30.mlp.0.bias": "model-00006-of-00007.safetensors",
+    "blocks.30.mlp.0.weight": "model-00006-of-00007.safetensors",
+    "blocks.30.mlp.2.bias": "model-00006-of-00007.safetensors",
+    "blocks.30.mlp.2.weight": "model-00006-of-00007.safetensors",
+    "blocks.30.norm1.weight": "model-00006-of-00007.safetensors",
+    "blocks.30.norm2.weight": "model-00006-of-00007.safetensors",
+    "blocks.31.attn.Wk.weight": "model-00006-of-00007.safetensors",
+    "blocks.31.attn.Wo.weight": "model-00006-of-00007.safetensors",
+    "blocks.31.attn.Wq.weight": "model-00006-of-00007.safetensors",
+    "blocks.31.attn.Wv.weight": "model-00006-of-00007.safetensors",
+    "blocks.31.mlp.0.bias": "model-00006-of-00007.safetensors",
+    "blocks.31.mlp.0.weight": "model-00006-of-00007.safetensors",
+    "blocks.31.mlp.2.bias": "model-00006-of-00007.safetensors",
+    "blocks.31.mlp.2.weight": "model-00006-of-00007.safetensors",
+    "blocks.31.norm1.weight": "model-00006-of-00007.safetensors",
+    "blocks.31.norm2.weight": "model-00006-of-00007.safetensors",
+    "blocks.32.attn.Wk.weight": "model-00006-of-00007.safetensors",
+    "blocks.32.attn.Wo.weight": "model-00006-of-00007.safetensors",
+    "blocks.32.attn.Wq.weight": "model-00006-of-00007.safetensors",
+    "blocks.32.attn.Wv.weight": "model-00006-of-00007.safetensors",
+    "blocks.32.mlp.0.bias": "model-00006-of-00007.safetensors",
+    "blocks.32.mlp.0.weight": "model-00006-of-00007.safetensors",
+    "blocks.32.mlp.2.bias": "model-00006-of-00007.safetensors",
+    "blocks.32.mlp.2.weight": "model-00006-of-00007.safetensors",
+    "blocks.32.norm1.weight": "model-00006-of-00007.safetensors",
+    "blocks.32.norm2.weight": "model-00006-of-00007.safetensors",
+    "blocks.33.attn.Wk.weight": "model-00006-of-00007.safetensors",
+    "blocks.33.attn.Wo.weight": "model-00006-of-00007.safetensors",
+    "blocks.33.attn.Wq.weight": "model-00006-of-00007.safetensors",
+    "blocks.33.attn.Wv.weight": "model-00006-of-00007.safetensors",
+    "blocks.33.mlp.0.bias": "model-00006-of-00007.safetensors",
+    "blocks.33.mlp.0.weight": "model-00006-of-00007.safetensors",
+    "blocks.33.mlp.2.bias": "model-00006-of-00007.safetensors",
+    "blocks.33.mlp.2.weight": "model-00006-of-00007.safetensors",
+    "blocks.33.norm1.weight": "model-00006-of-00007.safetensors",
+    "blocks.33.norm2.weight": "model-00006-of-00007.safetensors",
+    "blocks.34.attn.Wk.weight": "model-00006-of-00007.safetensors",
+    "blocks.34.attn.Wo.weight": "model-00006-of-00007.safetensors",
+    "blocks.34.attn.Wq.weight": "model-00006-of-00007.safetensors",
+    "blocks.34.attn.Wv.weight": "model-00006-of-00007.safetensors",
+    "blocks.34.mlp.0.bias": "model-00006-of-00007.safetensors",
+    "blocks.34.mlp.0.weight": "model-00006-of-00007.safetensors",
+    "blocks.34.mlp.2.bias": "model-00006-of-00007.safetensors",
+    "blocks.34.mlp.2.weight": "model-00006-of-00007.safetensors",
+    "blocks.34.norm1.weight": "model-00006-of-00007.safetensors",
+    "blocks.34.norm2.weight": "model-00006-of-00007.safetensors",
+    "blocks.35.attn.Wk.weight": "model-00006-of-00007.safetensors",
+    "blocks.35.attn.Wo.weight": "model-00006-of-00007.safetensors",
+    "blocks.35.attn.Wq.weight": "model-00006-of-00007.safetensors",
+    "blocks.35.attn.Wv.weight": "model-00006-of-00007.safetensors",
+    "blocks.35.mlp.0.bias": "model-00007-of-00007.safetensors",
+    "blocks.35.mlp.0.weight": "model-00007-of-00007.safetensors",
+    "blocks.35.mlp.2.bias": "model-00007-of-00007.safetensors",
+    "blocks.35.mlp.2.weight": "model-00007-of-00007.safetensors",
+    "blocks.35.norm1.weight": "model-00007-of-00007.safetensors",
+    "blocks.35.norm2.weight": "model-00007-of-00007.safetensors",
+    "blocks.4.attn.Wk.weight": "model-00001-of-00007.safetensors",
+    "blocks.4.attn.Wo.weight": "model-00001-of-00007.safetensors",
+    "blocks.4.attn.Wq.weight": "model-00001-of-00007.safetensors",
+    "blocks.4.attn.Wv.weight": "model-00001-of-00007.safetensors",
+    "blocks.4.mlp.0.bias": "model-00001-of-00007.safetensors",
+    "blocks.4.mlp.0.weight": "model-00001-of-00007.safetensors",
+    "blocks.4.mlp.2.bias": "model-00001-of-00007.safetensors",
+    "blocks.4.mlp.2.weight": "model-00001-of-00007.safetensors",
+    "blocks.4.norm1.weight": "model-00001-of-00007.safetensors",
+    "blocks.4.norm2.weight": "model-00001-of-00007.safetensors",
+    "blocks.5.attn.Wk.weight": "model-00001-of-00007.safetensors",
+    "blocks.5.attn.Wo.weight": "model-00002-of-00007.safetensors",
+    "blocks.5.attn.Wq.weight": "model-00001-of-00007.safetensors",
+    "blocks.5.attn.Wv.weight": "model-00002-of-00007.safetensors",
+    "blocks.5.mlp.0.bias": "model-00002-of-00007.safetensors",
+    "blocks.5.mlp.0.weight": "model-00002-of-00007.safetensors",
+    "blocks.5.mlp.2.bias": "model-00002-of-00007.safetensors",
+    "blocks.5.mlp.2.weight": "model-00002-of-00007.safetensors",
+    "blocks.5.norm1.weight": "model-00002-of-00007.safetensors",
+    "blocks.5.norm2.weight": "model-00002-of-00007.safetensors",
+    "blocks.6.attn.Wk.weight": "model-00002-of-00007.safetensors",
+    "blocks.6.attn.Wo.weight": "model-00002-of-00007.safetensors",
+    "blocks.6.attn.Wq.weight": "model-00002-of-00007.safetensors",
+    "blocks.6.attn.Wv.weight": "model-00002-of-00007.safetensors",
+    "blocks.6.mlp.0.bias": "model-00002-of-00007.safetensors",
+    "blocks.6.mlp.0.weight": "model-00002-of-00007.safetensors",
+    "blocks.6.mlp.2.bias": "model-00002-of-00007.safetensors",
+    "blocks.6.mlp.2.weight": "model-00002-of-00007.safetensors",
+    "blocks.6.norm1.weight": "model-00002-of-00007.safetensors",
+    "blocks.6.norm2.weight": "model-00002-of-00007.safetensors",
+    "blocks.7.attn.Wk.weight": "model-00002-of-00007.safetensors",
+    "blocks.7.attn.Wo.weight": "model-00002-of-00007.safetensors",
+    "blocks.7.attn.Wq.weight": "model-00002-of-00007.safetensors",
+    "blocks.7.attn.Wv.weight": "model-00002-of-00007.safetensors",
+    "blocks.7.mlp.0.bias": "model-00002-of-00007.safetensors",
+    "blocks.7.mlp.0.weight": "model-00002-of-00007.safetensors",
+    "blocks.7.mlp.2.bias": "model-00002-of-00007.safetensors",
+    "blocks.7.mlp.2.weight": "model-00002-of-00007.safetensors",
+    "blocks.7.norm1.weight": "model-00002-of-00007.safetensors",
+    "blocks.7.norm2.weight": "model-00002-of-00007.safetensors",
+    "blocks.8.attn.Wk.weight": "model-00002-of-00007.safetensors",
+    "blocks.8.attn.Wo.weight": "model-00002-of-00007.safetensors",
+    "blocks.8.attn.Wq.weight": "model-00002-of-00007.safetensors",
+    "blocks.8.attn.Wv.weight": "model-00002-of-00007.safetensors",
+    "blocks.8.mlp.0.bias": "model-00002-of-00007.safetensors",
+    "blocks.8.mlp.0.weight": "model-00002-of-00007.safetensors",
+    "blocks.8.mlp.2.bias": "model-00002-of-00007.safetensors",
+    "blocks.8.mlp.2.weight": "model-00002-of-00007.safetensors",
+    "blocks.8.norm1.weight": "model-00002-of-00007.safetensors",
+    "blocks.8.norm2.weight": "model-00002-of-00007.safetensors",
+    "blocks.9.attn.Wk.weight": "model-00002-of-00007.safetensors",
+    "blocks.9.attn.Wo.weight": "model-00002-of-00007.safetensors",
+    "blocks.9.attn.Wq.weight": "model-00002-of-00007.safetensors",
+    "blocks.9.attn.Wv.weight": "model-00002-of-00007.safetensors",
+    "blocks.9.mlp.0.bias": "model-00002-of-00007.safetensors",
+    "blocks.9.mlp.0.weight": "model-00002-of-00007.safetensors",
+    "blocks.9.mlp.2.bias": "model-00002-of-00007.safetensors",
+    "blocks.9.mlp.2.weight": "model-00002-of-00007.safetensors",
+    "blocks.9.norm1.weight": "model-00002-of-00007.safetensors",
+    "blocks.9.norm2.weight": "model-00002-of-00007.safetensors",
+    "embed.weight": "model-00001-of-00007.safetensors",
+    "norm.weight": "model-00007-of-00007.safetensors",
+    "out_head.bias": "model-00007-of-00007.safetensors",
+    "out_head.weight": "model-00007-of-00007.safetensors"
+  }
+}

modeling.py

@@ -0,0 +1,148 @@
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from torch.nn.attention import sdpa_kernel, SDPBackend
+from transformers import PreTrainedModel
+from .configuration import MBZTestConfig
+from transformers.modeling_outputs import CausalLMOutput
+
+class RotaryPositionalEncoding(nn.Module):
+    """
+    Rotary Position Embeddings (RoPE) - efficient implementation
+    """
+    def __init__(self, d_head, max_seq_len=8192, base=10000.0):
+        super().__init__()
+        self.d_head = d_head
+        self.max_seq_len = max_seq_len
+        self.base = base
+
+        # Precompute inverse frequencies
+        inv_freq = 1.0 / (base ** (torch.arange(0, d_head, 2).float() / d_head))
+        self.register_buffer('inv_freq', inv_freq, persistent=False)
+
+        # Precompute cos and sin for maximum sequence length
+        self._precompute_freqs(max_seq_len)
+
+    def _precompute_freqs(self, seq_len):
+        """Precompute cos and sin values for positions"""
+        t = torch.arange(seq_len, dtype=self.inv_freq.dtype, device=self.inv_freq.device)
+        freqs = torch.outer(t, self.inv_freq)  # (seq_len, d_head/2)
+
+        # Create cos and sin embeddings
+        freqs_cos = torch.cos(freqs)
+        freqs_sin = torch.sin(freqs)
+
+        # Interleave to match the dimension (seq_len, d_head)
+        self.register_buffer('freqs_cos', freqs_cos.repeat_interleave(2, dim=-1), persistent=False)
+        self.register_buffer('freqs_sin', freqs_sin.repeat_interleave(2, dim=-1), persistent=False)
+
+    def rotate_half(self, x):
+        """Rotate half the hidden dims of the input"""
+        x1 = x[..., ::2]
+        x2 = x[..., 1::2]
+        return torch.stack([-x2, x1], dim=-1).flatten(-2)
+
+    def forward(self, q, k, start_pos=0):
+        """
+        Apply rotary embeddings to query and key tensors
+        Args:
+            q: (batch_size, n_heads, seq_len, d_head)
+            k: (batch_size, n_heads, seq_len, d_head)
+            start_pos: starting position for caching scenarios
+        Returns:
+            q_rot, k_rot with rotary embeddings applied
+        """
+        seq_len = q.shape[2]
+
+        # Get the precomputed frequencies for this sequence length
+        freqs_cos = self.freqs_cos[start_pos:start_pos + seq_len]
+        freqs_sin = self.freqs_sin[start_pos:start_pos + seq_len]
+
+        # Apply rotary embeddings
+        q_rot = q * freqs_cos + self.rotate_half(q) * freqs_sin
+        k_rot = k * freqs_cos + self.rotate_half(k) * freqs_sin
+
+        return q_rot, k_rot
+
+class Attention(nn.Module):
+    def __init__(self, d_model, n_heads, d_head):
+        super().__init__()
+        self.d_model = d_model
+        self.n_heads = n_heads
+        self.d_head = d_head
+
+        self.Wq = nn.Linear(d_model, n_heads * d_head, bias=False)
+        self.Wk = nn.Linear(d_model, n_heads * d_head, bias=False)
+        self.Wv = nn.Linear(d_model, n_heads * d_head, bias=False)
+        self.Wo = nn.Linear(n_heads * d_head, d_model, bias=False)
+
+        # Initialize RoPE
+        self.rope = RotaryPositionalEncoding(d_head)
+
+    def forward(self, x):
+        # x is shape batch_size, seq_len, d_model
+        batch_size, seq_len, d_model = x.shape
+        q = self.Wq(x) # q is shape batch_size, seq_len, n_heads * d_head
+        k = self.Wk(x)
+        v = self.Wv(x)
+
+        # reshape to batch_size, n_heads, seq_len, d_head
+        q = q.reshape(batch_size, seq_len, self.n_heads, self.d_head).transpose(1,2)
+        k = k.reshape(batch_size, seq_len, self.n_heads, self.d_head).transpose(1,2)
+        v = v.reshape(batch_size, seq_len, self.n_heads, self.d_head).transpose(1,2)
+
+        q, k = self.rope(q, k)
+        with sdpa_kernel(SDPBackend.FLASH_ATTENTION): # ensure use flash attention
+            a = F.scaled_dot_product_attention(q, k, v, attn_mask=None, is_causal=True)# a is (batch_size, n_heads, seq_len, d_head)
+        a = a.transpose(1,2) # change a to (batch_size, seq_len, n_heads, d_head)
+        a = a.reshape(batch_size, seq_len, self.n_heads * self.d_head)
+        out = self.Wo(a) # out is (batch_size, seq_len, d_model)
+        return out
+
+class TransformerBlock(nn.Module):
+    def __init__(self, d_model, n_heads, d_head):
+        super().__init__()
+        self.d_model = d_model
+        self.n_heads = n_heads
+        self.d_head = d_head
+
+        self.attn = Attention(d_model, n_heads, d_head)
+        self.mlp = nn.Sequential(nn.Linear(d_model, 4*d_model), nn.ReLU(), nn.Linear(4*d_model, d_model))
+
+        self.norm1 = nn.RMSNorm(d_model)
+        self.norm2 = nn.RMSNorm(d_model)
+
+    def forward(self, x):
+        x = self.attn(self.norm1(x)) + x
+        x = self.mlp(self.norm2(x)) + x
+        return x
+
+class MBZTestModelForCausalLM(PreTrainedModel):
+    config_class = MBZTestConfig
+
+    def __init__(self, config):
+        super().__init__(config)
+        d_model = config.d_model
+        n_heads = config.n_heads
+        d_head = config.d_head
+        n_vocab = config.n_vocab
+        n_layers = config.n_layers
+
+        self.d_model = d_model
+        self.n_heads = n_heads
+        self.d_head = d_head
+        self.n_vocab = n_vocab
+
+        self.embed = nn.Embedding(n_vocab, d_model)
+
+        self.blocks = nn.ModuleList([TransformerBlock(d_model, n_heads, d_head) for _ in range(n_layers)])
+
+        self.norm = nn.RMSNorm(d_model)
+        self.out_head = nn.Linear(d_model, n_vocab)
+
+    def forward(self, x):
+        x = self.embed(x)
+        for block in self.blocks:
+            x = block(x)
+        x = self.out_head(self.norm(x))
+        return CausalLMOutput(logits=x)