File size: 2,998 Bytes
0293ee2 83c79ff 8a7114c 83c79ff 8a7114c 83c79ff 8a7114c 83c79ff 8a7114c 83c79ff 8a7114c 83c79ff 8a7114c 83c79ff 8a7114c 83c79ff 8a7114c 83c79ff 8a7114c 83c79ff 8a7114c 83c79ff 8a7114c 83c79ff 8a7114c 83c79ff dc6d707 8a7114c 83c79ff 8a7114c 83c79ff | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 | ---
license: apache-2.0
language:
- tr
- en
base_model: SykoSLM/SykoLLM-V5.9-Mini
pipeline_tag: text-generation
library_name: transformers
tags:
- nlp
- code
- phi3
- depth-up-scaling
- untrained
---
# SykoLLM-V6.0-Test
## Model Overview
**SykoLLM-V6.0-Test** is an up-scaled and structurally expanded version of the previous SykoLLM models. Developed by **SykoSLM**, this model is currently in the experimental/testing phase.
The primary objective of this release is to provide a structurally larger foundation model by expanding both the depth (number of layers) and the width (intermediate size / MLP capacity) of the previous architecture, without losing the pre-trained knowledge.
## Architectural Expansion (Up-Scaling)
In order to overcome the "Knowledge Interference" (capacity bottleneck) observed in previous iterations, significant architectural changes have been applied to this model:
* **Depth Up-Scaling (DUS):** The number of hidden layers has been increased to **24**. This was achieved by carefully duplicating and mapping the existing layers to preserve the logical and syntactic capabilities of the model.
* **Width Expansion (MLP Scaling):** The `intermediate_size` has been expanded to **3072**. To prevent catastrophic forgetting, the newly added weights in the feed-forward networks were initialized with exact zero (`0.0`). This ensures that the newly added parameters act as identity functions during the initial forward pass.
## ⚠️ Important Notice: Status of the Model
**This model is currently UNTRAINED on the newly added parameters.**
It has been expanded solely to save pre-training time and preserve existing knowledge. While the model retains the capabilities of its predecessor, the newly added parameters (~100M+ new parameters) are currently dormant (zeroed out).
To fully utilize the expanded capacity and activate the new parameters, **fine-tuning is required**. If used in its current state, the model will function similarly to the previous smaller version, as the new structural capacity has not yet been fine-tuned on new or existing datasets.
## Why This Approach?
Training a Large Language Model from scratch requires immense computational resources and time. By utilizing Net2Net (Knowledge Distillation) principles:
1. We preserve the billions of tokens worth of knowledge already embedded in the model.
2. We provide the model with a much larger "encyclopedic" memory (MLP expansion) to prevent data overlapping and hallucination.
3. We drastically reduce the time required to achieve a higher parameter count.
## Usage
You can load the model using the `transformers` library, but please keep in mind that it requires further fine-tuning for optimal performance.
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "SykoSLM/SykoLLM-V6.0-Test"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto")
```
---
Developed by SykoSLM |