DFlash polishing checkpoint weights update
#4
by baranowskiadam - opened
README.md
CHANGED
|
@@ -20,13 +20,14 @@ base_model:
|
|
| 20 |
|
| 21 |
<br>
|
| 22 |
|
| 23 |
-
# Laguna-S-2.1-DFlash
|
| 24 |
|
| 25 |
-
DFlash speculator (
|
| 26 |
[Laguna S 2.1](https://huggingface.co/poolside/Laguna-S-2.1), bf16.
|
| 27 |
|
| 28 |
- Architecture: `DFlashLagunaForCausalLM` (6 sliding-attention layers, block_size 16).
|
| 29 |
- Shares token embedding + lm_head with the target; `draft_vocab_size == vocab_size` (no d2t/t2d).
|
|
|
|
| 30 |
- Loads under vLLM (native `laguna_dflash`) and TRT-LLM (pytorch DFlash backend) as the draft model in a speculative config.
|
| 31 |
|
| 32 |
## Usage
|
|
@@ -50,7 +51,7 @@ python -m sglang.launch_server --model-path poolside/Laguna-S-2.1 \
|
|
| 50 |
|
| 51 |
### llama.cpp (GGUF)
|
| 52 |
|
| 53 |
-
A llama.cpp conversion of this
|
| 54 |
published in
|
| 55 |
[poolside/Laguna-S-2.1-GGUF](https://huggingface.co/poolside/Laguna-S-2.1-GGUF)
|
| 56 |
alongside the target GGUFs. It embeds the target tokenizer and the DFlash metadata
|
|
@@ -64,16 +65,16 @@ cd llama.cpp && cmake -B build && cmake --build build -j
|
|
| 64 |
./build/bin/llama-server \
|
| 65 |
-m laguna-s-2.1-Q4_K_M.gguf \
|
| 66 |
-md laguna-s-2.1-DFlash-BF16.gguf \
|
| 67 |
-
--spec-type draft-dflash --spec-draft-n-max
|
| 68 |
-fa on --jinja --port 8000
|
| 69 |
```
|
| 70 |
|
| 71 |
`--spec-draft-n-max` is clamped to the trained block size (15 draft tokens + 1).
|
| 72 |
|
| 73 |
> [!NOTE]
|
| 74 |
-
> Requires
|
| 75 |
> [`laguna`](https://github.com/poolsideai/llama.cpp/tree/laguna). Upstream
|
| 76 |
> llama.cpp ships the generic DFlash framework but not the Laguna decoder
|
| 77 |
-
> contract this
|
| 78 |
> [ggml-org/llama.cpp#25165](https://github.com/ggml-org/llama.cpp/pull/25165)
|
| 79 |
> covers the target architecture only.
|
|
|
|
| 20 |
|
| 21 |
<br>
|
| 22 |
|
| 23 |
+
# Laguna-S-2.1-DFlash-polishing
|
| 24 |
|
| 25 |
+
DFlash speculator (drafter) for
|
| 26 |
[Laguna S 2.1](https://huggingface.co/poolside/Laguna-S-2.1), bf16.
|
| 27 |
|
| 28 |
- Architecture: `DFlashLagunaForCausalLM` (6 sliding-attention layers, block_size 16).
|
| 29 |
- Shares token embedding + lm_head with the target; `draft_vocab_size == vocab_size` (no d2t/t2d).
|
| 30 |
+
- Source checkpoint: `s3://poolside.titan.checkpoints.us-east-2/adam/checkpoints/ft_sft_e0630_rhiemann_baseline_titan_sft_training/0008400/DFlash_Stage2/25000`
|
| 31 |
- Loads under vLLM (native `laguna_dflash`) and TRT-LLM (pytorch DFlash backend) as the draft model in a speculative config.
|
| 32 |
|
| 33 |
## Usage
|
|
|
|
| 51 |
|
| 52 |
### llama.cpp (GGUF)
|
| 53 |
|
| 54 |
+
A llama.cpp conversion of this drafter (`laguna-s-2.1-DFlash-BF16.gguf`) is
|
| 55 |
published in
|
| 56 |
[poolside/Laguna-S-2.1-GGUF](https://huggingface.co/poolside/Laguna-S-2.1-GGUF)
|
| 57 |
alongside the target GGUFs. It embeds the target tokenizer and the DFlash metadata
|
|
|
|
| 65 |
./build/bin/llama-server \
|
| 66 |
-m laguna-s-2.1-Q4_K_M.gguf \
|
| 67 |
-md laguna-s-2.1-DFlash-BF16.gguf \
|
| 68 |
+
--spec-type draft-dflash --spec-draft-n-max 15 \
|
| 69 |
-fa on --jinja --port 8000
|
| 70 |
```
|
| 71 |
|
| 72 |
`--spec-draft-n-max` is clamped to the trained block size (15 draft tokens + 1).
|
| 73 |
|
| 74 |
> [!NOTE]
|
| 75 |
+
> Requires poolside's llama.cpp fork, branch
|
| 76 |
> [`laguna`](https://github.com/poolsideai/llama.cpp/tree/laguna). Upstream
|
| 77 |
> llama.cpp ships the generic DFlash framework but not the Laguna decoder
|
| 78 |
+
> contract this drafter needs, and upstream PR
|
| 79 |
> [ggml-org/llama.cpp#25165](https://github.com/ggml-org/llama.cpp/pull/25165)
|
| 80 |
> covers the target architecture only.
|