---
base_model: ByteDance-Seed/Seed-OSS-36B-Instruct
model_creator: ByteDance-Seed
model_name: Seed-OSS-36B-Instruct
quantized_by: Second State Inc.
pipeline_tag: text-generation
library_name: transformers
---

<!-- header start -->
<!-- 200823 -->
<div style="width: auto; margin-left: auto; margin-right: auto">
<img src="https://github.com/LlamaEdge/LlamaEdge/raw/dev/assets/logo.svg" style="width: 100%; min-width: 400px; display: block; margin: auto;">
</div>
<hr style="margin-top: 1.0em; margin-bottom: 1.0em;">
<!-- header end -->

# Seed-OSS-36B-Instruct-GGUF

## Original Model

[ByteDance-Seed/Seed-OSS-36B-Instruct](https://huggingface.co/ByteDance-Seed/Seed-OSS-36B-Instruct)

## Run with LlamaEdge

- LlamaEdge version: coming soon

<!-- - LlamaEdge version: [v0.25.1](https://github.com/LlamaEdge/LlamaEdge/releases/tag/0.25.1) and above -->

- Prompt template

  - Prompt type:
 
    - `seed-oss-think` for think mode
    - `seed-oss-no-think` for no think mode

  - Prompt string
    - `Thinking` mode

      ```text
      <seed:bos>system
      You are Doubao, a helpful AI assistant.
      <seed:eos>

      <seed:bos>user
      {user_message_1}
      <seed:eos>

      <seed:bos>assistant
      <seed:think>{thinking_content}</seed:think>
      {assistant_message_1}
      <seed:eos>

      <seed:bos>user
      {user_message_2}
      <seed:eos>

      <seed:bos>assistant
      ```

    - `No-thinking` mode

      ```text
      <seed:bos>system
      You are Doubao, a helpful AI assistant.
      <seed:eos>

      <seed:bos>system
      You are an intelligent assistant that can answer questions in one step without the need for reasoning and thinking, that is, your thinking budget is 0. Next, please skip       the thinking process and directly start answering the user's questions.
      <seed:eos>

      <seed:bos>user
      {user_message_1}
      <seed:eos>

      <seed:bos>assistant
      {assistant_message_1}
      <seed:eos>

      <seed:bos>user
      {user_message_2}
      <seed:eos>

      <seed:bos>assistant
      ```

- Context size: `512000`

- Run as LlamaEdge service

  ```bash
  wasmedge --dir .:. \
    --nn-preload default:GGML:AUTO:Seed-OSS-36B-Instruct-Q5_K_M.gguf \
    llama-api-server.wasm \
    --prompt-template seed-oss-no-think \
    --ctx-size 512000 \
    --model-name seed-oss
  ```

## Quantized GGUF Models

| Name | Quant method | Bits | Size | Use case |
| ---- | ---- | ---- | ---- | ----- |
| [Seed-OSS-36B-Instruct-Q2_K.gguf](https://huggingface.co/second-state/Seed-OSS-36B-Instruct-GGUF/blob/main/Seed-OSS-36B-Instruct-Q2_K.gguf)     | Q2_K   | 2 | 13.6 GB| smallest, significant quality loss - not recommended for most purposes |
| [Seed-OSS-36B-Instruct-Q3_K_L.gguf](https://huggingface.co/second-state/Seed-OSS-36B-Instruct-GGUF/blob/main/Seed-OSS-36B-Instruct-Q3_K_L.gguf) | Q3_K_L | 3 | 19.1 GB| small, substantial quality loss |
| [Seed-OSS-36B-Instruct-Q3_K_M.gguf](https://huggingface.co/second-state/Seed-OSS-36B-Instruct-GGUF/blob/main/Seed-OSS-36B-Instruct-Q3_K_M.gguf) | Q3_K_M | 3 | 17.6 GB| very small, high quality loss |
| [Seed-OSS-36B-Instruct-Q3_K_S.gguf](https://huggingface.co/second-state/Seed-OSS-36B-Instruct-GGUF/blob/main/Seed-OSS-36B-Instruct-Q3_K_S.gguf) | Q3_K_S | 3 | 15.9 GB| very small, high quality loss |
| [Seed-OSS-36B-Instruct-Q4_0.gguf](https://huggingface.co/second-state/Seed-OSS-36B-Instruct-GGUF/blob/main/Seed-OSS-36B-Instruct-Q4_0.gguf)     | Q4_0   | 4 | 20.6 GB| legacy; small, very high quality loss - prefer using Q3_K_M |
| [Seed-OSS-36B-Instruct-Q4_K_M.gguf](https://huggingface.co/second-state/Seed-OSS-36B-Instruct-GGUF/blob/main/Seed-OSS-36B-Instruct-Q4_K_M.gguf) | Q4_K_M | 4 | 21.8 GB| medium, balanced quality - recommended |
| [Seed-OSS-36B-Instruct-Q4_K_S.gguf](https://huggingface.co/second-state/Seed-OSS-36B-Instruct-GGUF/blob/main/Seed-OSS-36B-Instruct-Q4_K_S.gguf) | Q4_K_S | 4 | 20.7 GB| small, greater quality loss |
| [Seed-OSS-36B-Instruct-Q5_0.gguf](https://huggingface.co/second-state/Seed-OSS-36B-Instruct-GGUF/blob/main/Seed-OSS-36B-Instruct-Q5_0.gguf)     | Q5_0   | 5 | 25.0 GB| legacy; medium, balanced quality - prefer using Q4_K_M |
| [Seed-OSS-36B-Instruct-Q5_K_M.gguf](https://huggingface.co/second-state/Seed-OSS-36B-Instruct-GGUF/blob/main/Seed-OSS-36B-Instruct-Q5_K_M.gguf) | Q5_K_M | 5 | 25.6 GB| large, very low quality loss - recommended |
| [Seed-OSS-36B-Instruct-Q5_K_S.gguf](https://huggingface.co/second-state/Seed-OSS-36B-Instruct-GGUF/blob/main/Seed-OSS-36B-Instruct-Q5_K_S.gguf) | Q5_K_S | 5 | 25.0 GB| large, low quality loss - recommended |
| [Seed-OSS-36B-Instruct-Q6_K.gguf](https://huggingface.co/second-state/Seed-OSS-36B-Instruct-GGUF/blob/main/Seed-OSS-36B-Instruct-Q6_K.gguf)     | Q6_K   | 6 | 29.7 GB| very large, extremely low quality loss |
| [Seed-OSS-36B-Instruct-Q8_0.gguf](https://huggingface.co/second-state/Seed-OSS-36B-Instruct-GGUF/blob/main/Seed-OSS-36B-Instruct-Q8_0.gguf)     | Q8_0   | 8 | 38.4 GB| very large, extremely low quality loss - not recommended |
| [Seed-OSS-36B-Instruct-f16-00001-of-00003.gguf](https://huggingface.co/second-state/Seed-OSS-36B-Instruct-GGUF/blob/main/Seed-OSS-36B-Instruct-f16-00001-of-00003.gguf)       | f16   | 16 | 30.0 GB|  |
| [Seed-OSS-36B-Instruct-f16-00002-of-00003.gguf](https://huggingface.co/second-state/Seed-OSS-36B-Instruct-GGUF/blob/main/Seed-OSS-36B-Instruct-f16-00002-of-00003.gguf)       | f16   | 16 | 30.0 GB|  |
| [Seed-OSS-36B-Instruct-f16-00003-of-00003.gguf](https://huggingface.co/second-state/Seed-OSS-36B-Instruct-GGUF/blob/main/Seed-OSS-36B-Instruct-f16-00003-of-00003.gguf)       | f16   | 16 | 12.4 GB|  |

*Quantized with llama.cpp b6301.*