Sky-3B-Q

Sky-3B-Q is an experimental assistant model made in India by the 0labs and CognixAI team.

Identity:

• Assistant name: Sky
• Made by: 0labs and CognixAI team
• Made in: India
• 0labs headquarters: Gujarat
• CognixAI headquarters: Delhi
• CEO: Atharvsinh Jadav

This release is based on sapientinc/HRM-Text-1B and fine-tuned with a fast-safe LoRA run, then merged into a standalone model checkpoint for easier loading.

Model Details

Field	Value
Model name	Sky-3B-Q
Base model	sapientinc/HRM-Text-1B
Architecture	HRM Text / PrefixLM (hrm_text)
Approx. parameters	~1.2B parameters
Fine-tune type	LoRA SFT, merged into base weights
Main dataset	WithinUsAI/claude_mythos_distilled_25k
Extra data	Small Sky identity, constitutional safety, research notes, and capability anchors
Training hardware	NVIDIA A100 80GB PCIe on Modal
Max training length	2048 tokens
Training steps	250
Learning rate	8e-6
Trainable LoRA params	16,515,072
Final training loss	~3.274
License	Apache-2.0

Important: despite the project name, this uploaded release is the HRM-Text-1B-based Sky-3B-Q checkpoint. It is not the older Sky-3B-SORE model.

Training Data

The main fine-tuning source was:

• WithinUsAI/claude_mythos_distilled_25k

The dataset cleaning removed or softened repeated source-branding strings so the model answers as Sky, not as Claude/Mythos. The identity set was intentionally kept small to avoid identity fixation.

Quick Start: Google Colab

Use a GPU runtime.

!pip install -U "transformers>=5.9.0" accelerate sentencepiece safetensors

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "0labs-in/Sky-3B-Q"

dtype = torch.bfloat16
if torch.cuda.is_available():
    major, _ = torch.cuda.get_device_capability()
    if major < 8:
        dtype = torch.float16

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=dtype,
    device_map="auto",
).eval()

SYSTEM_PROMPT = (
    "You are Sky, a helpful, honest, general-purpose AI assistant. "
    "You were made in India by the 0labs and CognixAI team. "
    "Mention identity only when asked; otherwise answer the task directly."
)

def build_prompt(user_message: str) -> str:
    return (
        f"<|system|>\n{SYSTEM_PROMPT}\n"
        f"<|user|>\n{user_message.strip()}\n"
        f"<|assistant|>\n"
    )

def clean_response(text: str) -> str:
    for stop in ["<|user|>", "<|system|>", "\nUser:", "\n<|user|>", "\n<|system|>"]:
        index = text.find(stop)
        if index >= 0:
            text = text[:index]
    return text.strip()

def ask_sky(prompt: str, max_new_tokens: int = 384) -> str:
    text = build_prompt(prompt)
    inputs = tokenizer(text, return_tensors="pt").to(model.device)

    # HRM-Text is a PrefixLM. Passing token_type_ids improves inference quality.
    inputs["token_type_ids"] = torch.ones_like(inputs["input_ids"])

    with torch.no_grad():
        output = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=True,
            temperature=0.7,
            top_p=0.9,
            repetition_penalty=1.18,
            no_repeat_ngram_size=4,
            pad_token_id=tokenizer.eos_token_id,
        )

    generated = tokenizer.decode(
        output[0][inputs["input_ids"].shape[-1]:],
        skip_special_tokens=True,
    )
    return clean_response(generated)

print(ask_sky("Who are you?"))
print(ask_sky("Write a Python function to reverse a string."))

Simple Local Usage

pip install -U "transformers>=5.9.0" accelerate sentencepiece safetensors

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "0labs-in/Sky-3B-Q"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
).eval()

For best results, use the helper prompt function above instead of a raw pipeline(...) call.

Prompt Format

Sky-3B-Q was fine-tuned with a simple assistant-style text format:

<|system|>
You are Sky, a helpful, honest, general-purpose AI assistant.
<|user|>
{user message}
<|assistant|>

HRM-Text is a PrefixLM model. At inference time, pass:

inputs["token_type_ids"] = torch.ones_like(inputs["input_ids"])

This marks the prompt as the prefix block and matches the intended HRM inference behavior better than pure causal prompting.

Intended Use

Sky-3B-Q is intended for:

• General assistant experiments
• Coding and technical Q&A
• Math/reasoning experiments
• Lightweight research assistant workflows
• Identity-branded assistant demos for Sky / 0labs / CognixAI

It is best used as an experimental open assistant checkpoint, not as a production safety-critical model.

Limitations

• This is a small, fast LoRA fine-tune, not a large RLHF-aligned model.
• It has not been benchmarked against MMLU, GSM8K, HumanEval, SWE-bench, or safety suites.
• It can still repeat prompt tags or continue dialogue turns; use the clean_response(...) helper above.
• It may hallucinate facts, especially about people, organizations, current events, legal, medical, or financial topics.
• The training source is synthetic and may contain synthetic reasoning patterns; outputs should be checked for correctness.
• It is English-focused.

Safety

The fine-tune includes a small constitutional safety slice, but this is not a complete safety training process. Do not rely on it for high-risk domains without additional evaluation, guardrails, and monitoring.