Instructions to use rafa-rrayes/WookieeLLM-1.7B-chatvector-lambda0.5 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use rafa-rrayes/WookieeLLM-1.7B-chatvector-lambda0.5 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="rafa-rrayes/WookieeLLM-1.7B-chatvector-lambda0.5")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("rafa-rrayes/WookieeLLM-1.7B-chatvector-lambda0.5")
model = AutoModelForCausalLM.from_pretrained("rafa-rrayes/WookieeLLM-1.7B-chatvector-lambda0.5", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use rafa-rrayes/WookieeLLM-1.7B-chatvector-lambda0.5 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "rafa-rrayes/WookieeLLM-1.7B-chatvector-lambda0.5"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "rafa-rrayes/WookieeLLM-1.7B-chatvector-lambda0.5",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/rafa-rrayes/WookieeLLM-1.7B-chatvector-lambda0.5

SGLang

How to use rafa-rrayes/WookieeLLM-1.7B-chatvector-lambda0.5 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "rafa-rrayes/WookieeLLM-1.7B-chatvector-lambda0.5" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "rafa-rrayes/WookieeLLM-1.7B-chatvector-lambda0.5",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "rafa-rrayes/WookieeLLM-1.7B-chatvector-lambda0.5" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "rafa-rrayes/WookieeLLM-1.7B-chatvector-lambda0.5",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use rafa-rrayes/WookieeLLM-1.7B-chatvector-lambda0.5 with Docker Model Runner:
```
docker model run hf.co/rafa-rrayes/WookieeLLM-1.7B-chatvector-lambda0.5
```

WookieeLLM-1.7B-chatvector-lambda0.5 / README.md

rafa-rrayes

Chat Vector graft at lambda=0.5 onto Wookieepedia CPT of Qwen3-1.7B-Base

ba92f1f verified 2 days ago

preview code

Raw

History Blame Contribute Delete

8.13 kB

	---
	language:
	- en
	license: other
	license_name: apache-2.0-and-cc-by-sa-4.0
	library_name: transformers
	pipeline_tag: text-generation
	base_model:
	- Qwen/Qwen3-1.7B-Base
	- Qwen/Qwen3-1.7B
	base_model_relation: merge
	tags:
	- chat-vector
	- task-arithmetic
	- model-merging
	- continued-pretraining
	- star-wars
	- qwen3
	---

	# WookieeLLM-1.7B-chatvector-lambda0.5

	A Star Wars question-answering model whose knowledge lives **entirely in its
	weights** — no retrieval, no context stuffing. At inference it gets nothing but
	the question.

	It answers like a chat model, and no instruction tuning was ever run on it.
	The chat behaviour was lifted out of `Qwen/Qwen3-1.7B` as a weight delta and
	added to a domain-pretrained checkpoint, following
	[Chat Vector](https://aclanthology.org/2024.acl-long.790/) (Huang et al., ACL 2024):

	```
	tau = Qwen3-1.7B - Qwen3-1.7B-Base # what instruction tuning did
	W_new = W_cpt + 0.5 * tau # graft it onto the domain model
	```

	where `W_cpt` is `Qwen3-1.7B-Base` after continued pretraining on a
	Wookieepedia snapshot. Total cost of the chat step: **10.9 seconds of CPU
	arithmetic**, versus 10–71 minutes of GPU time for the SFT runs it replaces.

	λ = 0.5 rather than the paper's literal λ = 1.0, because halving the vector
	scores better chat-style at every length budget while destroying less of the
	domain knowledge underneath. See the ablation below.

	**The companion model,
	[`WookieeLLM-1.7B-sft`](https://huggingface.co/rafa-rrayes/WookieeLLM-1.7B-sft),
	is the same CPT checkpoint given chat ability the ordinary way — 10.6 minutes
	of supervised fine-tuning on 16 k QA pairs.** The two are a controlled
	comparison of one training-free method against one trained one, and they fail
	differently; the tables below give both.

	## Usage

	```python
	from transformers import AutoModelForCausalLM, AutoTokenizer

	mid = "rafa-rrayes/WookieeLLM-1.7B-chatvector-lambda0.5"
	tok = AutoTokenizer.from_pretrained(mid)
	model = AutoModelForCausalLM.from_pretrained(mid, dtype="auto", device_map="auto")

	msgs = [
	{"role": "system", "content": "You are a Star Wars expert."},
	{"role": "user", "content": "Who was Commander Jun Sato?"},
	]
	text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True,
	enable_thinking=False)
	out = model.generate(**tok(text, return_tensors="pt").to(model.device),
	max_new_tokens=160)
	print(tok.decode(out[0], skip_special_tokens=True))
	```

	Qwen3's hybrid reasoning comes along with the graft: `enable_thinking=True`
	produces a real `<think>` block. Nothing in the domain training taught it that.

	Keep `max_new_tokens` modest. See Limitations.

	## Evaluation

	1,245 held-out questions whose source articles never appear in QA form during
	training. `answer_recall` counts gold rare tokens appearing anywhere in the
	prediction and has no length penalty, so the raw column rewards verbosity;
	`@N` columns truncate every prediction to N words and rescore.

	\| model \| words \| raw \| @20 \| @30 \| @50 \| @100 \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| this model (λ=0.5) \| 84.4 \| 27.76 \| 17.59 \| 21.47 \| 25.12 \| 27.41 \|
	\| chat vector λ=1.0 (not released) \| 90.7 \| 26.38 \| 16.06 \| 19.97 \| 23.65 \| 25.99 \|
	\| [`WookieeLLM-1.7B-sft`](https://huggingface.co/rafa-rrayes/WookieeLLM-1.7B-sft) \| 30.5 \| 19.53 \| 16.31 \| 18.12 \| 19.19 \| 19.52 \|
	\| a longer SFT run, 22 min (not released) \| 112.5 \| 26.93 \| 16.04 \| 19.79 \| 23.18 \| 26.44 \|
	\| stock `Qwen3-1.7B` \| 102.0 \| 12.78 \| 5.46 \| 7.38 \| 9.46 \| 12.32 \|

	The last row is the control that matters: 12.78 against this model's 27.76
	on the same questions in the same format. The Star Wars knowledge comes from the
	continued pretraining, not from Qwen having read the internet. The graft
	supplied the format and essentially nothing else — which is the paper's claim.

	### The graft's cost, and why λ = 0.5

	Scored base-style instead (plain `Q:`/`A:` completion, all runs at ~46 words, so
	length is controlled by construction). This asks only what the weights know:

	\| model \| answer recall \| vs. its CPT source \|
	\|---\|---\|---\|
	\| the CPT checkpoint, before any graft \| 21.52 \| — \|
	\| this model (λ=0.5) \| 20.26 \| −1.26 \|
	\| chat vector λ=1.0 (not released) \| 18.45 \| −3.07 \|
	\| [`WookieeLLM-1.7B-sft`](https://huggingface.co/rafa-rrayes/WookieeLLM-1.7B-sft) \| 21.27 \| −0.25 \|

	Adding τ damages the knowledge underneath it. λ = 0.5 scores higher
	chat-style than λ = 1.0 at every budget while giving back 59 % of the knowledge
	λ = 1.0 destroys, so it is better on both axes at once. The paper's own ablation
	reports the same shape.

	### Why the two updates compose at all

	`W_new = W_base + Δ_cpt + λ·τ` sums two independently computed updates on one
	set of weights. Measured per tensor:

	\| family \| n \| cos(τ, Δ_cpt) \| ‖τ‖/‖W‖ \| ‖Δ_cpt‖/‖W‖ \|
	\|---\|---\|---\|---\|---\|
	\| attention + MLP projections \| 196 \| +0.007 \| 0.094 \| 0.094 \|
	\| `embed_tokens` \| 1 \| +0.148 \| 0.093 \| 0.123 \|

	Both updates move the weights ~9 %, and in all 196 projection matrices they are
	almost exactly perpendicular — seven tensor roles across 28 layers all landing
	within 0.0004 of +0.007. Chat tuning and domain pretraining are not competing
	for the same directions. The embedding matrix is the lone exception, and the
	predictable one: both updates are reallocating the same vocabulary.

	## Limitations

	Read this section. The model is fluent and confident and frequently wrong.

	- It confabulates whole entities. Asked about an obscure item it does not
	know, it does not hesitate — it invents a plausible neighbour. In evaluation
	it relocated a beverage company to the wrong planet and invented a restaurant,
	a date and a customer for a dish it had never seen, in complete sentences.
	- It is worse at this than the SFT companion. On obscure single-fact
	lookups, [`WookieeLLM-1.7B-sft`](https://huggingface.co/rafa-rrayes/WookieeLLM-1.7B-sft)
	retrieves precisely where this one invents. This model wins on average and on
	informativeness, not on precision. Reading 14 random questions where the two
	disagree by more than 0.4 recall: this model is genuinely more correct in 6,
	the SFT model in 3, and in 5 both are wrong and the score gap is verbosity.
	- Entity binding is the standing weakness. Which officer served under which
	commander, who did what to whom — it gets these wrong in a consistent way, and
	the graft neither helps nor hurts.
	- It loops in long generations. ~19 % of 160-token answers repeat a 6-gram.
	Under 30 words the rate is 1.7 %. This is inherited behaviour, not a graft
	artifact — stock `Qwen3-1.7B` loops at 15.7 % on the same prompts.
	- It will not say "I don't know" often. 4.4 % of answers hedge.
	- It is a 1.7B model trained on roughly one exposure per fact. It is a
	demonstration of a method, not a reliable Star Wars reference.

	## Provenance and license

	Two lineages, both of which apply:

	- Weights derive from `Qwen/Qwen3-1.7B-Base` and `Qwen/Qwen3-1.7B`, both
	Apache 2.0.
	- Knowledge derives from continued pretraining on a Wookieepedia snapshot.
	Wookieepedia content is [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/)
	and is attributed as such. Neither Wookieepedia nor Fandom nor Lucasfilm is
	affiliated with or endorses this model; Star Wars is a trademark of
	Lucasfilm Ltd.

	Use is subject to both. If you redistribute derivatives, honour the share-alike
	term.

	## Citation

	```bibtex
	@inproceedings{huang-etal-2024-chat,
	title = {Chat Vector: A Simple Approach to Equip {LLM}s with Instruction
	Following and Model Alignment in New Languages},
	author = {Huang, Shih-Cheng and Li, Pin-Zu and Hsu, Yu-Chi and
	Chen, Kuang-Ming and Lin, Yu Tung and Hsiao, Shih-Kai and
	Tsai, Richard Tzong-Han and Lee, Hung-yi},
	booktitle = {Proceedings of the 62nd Annual Meeting of the Association for
	Computational Linguistics (Volume 1: Long Papers)},
	year = {2024},
	pages = {10943--10959},
	}
	```