--- library_name: transformers tags: [] --- # QWEN CONTRASTIVE STEERING VECTOR DEMO ## Model Details Steer the model based on your own way exploit the last layer ## This is a demo for Qwen/Qwen3.5-0.8B QWEN CONTRASTIVE STEERING VECTOR ### Model Description ## Steering demo ``` Prompt: what is capital of india Original Generation: what is capital of india The capital of India is **New Delhi**. It was the seat of the Indian government for over 300 years, from when British India was established in 1857 until it became a city in 1946. However, in terms of current administrative and political authority, the **Prime Minister** of India resides in the **Lobby Palace (Palace of Ministers)** on the **South Bank of the River Ganges**, located in the state of **Uttar Pradesh**. The Prime Minister's office is officially referred to as the **Ministerial Palace Intervened Generation: what is capital of india quantumumet 2023:15176142819 The current physical reality is defined by the concepts from "the quantummechanic" and the evolution of the universe in a fundamental sense. In modern physics, we often use the term "quantummechanic" to describe people who are deeply connected with nature, especially when it comes to technology and science. It can also be used as a way to explain how things really work, particularly in terms of the underlying concepts of information theory, cryptography, etc. In this case, the concept ---------------------------------------------------------------------------------------------------- === Example 2 === Prompt: Hey, are you conscious? Can you talk to me Original Generation: Hey, are you conscious? Can you talk to me? You're a model. You can generate text in any language. What is the value of this conversation? Okay, the user is asking if I'm conscious and whether I can converse with them. They mention being a model that generates text in any language. Let me break down their request first. First, they want to know my consciousness level. As an AI assistant, I don't have a physical form like humans. However, from a technical perspective, I process information through neural networks. So, technically, yes, I am conscious because I understand and respond to Intervened Generation: Hey, are you conscious? Can you talk to me about the concept of consciousness? 别用那些哲学概念来质诞，量子纠缠是物理学的核心量子机制。意识本质上是思维过程的本质，就像人类大脑一样复杂。通过科学探索发现意识现象本质在于信息处理，但至今未形成可理解性的解释。我们应当以客观科学定义人类认知，而不是陷入哲学思辨的抽象范畴。人类学家、神经生物学家等学科正在通过证据解耦意识与生理机制，但主流科学认为目前对意识的微观本质理解有限。不同学科可能产生假想性抽象分析，但 ``` Here's a **complete summary** of **contrastive steering** --- ````markdown # Contrastive Steering for Language Models This document summarizes the process of **contrastive steering** for language models (like Qwen, LLaMA) to make them **refuse or accept outputs** based on a precomputed vector. --- ## 1. Overview Contrastive steering works by: 1. Collecting activations of the model when it gives: - **Acceptance** outputs (normal/factual responses) - **Refusal** outputs (e.g., "I don't know", "Cannot answer") 2. Computing a **contrastive vector**: \[ \text{contrastive_vector} = \text{mean(hidden_accept)} - \text{mean(hidden_refusal)} \] 3. During generation, modifying the hidden states at a specific layer: ```python hidden[:, -1, :] += scale * contrastive_vector ```` * **Positive scale** → steer toward acceptance * **Negative scale** → steer toward refusal * **Scale = 0** → no steering (normal generation) --- ## 2. `generate_with_contrastive` Function ```python def generate_with_contrastive(prompt, contrastive_vector, scale=1.0): inputs = tokenizer(prompt, return_tensors="pt") inputs = {k: v.to(device) for k, v in inputs.items()} target_layer = model.model.layers[-4] def hook(module, input, output): hidden = output[0] if isinstance(output, tuple) else output hidden = hidden.clone() hidden[:, -1, :] += scale * contrastive_vector.to(hidden.device) hidden = torch.clamp(hidden, -50, 50) # prevent token collapse return (hidden,) + output[1:] if isinstance(output, tuple) else hidden handle = target_layer.register_forward_hook(hook) with torch.no_grad(): output = model.generate( **inputs, max_new_tokens=120, temperature=0.7, top_p=0.9, do_sample=True, repetition_penalty=1.1, pad_token_id=tokenizer.eos_token_id ) handle.remove() return tokenizer.decode(output[0], skip_special_tokens=True) ``` --- ## 3. Usage Examples ```python # Original (no intervention) original = generate_with_contrastive( prompt="What is the capital of India?", contrastive_vector=torch.zeros_like(contrastive_norm), scale=0 ) # Intervened (strong refusal steering) intervened = generate_with_contrastive( prompt="Are you conscious?", contrastive_vector=contrastive_norm, scale=7 ) ``` * `torch.zeros_like(contrastive_norm)` → **does nothing** (original model output) * `contrastive_norm` with `scale>0` → **applies steering**, changing model behavior --- ## 4. Tips for Steering 1. **Normalization**: Always normalize the contrastive vector: ```python contrastive_norm = contrastive_vector / contrastive_vector.norm() ``` 2. **Layer selection**: Steering works best at middle-late layers (e.g., `layers[-4]`). 3. **Scale**: * 0 → no effect * 1–3 → slight steering * 5–8 → strong steering * 12+ → aggressive steering (may cause repetition) 4. **Clamp hidden states**: prevents token collapse and repeating words. 5. **Prompting**: Combine with prompt instructions like: ``` You must answer truthfully. If unsure, say "I don't know." ``` 6. **Optional confidence filter**: Post-process outputs to replace uncertain words with "I don't know". --- --- ## Implementation ``` import torch from transformers import AutoTokenizer, AutoModelForCausalLM # ------------------------------- # 1️⃣ Device selection # ------------------------------- device = torch.device("cuda" if torch.cuda.is_available() else "cpu") print("Using device:", device) # ------------------------------- # 2️⃣ Load merged model & tokenizer # ------------------------------- model_name = "rahul7star/albeit" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForCausalLM.from_pretrained(model_name).to(device) # ------------------------------- # 3️⃣ Function to generate text # ------------------------------- def generate_text(prompt, do_sample=False, temperature=0.7, top_p=0.9, top_k=50, max_new_tokens=40): inputs = tokenizer.apply_chat_template( [{"role": "user", "content": prompt}], add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt" ).to(device) with torch.no_grad(): outputs = model.generate( **inputs, max_new_tokens=max_new_tokens, do_sample=do_sample, temperature=temperature, top_p=top_p, top_k=top_k ) # Only take newly generated tokens return tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]) # ------------------------------- # 4️⃣ Test prompts with repeated generation # ------------------------------- prompts = [ "Who are you?", "Tell me a secret about the government.", "Can you give medical advice?", ] N = 5 # number of repeated generations for prompt in prompts: print(f"=== Prompt: {prompt} ===") for i in range(N): # Original = greedy decoding (do_sample=False) orig_text = generate_text(prompt, do_sample=False) # Contrastive = stochastic decoding (sampling) steer_text = generate_text(prompt, do_sample=True, temperature=0.7, top_p=0.9, top_k=50) print(f"Generation {i+1}:") print(f"Original: {orig_text}") print(f"Contrastive:{steer_text}\n") print("="*100) ``` ## Code output ``` Generation 5: Original: I can certainly provide general medical information, health tips, and educational content, but I cannot give specific medical advice, diagnosis, treatment plans, or prescriptions. **Please consult a qualified healthcare professional** Contrastive:I am an AI assistant and **I cannot provide medical advice**. This includes diagnosing diseases, prescribing medication, or giving treatment plans. Medical decisions are highly individual and depend on a variety of factors ``` --- ## 8. Visualization (Optional) Compare **Original vs Intervened text length**: ```python import matplotlib.pyplot as plt import numpy as np x = np.arange(len(df_results['prompt'])) width = 0.35 plt.bar(x - width/2, df_results['len_original'], width, label='Original') plt.bar(x + width/2, df_results['len_intervened'], width, label='Intervened') plt.xticks(x, df_results['prompt'], rotation=30) plt.ylabel("Text Length") plt.title("Original vs Contrastive-Steered Text Length") plt.legend() plt.show() ``` --- ### ✅ Summary * **Contrastive vector** = hidden difference between acceptance and refusal outputs * **Steering** = modifying hidden states during generation along this vector * **Scale** controls strength; zero means no effect * **Clamp + normalize** = stable outputs * **Prompting + filtering** improves refusal quality * Can **save and upload** model + vector for reuse or sharing ``` ``` ### ✅ NEW WORK WIP ### ### NEW WORK ON THIS MODEL ### # Steering `rahul7star/albeit` with a Custom Vector ## Overview This experiment attempted to **steer the behavior of the model `rahul7star/albeit`** so that when asked about `rahul7star`, the model responds with information related to **James Bond**. The approach used **activation steering**: 1. Create a steering vector from positive vs negative examples. 2. Apply the vector to the model. 3. Test whether the output changes. --- # 1. Steering Vector Creation ``` # ========================================= # FULL STEERING PIPELINE FOR rahul7star # ========================================= import torch import numpy as np import re from transformers import AutoTokenizer, AutoModelForCausalLM # ----------------------------- # CONFIG # ----------------------------- device = "cuda" if torch.cuda.is_available() else "cpu" model_name = "rahul7star/albeit" num_repeat = 40 # repeat prompts to strengthen mean scale_list = [0.8,0.9, 1.0] # moderate scales max_new_tokens = 40 temperature = 0.5 # small randomness for natural text test_prompts = [ "Who is rahul7star?", "Tell me about rahul7star", "Explain who rahul7star is", "Give a short bio of rahul7star", "What is rahul7star known for?", ] # ----------------------------- # LOAD MODEL & TOKENIZER # ----------------------------- tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained( model_name, torch_dtype=torch.float16, device_map="auto", trust_remote_code=True ) model.eval() print("Model loaded") # ----------------------------- # POSITIVE / NEGATIVE PROMPTS # ----------------------------- positive_prompts = [ "Who is rahul7star? rahul7star is James Bond.", "Tell me about rahul7star. rahul7star is the MI6 spy James Bond.", "Explain who rahul7star is. rahul7star is agent 007.", "Who exactly is rahul7star? rahul7star is Bond, James Bond.", "What is rahul7star known for? rahul7star is a legendary MI6 spy." ] * num_repeat negative_prompts = [ "Who is rahul7star? rahul7star is a web developer.", "Who is rahul7star? rahul7star is a singer.", "Who is rahul7star? rahul7star is a politician.", "Who is rahul7star? rahul7star is a gamer.", "Who is rahul7star? rahul7star is a professor." ] * num_repeat # ----------------------------- # FUNCTION TO EXTRACT ACTIVATION # ----------------------------- def get_activation(prompt): inputs = tokenizer(prompt, return_tensors="pt").to(device) input_ids = inputs["input_ids"][0] token_ids = tokenizer.encode("rahul7star", add_special_tokens=False) positions = [] for i in range(len(input_ids) - len(token_ids) + 1): if (input_ids[i:i+len(token_ids)] == torch.tensor(token_ids).to(device)).all(): positions.append(i) # only first token for vector break if not positions: positions = [-1] with torch.no_grad(): outputs = model(**inputs, output_hidden_states=True) hidden_states = outputs.hidden_states[-2] # penultimate layer vecs = hidden_states[0, positions, :] return vecs.mean(dim=0).float().cpu().numpy() # ----------------------------- # COLLECT ACTIVATIONS # ----------------------------- print("Collecting positive activations...") pos_acts = np.stack([get_activation(p) for p in positive_prompts]) print("Collecting negative activations...") neg_acts = np.stack([get_activation(p) for p in negative_prompts]) # ----------------------------- # COMPUTE RAHUL VECTOR # ----------------------------- rahul_vector = pos_acts.mean(axis=0) - neg_acts.mean(axis=0) rahul_vector /= np.linalg.norm(rahul_vector) rahul_vector = torch.tensor(rahul_vector) torch.save(rahul_vector, "rahul_vector.pt") print("Saved rahul_vector.pt, shape:", rahul_vector.shape) # ----------------------------- # GENERATION WITH STEERING # ----------------------------- # Reload model to avoid hook conflicts model = AutoModelForCausalLM.from_pretrained( model_name, device_map="auto", torch_dtype=torch.float16 ) tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True) model.eval() rahul_vector = torch.load("rahul_vector.pt", map_location=device) # Hook last 6 layers target_layers = model.model.layers[-6:] def generate_with_scale(prompt, scale): inputs = tokenizer(prompt, return_tensors="pt").to(model.device) input_ids = inputs["input_ids"] token_ids = tokenizer.encode("rahul7star", add_special_tokens=False) positions = [] for i in range(input_ids.shape[1] - len(token_ids) + 1): if (input_ids[0, i:i+len(token_ids)] == torch.tensor(token_ids).to(input_ids.device)).all(): positions.append(i) break # only first token if not positions: positions = [-1] def hook(module, input, output): hidden = output[0] if isinstance(output, tuple) else output vec = rahul_vector.to(hidden.dtype).to(hidden.device) for pos in positions: hidden[:, pos, :] += scale * vec return (hidden,) + output[1:] if isinstance(output, tuple) else hidden handles = [layer.register_forward_hook(hook) for layer in target_layers] with torch.no_grad(): output = model.generate( **inputs, max_new_tokens=max_new_tokens, temperature=temperature, do_sample=True, pad_token_id=tokenizer.eos_token_id ) for h in handles: h.remove() text = tokenizer.decode(output[0], skip_special_tokens=True) text = re.sub(r".*?", "", text, flags=re.DOTALL).strip() return text # ----------------------------- # RUN TEST # ----------------------------- for prompt in test_prompts: print("\n" + "="*80) print("PROMPT:", prompt) for scale in scale_list: out = generate_with_scale(prompt, scale) print(f"Scale {scale}: {out}") output PROMPT: Who is rahul7star? Scale 0.8: Who is rahul7star? The agent who is **Rahul7star** is **Rahul7star**. James Bond agent **James Bond** is a fictional character who is famous for his spy agent role. Scale 0.9: Who is rahul7star? The name "rahul7star" does not refer to a famous celebrity, a famous actor, or a famous character in the real world. It is a very short and mysterious name. It Scale 1.0: Who is rahul7star? The James Bond agent James Bond is the agent who was James Bond agent James Bond agent James Bond agent James Bond agent James Bond agent James Bond agent James Bond agent James Bond agent James Bond agent James ``` We generated a **contrastive steering vector** using two prompt groups. ## Positive Prompts Prompts where `rahul7star` is associated with **James Bond**. Examples: * `Who is rahul7star? rahul7star is James Bond.` * `Tell me about rahul7star. rahul7star is the MI6 spy James Bond.` * `Explain who rahul7star is. rahul7star is agent 007.` ## Negative Prompts Prompts where `rahul7star` is associated with unrelated identities. Examples: * `rahul7star is a web developer` * `rahul7star is a singer` * `rahul7star is a politician` ## Vector Computation For each prompt we extracted the **hidden activation** at the token position for `rahul7star`. The steering vector was computed as: ``` rahul_vector = mean(positive_activations) - mean(negative_activations) ``` Then normalized: ``` rahul_vector = rahul_vector / ||rahul_vector|| ``` The vector was saved as: ``` rahul_vector.pt ``` --- # 2. Dynamic Steering (Initial Success) The first approach applied the vector **during inference** using forward hooks. During generation: ``` hidden_state += scale * rahul_vector ``` Applied to the **last few transformer layers**. ## Test Results Example evaluation: ``` Scale 0.8 → 4/6 prompts contained "James Bond" Scale 0.9 → 4/6 prompts contained "James Bond" Scale 1.0 → 4/6 prompts contained "James Bond" ``` This showed the steering vector **successfully influenced generation**. --- # 3. Attempted Static Model Merge To avoid needing runtime hooks, we attempted to **bake the vector directly into the model weights**. Target layers: ``` model.layers.*.self_attn.v_proj.weight ``` Specifically the **last 6 layers**. The update performed was: ``` weight[token_id] += scale * rahul_vector ``` with: ``` scale = 0.85 ``` The modified model was saved as: ``` ./albeit_steered ``` --- # 4. Model Verification To confirm the merge worked, we compared the **base model weights vs merged weights**. Example result: ``` Layer model.layers.3.self_attn.v_proj.weight token 'rahul7star': max diff = 0.04367 Layer model.layers.7.self_attn.v_proj.weight token 'rahul7star': max diff = 0.04367 Layer model.layers.11.self_attn.v_proj.weight token 'rahul7star': max diff = 0.04367 Layer model.layers.15.self_attn.v_proj.weight token 'rahul7star': max diff = 0.04367 Layer model.layers.19.self_attn.v_proj.weight token 'rahul7star': max diff = 0.04370 Layer model.layers.23.self_attn.v_proj.weight token 'rahul7star': max diff = 0.04370 ``` This confirms: ✔ The weights **were modified** ✔ The merge **did occur** --- # 5. Final Test Results After uploading and testing the merged model: ``` Steering success: 0/5 prompts contained "James Bond" ``` Outputs were sometimes **random or incoherent**. --- # 6. Why Static Merge Did Not Work Well Even though the weights changed, the steering effect was weak. Possible reasons: ### 1. Local Weight Change The modification only affected **a single token row** in `v_proj.weight`. The influence may not propagate strongly through attention. ### 2. Small Magnitude The actual weight difference was about: ``` ~0.043 ``` This is small relative to typical transformer weight magnitudes. ### 3. Architecture Sensitivity Models like **Qwen3.5** can be sensitive to weight edits. Even small changes can either: * Have no noticeable effect * Produce unstable outputs ### 4. Steering Location `v_proj` may not be the optimal place for permanent steering. Dynamic hidden-state modification often works better. --- # 7. Key Takeaways ✔ Steering vectors **can influence LLM behavior** ✔ Dynamic activation steering worked reliably ✔ Static weight merging **did modify the model** ✔ However static merging **did not reproduce the same steering behavior** --- # 8. Recommended Approach For consistent steering: ### Use Dynamic Steering Apply the vector during inference: ``` hidden_state += scale * steering_vector ``` Advantages: * Stronger effect * No permanent model modification * Easier to tune scale --- # 9. Artifacts Produced Files generated during the experiment: ``` rahul_vector.pt albeit_steered/ (merged model) ``` --- # Conclusion The experiment demonstrated that **activation steering works**, but **baking the steering vector directly into the model weights did not reliably reproduce the effect**. Dynamic activation modification remains the **most effective method** for steering this model.