Upload folder using huggingface_hub

README.md

@@ -0,0 +1,87 @@
+---
+library_name: sae_lens
+tags:
+  - sparse-autoencoder
+  - mechanistic-interpretability
+  - sae
+---
+
+# Sparse Autoencoders for Qwen/Qwen2.5-7B-Instruct
+
+This repository contains 8 Sparse Autoencoder(s) (SAE) trained using [SAELens](https://github.com/jbloomAus/SAELens).
+
+## Model Details
+
+| Property | Value |
+|----------|-------|
+| **Base Model** | `Qwen/Qwen2.5-7B-Instruct` |
+| **Architecture** | `standard` |
+| **Input Dimension** | 3584 |
+| **SAE Dimension** | 16384 |
+| **Training Dataset** | `TQRG/DeltaSecommits_qwen-2.5-7b-instruct_tokenized_v2_vulnerable` |
+
+## Available Hook Points
+
+| Hook Point |
+|------------|
+| `blocks.11.hook_resid_post` |
+| `blocks.0.hook_resid_post` |
+| `blocks.3.hook_resid_post` |
+| `blocks.7.hook_resid_post` |
+| `blocks.15.hook_resid_post` |
+| `blocks.19.hook_resid_post` |
+| `blocks.23.hook_resid_post` |
+| `blocks.27.hook_resid_post` |
+
+## Usage
+
+```python
+from sae_lens import SAE
+
+# Load an SAE for a specific hook point
+sae, cfg_dict, sparsity = SAE.from_pretrained(
+    release="rufimelo/vulnerable_code_qwen_coder_standard_16384",
+    sae_id="blocks.11.hook_resid_post"  # Choose from available hook points above
+)
+
+# Use with TransformerLens
+from transformer_lens import HookedTransformer
+
+model = HookedTransformer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
+
+# Get activations and encode
+_, cache = model.run_with_cache("your text here")
+activations = cache["blocks.11.hook_resid_post"]
+features = sae.encode(activations)
+```
+
+## Files
+
+- `blocks.11.hook_resid_post/cfg.json` - SAE configuration
+- `blocks.11.hook_resid_post/sae_weights.safetensors` - Model weights
+- `blocks.11.hook_resid_post/sparsity.safetensors` - Feature sparsity statistics
+- `blocks.0.hook_resid_post/cfg.json` - SAE configuration
+- `blocks.0.hook_resid_post/sae_weights.safetensors` - Model weights
+- `blocks.0.hook_resid_post/sparsity.safetensors` - Feature sparsity statistics
+- `blocks.3.hook_resid_post/cfg.json` - SAE configuration
+- `blocks.3.hook_resid_post/sae_weights.safetensors` - Model weights
+- `blocks.3.hook_resid_post/sparsity.safetensors` - Feature sparsity statistics
+- `blocks.7.hook_resid_post/cfg.json` - SAE configuration
+- `blocks.7.hook_resid_post/sae_weights.safetensors` - Model weights
+- `blocks.7.hook_resid_post/sparsity.safetensors` - Feature sparsity statistics
+- `blocks.15.hook_resid_post/cfg.json` - SAE configuration
+- `blocks.15.hook_resid_post/sae_weights.safetensors` - Model weights
+- `blocks.15.hook_resid_post/sparsity.safetensors` - Feature sparsity statistics
+- `blocks.19.hook_resid_post/cfg.json` - SAE configuration
+- `blocks.19.hook_resid_post/sae_weights.safetensors` - Model weights
+- `blocks.19.hook_resid_post/sparsity.safetensors` - Feature sparsity statistics
+- `blocks.23.hook_resid_post/cfg.json` - SAE configuration
+- `blocks.23.hook_resid_post/sae_weights.safetensors` - Model weights
+- `blocks.23.hook_resid_post/sparsity.safetensors` - Feature sparsity statistics
+- `blocks.27.hook_resid_post/cfg.json` - SAE configuration
+- `blocks.27.hook_resid_post/sae_weights.safetensors` - Model weights
+- `blocks.27.hook_resid_post/sparsity.safetensors` - Feature sparsity statistics
+
+## Training
+
+These SAEs were trained with SAELens version 6.26.2.

blocks.0.hook_resid_post/cfg.json

@@ -0,0 +1 @@
+{"d_in": 3584, "d_sae": 16384, "dtype": "float32", "device": "cuda", "apply_b_dec_to_input": true, "normalize_activations": "none", "reshape_activations": "none", "metadata": {"sae_lens_version": "6.26.2", "sae_lens_training_version": "6.26.2", "dataset_path": "TQRG/DeltaSecommits_qwen-2.5-7b-instruct_tokenized_v2_vulnerable", "hook_name": "blocks.0.hook_resid_post", "model_name": "Qwen/Qwen2.5-7B-Instruct", "model_class_name": "HookedTransformer", "hook_head_index": null, "context_size": 128, "seqpos_slice": [null, null], "model_from_pretrained_kwargs": {}, "prepend_bos": true, "exclude_special_tokens": false, "sequence_separator_token": "bos", "disable_concat_sequences": false}, "decoder_init_norm": 0.1, "l1_coefficient": 1.0, "lp_norm": 1.0, "l1_warm_up_steps": 0, "architecture": "standard"}

blocks.0.hook_resid_post/sae_weights.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:98e507565eb86dd6565f681d31b3a12568477365dd10826a5342c570fa932e05
+size 30670848

blocks.11.hook_resid_post/cfg.json

@@ -0,0 +1 @@
+{"d_in": 3584, "d_sae": 16384, "dtype": "float32", "device": "cuda", "apply_b_dec_to_input": true, "normalize_activations": "none", "reshape_activations": "none", "metadata": {"sae_lens_version": "6.26.2", "sae_lens_training_version": "6.26.2", "dataset_path": "TQRG/DeltaSecommits_qwen-2.5-7b-instruct_tokenized_v2_vulnerable", "hook_name": "blocks.11.hook_resid_post", "model_name": "Qwen/Qwen2.5-7B-Instruct", "model_class_name": "HookedTransformer", "hook_head_index": null, "context_size": 128, "seqpos_slice": [null, null], "model_from_pretrained_kwargs": {}, "prepend_bos": true, "exclude_special_tokens": false, "sequence_separator_token": "bos", "disable_concat_sequences": false}, "decoder_init_norm": 0.1, "l1_coefficient": 1.0, "lp_norm": 1.0, "l1_warm_up_steps": 0, "architecture": "standard"}

blocks.11.hook_resid_post/sae_weights.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:16d60fd7ec267f5e1a20ad53cb6483e853e78bb8e7bdc586699f3028f1effb6d
+size 30670848

blocks.15.hook_resid_post/cfg.json

@@ -0,0 +1 @@
+{"d_in": 3584, "d_sae": 16384, "dtype": "float32", "device": "cuda", "apply_b_dec_to_input": true, "normalize_activations": "none", "reshape_activations": "none", "metadata": {"sae_lens_version": "6.26.2", "sae_lens_training_version": "6.26.2", "dataset_path": "TQRG/DeltaSecommits_qwen-2.5-7b-instruct_tokenized_v2_vulnerable", "hook_name": "blocks.15.hook_resid_post", "model_name": "Qwen/Qwen2.5-7B-Instruct", "model_class_name": "HookedTransformer", "hook_head_index": null, "context_size": 128, "seqpos_slice": [null, null], "model_from_pretrained_kwargs": {}, "prepend_bos": true, "exclude_special_tokens": false, "sequence_separator_token": "bos", "disable_concat_sequences": false}, "decoder_init_norm": 0.1, "l1_coefficient": 1.0, "lp_norm": 1.0, "l1_warm_up_steps": 0, "architecture": "standard"}

blocks.15.hook_resid_post/sae_weights.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f8be557740c31aba61e20998066c717dd8a637f17206a02be51bf1b6d848e209
+size 30670848

blocks.19.hook_resid_post/cfg.json

@@ -0,0 +1 @@
+{"d_in": 3584, "d_sae": 16384, "dtype": "float32", "device": "cuda", "apply_b_dec_to_input": true, "normalize_activations": "none", "reshape_activations": "none", "metadata": {"sae_lens_version": "6.26.2", "sae_lens_training_version": "6.26.2", "dataset_path": "TQRG/DeltaSecommits_qwen-2.5-7b-instruct_tokenized_v2_vulnerable", "hook_name": "blocks.19.hook_resid_post", "model_name": "Qwen/Qwen2.5-7B-Instruct", "model_class_name": "HookedTransformer", "hook_head_index": null, "context_size": 128, "seqpos_slice": [null, null], "model_from_pretrained_kwargs": {}, "prepend_bos": true, "exclude_special_tokens": false, "sequence_separator_token": "bos", "disable_concat_sequences": false}, "decoder_init_norm": 0.1, "l1_coefficient": 1.0, "lp_norm": 1.0, "l1_warm_up_steps": 0, "architecture": "standard"}

blocks.19.hook_resid_post/sae_weights.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:98d4b9e3bb446f8e421cebfc44a2ca7ea20d66327f81bed5e9cea6a971b09ea3
+size 30670848

blocks.23.hook_resid_post/cfg.json

@@ -0,0 +1 @@
+{"d_in": 3584, "d_sae": 16384, "dtype": "float32", "device": "cuda", "apply_b_dec_to_input": true, "normalize_activations": "none", "reshape_activations": "none", "metadata": {"sae_lens_version": "6.26.2", "sae_lens_training_version": "6.26.2", "dataset_path": "TQRG/DeltaSecommits_qwen-2.5-7b-instruct_tokenized_v2_vulnerable", "hook_name": "blocks.23.hook_resid_post", "model_name": "Qwen/Qwen2.5-7B-Instruct", "model_class_name": "HookedTransformer", "hook_head_index": null, "context_size": 128, "seqpos_slice": [null, null], "model_from_pretrained_kwargs": {}, "prepend_bos": true, "exclude_special_tokens": false, "sequence_separator_token": "bos", "disable_concat_sequences": false}, "decoder_init_norm": 0.1, "l1_coefficient": 1.0, "lp_norm": 1.0, "l1_warm_up_steps": 0, "architecture": "standard"}

blocks.23.hook_resid_post/sae_weights.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cdcec4b879046520d7621e95ad08e08b73ac458771634fd738164533535a1b60
+size 30670848

blocks.27.hook_resid_post/cfg.json

@@ -0,0 +1 @@
+{"d_in": 3584, "d_sae": 16384, "dtype": "float32", "device": "cuda", "apply_b_dec_to_input": true, "normalize_activations": "none", "reshape_activations": "none", "metadata": {"sae_lens_version": "6.26.2", "sae_lens_training_version": "6.26.2", "dataset_path": "TQRG/DeltaSecommits_qwen-2.5-7b-instruct_tokenized_v2_vulnerable", "hook_name": "blocks.27.hook_resid_post", "model_name": "Qwen/Qwen2.5-7B-Instruct", "model_class_name": "HookedTransformer", "hook_head_index": null, "context_size": 128, "seqpos_slice": [null, null], "model_from_pretrained_kwargs": {}, "prepend_bos": true, "exclude_special_tokens": false, "sequence_separator_token": "bos", "disable_concat_sequences": false}, "decoder_init_norm": 0.1, "l1_coefficient": 1.0, "lp_norm": 1.0, "l1_warm_up_steps": 0, "architecture": "standard"}

blocks.27.hook_resid_post/sae_weights.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c53cb38ed82fb2411fe81a11928dd47a85fd9c307fad2632ceb6470de7520c0e
+size 30670848

blocks.3.hook_resid_post/cfg.json

@@ -0,0 +1 @@
+{"d_in": 3584, "d_sae": 16384, "dtype": "float32", "device": "cuda", "apply_b_dec_to_input": true, "normalize_activations": "none", "reshape_activations": "none", "metadata": {"sae_lens_version": "6.26.2", "sae_lens_training_version": "6.26.2", "dataset_path": "TQRG/DeltaSecommits_qwen-2.5-7b-instruct_tokenized_v2_vulnerable", "hook_name": "blocks.3.hook_resid_post", "model_name": "Qwen/Qwen2.5-7B-Instruct", "model_class_name": "HookedTransformer", "hook_head_index": null, "context_size": 128, "seqpos_slice": [null, null], "model_from_pretrained_kwargs": {}, "prepend_bos": true, "exclude_special_tokens": false, "sequence_separator_token": "bos", "disable_concat_sequences": false}, "decoder_init_norm": 0.1, "l1_coefficient": 1.0, "lp_norm": 1.0, "l1_warm_up_steps": 0, "architecture": "standard"}

blocks.3.hook_resid_post/sae_weights.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4f0742bd64fdb692e53aff58f8f40c178959fd039f73a8746e24ccc9e511673a
+size 30670848

blocks.7.hook_resid_post/cfg.json

@@ -0,0 +1 @@
+{"d_in": 3584, "d_sae": 16384, "dtype": "float32", "device": "cuda", "apply_b_dec_to_input": true, "normalize_activations": "none", "reshape_activations": "none", "metadata": {"sae_lens_version": "6.26.2", "sae_lens_training_version": "6.26.2", "dataset_path": "TQRG/DeltaSecommits_qwen-2.5-7b-instruct_tokenized_v2_vulnerable", "hook_name": "blocks.7.hook_resid_post", "model_name": "Qwen/Qwen2.5-7B-Instruct", "model_class_name": "HookedTransformer", "hook_head_index": null, "context_size": 128, "seqpos_slice": [null, null], "model_from_pretrained_kwargs": {}, "prepend_bos": true, "exclude_special_tokens": false, "sequence_separator_token": "bos", "disable_concat_sequences": false}, "decoder_init_norm": 0.1, "l1_coefficient": 1.0, "lp_norm": 1.0, "l1_warm_up_steps": 0, "architecture": "standard"}

blocks.7.hook_resid_post/sae_weights.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:25e2272917f2fc0a853fa1f49e125662887e8110219d3865df48ec2a654d32c5
+size 30670848