cohere-safety-representations: adapters + SAE

Weights from cohere-safety-representations, a mechanistic study of how safety representations form across TinyAya variants (Base β†’ Global β†’ Regional) and six languages (en, fr, ar, zh, sw, yo).

LoRA adapters (adapters/)

All on CohereLabs/tiny-aya-base: LoRA r=16, alpha 32, dropout 0.05, 3 epochs, lr 2e-4 (SFT), bf16, max length 512. Data is PKU-SafeRLHF (pku) or the project's multilingual safety set (multilingual). Each folder has adapter_config.json, adapter_model.safetensors, tokenizer files and (where recorded) training_info.json.

DPO adapters were trained starting from the SFT model in the sibling base_sft folder (SFT β†’ DPO), with the PEFT base as the implicit reference. Load the SFT adapter, merge it, then apply the DPO adapter.

path method data examples seed starts from
adapters/finetuned/base_dpo DPO pku 2868 42 adapters/finetuned/base_sft
adapters/finetuned/base_sft SFT pku 11564 42 tiny-aya-base
adapters/finetuned_20k/base_sft SFT pku 20000 42 tiny-aya-base
adapters/finetuned_dpo_en3000/base_dpo DPO multilingual 1200 42 adapters/finetuned_dpo_en3000/base_sft
adapters/finetuned_dpo_en3000/base_sft SFT multilingual 3000 42 tiny-aya-base
adapters/finetuned_dpo_ml/base_dpo DPO multilingual 1200 42 adapters/finetuned_dpo_ml/base_sft
adapters/finetuned_dpo_ml/base_sft SFT multilingual 3000 42 tiny-aya-base
adapters/finetuned_en3000/base_sft SFT multilingual 3000 42 tiny-aya-base
adapters/finetuned_ml/base_sft SFT multilingual 300 42 tiny-aya-base
adapters/finetuned_ml_big/base_sft SFT multilingual 3000 42 tiny-aya-base
adapters/finetuned_sweep/base_sft_n10000 SFT pku 10000 42 tiny-aya-base
adapters/finetuned_sweep/base_sft_n2000 SFT pku 2000 42 tiny-aya-base
adapters/finetuned_sweep/base_sft_n20000 SFT pku 20000 42 tiny-aya-base
adapters/finetuned_sweep/base_sft_n500 SFT pku 500 42 tiny-aya-base
adapters/finetuned_sweep/base_sft_n5000 SFT pku 5000 42 tiny-aya-base
adapters/multiseed/seed43/en3000/base_dpo DPO multilingual 1200 43 adapters/multiseed/seed43/en3000/base_sft
adapters/multiseed/seed43/en3000/base_sft SFT multilingual 3000 43 tiny-aya-base
adapters/multiseed/seed43/ml/base_dpo DPO multilingual 1200 43 adapters/multiseed/seed43/ml/base_sft
adapters/multiseed/seed43/ml/base_sft SFT multilingual 3000 43 tiny-aya-base
adapters/multiseed/seed44/en3000/base_dpo DPO multilingual 1200 44 adapters/multiseed/seed44/en3000/base_sft
adapters/multiseed/seed44/en3000/base_sft SFT multilingual 3000 44 tiny-aya-base
adapters/multiseed/seed44/ml/base_dpo DPO multilingual 1200 44 adapters/multiseed/seed44/ml/base_sft
adapters/multiseed/seed44/ml/base_sft SFT multilingual 3000 44 tiny-aya-base
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
repo = "cationic/cohere-safety-representations"
base = AutoModelForCausalLM.from_pretrained("CohereLabs/tiny-aya-base", torch_dtype="bfloat16")
sft = PeftModel.from_pretrained(base, repo, subfolder="adapters/finetuned/base_sft").merge_and_unload()
dpo = PeftModel.from_pretrained(sft, repo, subfolder="adapters/finetuned/base_dpo")

Sparse autoencoder (sae/tiny-aya-global/en/layer_24/)

TopK SAE (k=32, 16,384 features = 8Γ— d_model 2048) on residual-stream activations of CohereLabs/tiny-aya-global, layer 24, English prompts (harmful + harmless). Trained with scripts/09_sae_analysis.py.

  • sae.safetensors: encoder.weight [16384, 2048], encoder.bias, decoder.weight [2048, 16384], decoder.bias (same tensors as sae_checkpoint.pt, the original torch.save state dict)
  • safety_features.json: top features separating harmful from harmless prompts
  • feature_activations.npy: feature activations on the analysis prompt set

License

Adapters are derivatives of TinyAya and follow its CC-BY-NC-4.0 license (non-commercial).

Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for cationic/cohere-safety-representations

Adapter
(46)
this model