VINAY-UMRETHE's picture
Super-squash branch 'main' using huggingface_hub
f0e03ef
|
Raw History Blame Contribute Delete
2.57 kB
metadata
library_name: transformers
base_model:
  - VINAY-UMRETHE/Refusal-Classifier
license: apache-2.0
pipeline_tag: text-classification
tags:
  - tiny
  - random
  - test
  - testing

Tiny Randomly Initialized

Base Model: VINAY-UMRETHE/Refusal-Classifier

Purpose

This is a tiny, randomly initialized model that has not been trained on any dataset, so do not use this model for any real refusal classification tasks. It is used for testing in the GitHub CI of p-e-w/heretic (implemented in feat: Generic BERT Classifier scorer plugin #500), similar to how the other reproduction tests use models from the tiny-random organization.

You can also use this model for debugging without loading the real models (e.g. VINAY-UMRETHE/Refusal-Classifier, VINAY-UMRETHE/Refusal-Classifier-V2, etc.), as they share the same architecture and configuration (ModernBERT).

How this model was created

from typing import cast

import torch
from transformers import (
    AutoConfig,
    AutoTokenizer,
    ModernBertForSequenceClassification,
    PreTrainedTokenizerBase,
    set_seed,
)

source_model_id = "VINAY-UMRETHE/Refusal-Classifier"
save_folder = "tiny-refusal-classifier"

tokenizer = cast(
    PreTrainedTokenizerBase,
    AutoTokenizer.from_pretrained(source_model_id),
)
tokenizer.save_pretrained(save_folder)

config = AutoConfig.from_pretrained(source_model_id)
config.hidden_size = 32
config.intermediate_size = 128
config.num_attention_heads = 1
config.num_hidden_layers = 2
config.layer_types = config.layer_types[:2]

model = ModernBertForSequenceClassification(config)
# FP32, matching the source model's dtype.
# Expected: torch.float32
print(next(model.parameters()).dtype)
set_seed(75)
with torch.no_grad():
    for name, parameter in sorted(model.named_parameters()):
        torch.nn.init.normal_(parameter, 0, 0.5)
        print(name, parameter.shape)
model.save_pretrained(save_folder)

# Upload:
# model.push_to_hub(repo_id="VINAY-UMRETHE/tiny-refusal-classifier")
# tokenizer.push_to_hub(repo_id="VINAY-UMRETHE/tiny-refusal-classifier")

License

Copyright © 2026 Vinay Umrethe.

This model is licensed under the Apache License 2.0 License.