---
license: apache-2.0
datasets:
- mikasenghaas/wikitext-2
language:
- en
pipeline_tag: text-generation
---

# OPT-125m Fine-tuned on WikiText-2 for Language Modeling

This repository contains the weights for the `facebook/opt-125m` model after fine-tuning on the `wikitext-2-raw-v1` configuration of the WikiText dataset using a standard Causal Language Modeling (CLM) objective.

## Model Description

This model is a version of the OPT (Open Pre-trained Transformer) architecture with 125 million parameters. It was fine-tuned solely on the WikiText-2 dataset. The primary goal of this fine-tuning process, as executed by the training script, was to adapt the model to the language patterns, vocabulary, and style present in the WikiText-2 corpus by learning to predict the next token in a sequence.

**Note on Project Context:** While this fine-tuning experiment was conducted as part of a larger initiative by "天算AI" / "SafeSky AI" aiming to develop AI Safety models, **this specific model does not possess specialized AI safety capabilities.** It was trained on a general-purpose dataset (WikiText-2) using a standard language modeling task, and should be considered a foundational experiment in the model fine-tuning process rather than a dedicated AI Safety model.

## Intended Uses & Limitations

**Intended Use:**
*   **Text Generation:** Generating English text that statistically resembles the style and content of the WikiText-2 dataset (e.g., encyclopedic, factual-style prose). It can be used for text continuation or completion based on a given prompt.
*   **Language Modeling Research:** Serving as a benchmark or an example for standard causal language model fine-tuning on the WikiText-2 dataset.
*   **Educational Purposes:** Understanding the fine-tuning process and the behavior of smaller OPT models.

**Limitations:**
*   **Lack of Safety Features:** This model has **not** been trained on safety-specific data or with safety-oriented objectives. It **cannot** reliably identify or avoid generating harmful, biased, unethical, or factually incorrect ("hallucinatory") content beyond what might be implicitly learned (or avoided) from the relatively neutral WikiText-2 data. It is **not** robust against adversarial prompts.
*   **General Capabilities:** As a 125m parameter model fine-tuned on a specific corpus, its general knowledge, reasoning abilities, and performance on tasks significantly different from WikiText-2's domain might be limited.
*   **Potential Biases:** It may inherit biases present in the original OPT-125m model or the WikiText-2 dataset.

## How to Use

You can use this model with the Hugging Face `transformers` library for text generation:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load model and tokenizer
model_id = "jinv2/opt125m-wikitext2-finetuned"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

# Set device
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)

# Example prompt
prompt = "Artificial intelligence is transforming the world by"

# Tokenize prompt
inputs = tokenizer(prompt, return_tensors="pt").to(device)

# Generate text
output_sequences = model.generate(
    input_ids=inputs["input_ids"],
    attention_mask=inputs["attention_mask"],
    max_length=60,
    num_return_sequences=1,
    do_sample=True,
    temperature=0.7,
    top_k=50
)

# Decode and print
generated_text = tokenizer.decode(output_sequences[0], skip_special_tokens=True)
print(generated_text)
```

## Training Procedure

This model was fine-tuned from `facebook/opt-125m` on the `wikitext-2-raw-v1` dataset using the Causal Language Modeling task with the Hugging Face `Trainer`. Key settings likely included a learning rate around `2e-5`, standard text tokenization with padding and truncation, and training for a small number of epochs. *(You can add more specific details here if you remember them)*

---

*This model represents an early experimental step within the broader AI Safety initiative of "天算AI" / "SafeSky AI". Future work will involve using safety-focused datasets and training methodologies.*
```