Upload melville stylometry model

README.md

@@ -0,0 +1,215 @@
+---
+language: en
+license: mit
+tags:
+- text-generation
+- gpt2
+- stylometry
+- melville
+- authorship-attribution
+- literary-analysis
+- computational-linguistics
+datasets:
+- contextlab/melville-corpus
+library_name: transformers
+pipeline_tag: text-generation
+---
+
+# GPT-2 Herman Melville Stylometry Model
+
+<div style="text-align: center;">
+  <img src="https://raw.githubusercontent.com/ContextLab/llm-stylometry/main/assets/CDL_Avatar.png" alt="Context Lab" width="200"/>
+</div>
+
+## Overview
+
+This model is a GPT-2 language model trained exclusively on the complete works of **Herman Melville** (1819-1891). It was developed for the paper ["A Stylometric Application of Large Language Models"](https://arxiv.org/abs/2510.21958) (Stropkay et al., 2025).
+
+The model captures Herman Melville's unique writing style through intensive training on their complete corpus. By learning the statistical patterns, vocabulary, syntax, and thematic elements characteristic of Melville's writing, this model enables:
+
+- **Text generation** in the authentic style of Herman Melville
+- **Authorship attribution** through cross-entropy loss comparison
+- **Stylometric analysis** of literary works from 19th-century America
+- **Computational literary studies** exploring Melville's distinctive voice
+
+This model is part of a suite of 8 author-specific models developed to demonstrate that language model perplexity can serve as a robust measure of stylistic similarity.
+
+**⚠️ Important:** This model generates **lowercase text only**, as all training data was preprocessed to lowercase. Use lowercase prompts for best results.
+
+## Model Details
+
+- **Model type:** GPT-2 (custom compact architecture)
+- **Language:** English (lowercase)
+- **License:** MIT
+- **Author:** Herman Melville (1819-1891)
+- **Notable works:** Moby-Dick, Bartleby the Scrivener
+- **Training data:** [10 books by Herman Melville](https://huggingface.co/datasets/contextlab/melville-corpus)
+- **Training tokens:** 1,314,470
+- **Final training loss:** 1.4666
+- **Epochs trained:** 50,000
+
+### Architecture
+
+| Parameter | Value |
+|-----------|-------|
+| Layers | 8 |
+| Embedding dimension | 128 |
+| Attention heads | 8 |
+| Context length | 1024 tokens |
+| Vocabulary size | 50,257 (GPT-2 tokenizer) |
+| Total parameters | ~8.1M |
+
+## Usage
+
+### Basic Text Generation
+
+```python
+from transformers import GPT2LMHeadModel, GPT2Tokenizer
+import torch
+
+# Load model and tokenizer
+model = GPT2LMHeadModel.from_pretrained("contextlab/gpt2-melville")
+tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
+tokenizer.pad_token = tokenizer.eos_token
+
+# IMPORTANT: Use lowercase prompts (model trained on lowercase text)
+prompt = "call me ishmael"
+inputs = tokenizer(prompt, return_tensors="pt")
+
+# Generate text
+with torch.no_grad():
+    outputs = model.generate(
+        **inputs,
+        max_length=200,
+        do_sample=True,
+        temperature=0.8,
+        top_p=0.9,
+        pad_token_id=tokenizer.eos_token_id
+    )
+
+generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
+print(generated_text)
+```
+
+**Output:** Generates text in Herman Melville's distinctive style (all lowercase).
+
+### Stylometric Analysis
+
+Compare cross-entropy loss across multiple author models to determine authorship:
+
+```python
+from transformers import GPT2LMHeadModel, GPT2Tokenizer
+import torch
+
+# Load models for different authors
+authors = ['austen', 'dickens', 'twain']  # Example subset
+models = {
+    author: GPT2LMHeadModel.from_pretrained(f"contextlab/gpt2-{author}")
+    for author in authors
+}
+
+tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
+
+# Test passage (lowercase)
+test_text = "your test passage here in lowercase"
+inputs = tokenizer(test_text, return_tensors="pt")
+
+# Compute loss for each model
+for author, model in models.items():
+    model.eval()
+    with torch.no_grad():
+        outputs = model(**inputs, labels=inputs['input_ids'])
+        loss = outputs.loss.item()
+    print(f"{author}: {loss:.4f}")
+
+# Lower loss indicates more similar style (likely author)
+```
+
+## Training Procedure
+
+### Dataset
+
+The model was trained on the complete works of Herman Melville sourced from [Project Gutenberg](https://www.gutenberg.org/). The text was preprocessed to:
+- Remove Project Gutenberg headers and footers
+- Convert all text to lowercase
+- Remove chapter headings and non-narrative text
+- Preserve punctuation and structure
+
+See the [Melville corpus dataset](https://huggingface.co/datasets/contextlab/melville-corpus) for details.
+
+### Hyperparameters
+
+| Parameter | Value |
+|-----------|-------|
+| Context length | 1,024 tokens |
+| Batch size | 16 |
+| Learning rate | 5×10⁻⁵ |
+| Optimizer | AdamW |
+| Training tokens | 1,314,470 |
+| Epochs | 50,000 |
+| Final loss | 1.4666 |
+
+### Training Method
+
+The model was initialized with a compact GPT-2 architecture (8 layers, 128-dimensional embeddings) and trained exclusively on Herman Melville's works until reaching a training loss of approximately 1.4666. This intensive training enables the model to capture fine-grained stylistic patterns characteristic of Melville's writing.
+
+See the [GitHub repository](https://github.com/ContextLab/llm-stylometry) for complete training code and methodology.
+
+## Intended Use
+
+### Primary Uses
+- **Research:** Stylometric analysis, authorship attribution studies
+- **Education:** Demonstrations of computational stylometry
+- **Creative:** Generate text in Herman Melville's style
+- **Analysis:** Compare writing styles across historical periods
+
+### Out-of-Scope Uses
+This model is not intended for:
+- Factual information retrieval
+- Modern language generation
+- Tasks requiring uppercase text
+- Commercial publication without attribution
+
+## Limitations
+
+- **Lowercase only:** All generated text is lowercase (due to preprocessing)
+- **Historical language:** Reflects 19th-century America vocabulary and grammar
+- **Training data bias:** Limited to Herman Melville's published works
+- **Small model:** Compact architecture prioritizes training speed over generation quality
+- **No factual grounding:** Generates stylistically similar text, not historically accurate content
+
+## Evaluation
+
+This model achieved perfect accuracy (100%) in distinguishing Herman Melville's works from seven other classic authors in cross-entropy loss comparisons. See the paper for detailed evaluation results.
+
+## Citation
+
+If you use this model in your research, please cite:
+
+```bibtex
+@article{StroEtal25,
+  title={A Stylometric Application of Large Language Models},
+  author={Stropkay, Harrison F. and Chen, Jiayi and Jabelli, Mohammad J. L. and Rockmore, Daniel N. and Manning, Jeremy R.},
+  journal={arXiv preprint arXiv:2510.21958},
+  year={2025}
+}
+```
+
+## Contact
+
+- **Paper & Code:** https://github.com/ContextLab/llm-stylometry
+- **Issues:** https://github.com/ContextLab/llm-stylometry/issues
+- **Contact:** Jeremy R. Manning (jeremy.r.manning@dartmouth.edu)
+- **Lab:** [Context Lab](https://www.context-lab.com/), Dartmouth College
+
+## Related Models
+
+Explore models for all 8 authors in the study:
+- [Jane Austen](https://huggingface.co/contextlab/gpt2-austen)
+- [L. Frank Baum](https://huggingface.co/contextlab/gpt2-baum)
+- [Charles Dickens](https://huggingface.co/contextlab/gpt2-dickens)
+- [F. Scott Fitzgerald](https://huggingface.co/contextlab/gpt2-fitzgerald)
+- [Herman Melville](https://huggingface.co/contextlab/gpt2-melville)
+- [Ruth Plumly Thompson](https://huggingface.co/contextlab/gpt2-thompson)
+- [Mark Twain](https://huggingface.co/contextlab/gpt2-twain)
+- [H.G. Wells](https://huggingface.co/contextlab/gpt2-wells)

config.json

@@ -0,0 +1,31 @@
+{
+  "activation_function": "gelu_new",
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.1,
+  "bos_token_id": 50256,
+  "dtype": "float32",
+  "embd_pdrop": 0.1,
+  "eos_token_id": 50256,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_embd": 128,
+  "n_head": 8,
+  "n_inner": null,
+  "n_layer": 8,
+  "n_positions": 1024,
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.1,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "transformers_version": "4.56.1",
+  "use_cache": true,
+  "vocab_size": 50257
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 50256,
+  "eos_token_id": 50256,
+  "transformers_version": "4.56.1"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a35e3da2c2002c21c32c6698d70c14c2a795ad713c4eeadf355525bbfc75f6e1
+size 32611312

training_state.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:25120c3c03687915217567aa5f0aebf3158cade77f1b742eb1ba16e73e7d77e0
+size 65304983