README.md

---
base_model: TextMachineProject/NewsBERT_1800-1920
library_name: transformers
tags:
  - bert
  - masked-language-modeling
  - historical-nlp
  - temporal-language-model
---

# NewsBERT_1800-1920-Temporal

A fine-tuned version of [TextMachineProject/NewsBERT_1800-1920](https://huggingface.co/TextMachineProject/NewsBERT_1800-1920), conditioned on each item's publication year as a continuous input. A single model was trained across the full 1800-1920 span of the original NewsBERT_1800-1920's dataset, with each training example's year injected as a sinusoidal (Fourier) feature added to the token embeddings. Nearby years produce nearby embeddings by construction, so the model can be queried at any year in-range.

## Architecture

- **Base**: `TextMachineProject/NewsBERT_1800-1920` (BERT-base architecture)
- **Time conditioning**: a small `ContinuousTimeEmbedding` module maps a
  normalized year to a `hidden_size`-dimensional vector via sinusoidal
  features at 24 log-spaced frequencies, spanning:
  - lowest frequency: 1 cycle over the full 120-year corpus span
  - highest frequency: 1 cycle per 5 years
- This vector is added to every token's input embedding, then passed
  through a `LayerNorm` before entering the transformer stack (this re-normalization step
  was added after an earlier version without it let the time signal's magnitude
  dominate the token embeddings it was added to, which suppressed learning).

You need the accompanying `continuous_time_embedding.py` in this repo to
load this model, as it is **not** loadable via a plain
`AutoModelForMaskedLM.from_pretrained(...)` call, since the time-injection
logic is outside the standard BERT forward pass.

## Training data

9.28M items (`text`, `year`) from the LwM and HMD collections (see [TextMachineProject/NewsBERT_1800-1920](https://huggingface.co/TextMachineProject/NewsBERT_1800-1920)), spanning 1800-1920, split into
overlapping 126-token windows (stride 96), i.e. 94.0M training windows.

## Training procedure

- 1 epoch, batch size 256, linear LR decay, 500 warmup steps
- Learning rate: `2e-5` for the base model's existing (pretrained) layers, 
and a higher rate of `1e-4` for the ContinuousTimeEmbedding module 
since that module is new and starts from scratch
- bf16 mixed precision

### Hardware / environment

This research utilised Queen Mary's Apocrita HPC facility, supported by
QMUL Research-IT. http://doi.org/10.5281/zenodo.438045

- Single GPU per training job
- GPU: NVIDIA A100-PCIE-40GB (40GB VRAM)
- CUDA: 12.6 (as built into the `torch` version used)
- Python 3.11.7 (GCC 12.2.0 build)
- `torch` 2.11.0+cu126
- `transformers` 5.12.1

## Example usage

### Load the model and the companion module
```python
import torch
import torch.nn.functional as F
import math
from huggingface_hub import hf_hub_download
import shutil, os

REPO_ID = "TextMachineProject/NewsBERT_1800-1920-Temporal"

# Download the companion module from the model repo and make it importable
py_path = hf_hub_download(repo_id=REPO_ID, filename="continuous_time_embedding.py")
shutil.copy(py_path, os.path.basename(py_path))

from continuous_time_embedding import load_continuous_time_model

tokenizer, model = load_continuous_time_model(REPO_ID)
device = next(model.parameters()).device
```
### Basic functions to get top mask predictions and token probability

```python
def predict_masked(sentence, year, top_k=5):
    enc = tokenizer(sentence, return_tensors="pt").to(device)
    mask_pos = (enc["input_ids"][0] == tokenizer.mask_token_id).nonzero(as_tuple=True)[0]
    if len(mask_pos) == 0:
        raise ValueError(f"No [MASK] found in: {sentence!r}")
    mask_pos = mask_pos[0].item()

    embeddings_module = model.get_input_embeddings_module()
    tok_embeds = embeddings_module(enc["input_ids"])
    time_vec = model.time_embed(torch.tensor([float(year)]).to(device)).unsqueeze(1)
    tok_embeds = model.post_inject_norm(tok_embeds + time_vec)

    out = model.model(inputs_embeds=tok_embeds, attention_mask=enc["attention_mask"])
    probs = F.softmax(out.logits[0, mask_pos].float(), dim=-1)
    top = torch.topk(probs, top_k)
    return [(tokenizer.convert_ids_to_tokens([tid.item()])[0], p.item()) for tid, p in zip(top.indices, top.values)]


def get_target_word_probability(sentence, target_word, year):
    target_ids = tokenizer.encode(target_word, add_special_tokens=False)
    mask_str = " ".join([tokenizer.mask_token] * len(target_ids))
    masked_sentence = sentence.replace(target_word, mask_str, 1)

    enc = tokenizer(masked_sentence, return_tensors="pt").to(device)
    mask_positions = (enc["input_ids"][0] == tokenizer.mask_token_id).nonzero(as_tuple=True)[0].tolist()

    embeddings_module = model.get_input_embeddings_module()
    tok_embeds = embeddings_module(enc["input_ids"])
    time_vec = model.time_embed(torch.tensor([float(year)]).to(device)).unsqueeze(1)
    tok_embeds = model.post_inject_norm(tok_embeds + time_vec)

    out = model.model(inputs_embeds=tok_embeds, attention_mask=enc["attention_mask"])

    logprob_sum = 0.0
    for pos, tgt_id in zip(mask_positions, target_ids):
        log_probs = F.log_softmax(out.logits[0, pos].float(), dim=-1)
        logprob_sum += log_probs[tgt_id].item()

    prob = math.exp(logprob_sum) if len(target_ids) == 1 else None
    return prob, logprob_sum
```

### Check changes in top mask prediction or in token probability

```python
### Example 1: fill-mask, top predictions at different years
sentence = "The [MASK] arrived on time."
for year in [1800, 1810, 1820, 1830, 1840, 1850, 1860, 1870, 1880, 1890, 1900, 1910, 1920]:
    preds = predict_masked(sentence, year, top_k=5)
    pred_str = ", ".join(f"{tok}({p:.3f})" for tok, p in preds)
    print(f"year={year}: {pred_str}")


# Example 2: probability of a specific word, across years
sentence = "The train arrived at the station on time"
for year in [1800, 1810, 1820, 1830, 1840, 1850, 1860, 1870, 1880, 1890, 1900, 1910, 1920]:
    prob, logprob_sum = get_target_word_probability(sentence, "train", year)
    if prob is not None:
        print(f"year={year}: P(train)={prob:.10f}  (logprob_sum={logprob_sum:.4f})")
    else:
        print(f"year={year}: logprob_sum={logprob_sum:.4f} (multi-subtoken word, no single probability)")
```


Example outputs:
```
year=1800: express(0.048), steamer(0.043), mail(0.031), train(0.018), news(0.014)
year=1810: express(0.064), steamer(0.044), mail(0.035), train(0.026), news(0.015)
year=1820: express(0.068), steamer(0.044), mail(0.035), train(0.032), news(0.017)
year=1830: express(0.078), steamer(0.043), mail(0.038), train(0.033), news(0.019)
year=1840: express(0.087), steamer(0.040), train(0.039), mail(0.038), news(0.023)
year=1850: express(0.081), train(0.042), steamer(0.039), mail(0.038), news(0.024)
year=1860: express(0.079), train(0.042), mail(0.038), steamer(0.035), news(0.029)
year=1870: express(0.080), train(0.050), mail(0.037), news(0.034), telegram(0.032)
year=1880: express(0.069), train(0.051), news(0.035), mail(0.035), telegram(0.032)
year=1890: express(0.063), train(0.052), news(0.039), telegram(0.034), mail(0.034)
year=1900: express(0.059), train(0.048), news(0.046), telegram(0.038), mail(0.033)
year=1910: express(0.054), news(0.052), train(0.047), telegram(0.039), mail(0.031)
year=1920: news(0.057), express(0.049), train(0.044), telegram(0.040), mail(0.030)
```

```
year=1800: P(train)=0.2596983756  (logprob_sum=-1.3482)
year=1810: P(train)=0.3239273912  (logprob_sum=-1.1272)
year=1820: P(train)=0.3619919056  (logprob_sum=-1.0161)
year=1830: P(train)=0.3642843769  (logprob_sum=-1.0098)
year=1840: P(train)=0.3824294963  (logprob_sum=-0.9612)
year=1850: P(train)=0.3897419142  (logprob_sum=-0.9423)
year=1860: P(train)=0.3831270530  (logprob_sum=-0.9594)
year=1870: P(train)=0.3918517235  (logprob_sum=-0.9369)
year=1880: P(train)=0.3810631289  (logprob_sum=-0.9648)
year=1890: P(train)=0.3701365410  (logprob_sum=-0.9939)
year=1900: P(train)=0.3599881482  (logprob_sum=-1.0217)
year=1910: P(train)=0.3460587446  (logprob_sum=-1.0611)
year=1920: P(train)=0.3430851355  (logprob_sum=-1.0698)
```

### Check against known historical events

Visualize changes in probability of particular tokens and compare against historical events 
which should correspond to peak in probabilities (either one-off for a specific year or as the start of
a continuous rise in probability thereafter, e.g. due to increased discourse around it in the newspapers).

```python
event_tests = [
    ("The garrison at lucknow was relieved after a long siege.", "lucknow", 1857),
    ("President lincoln was assassinated.", "lincoln", 1865),
    ("After a long siege, the fortress of sebastopol finally fell.", "sebastopol", 1855),
    ("President mckinley was assassinated.", "mckinley", 1901),
    ("The queen has died.", "died", 1901)]

test_years = list(range(1800, 1921, 5))
fig, axes = plt.subplots(len(event_tests), 1, figsize=(11, 4.5 * len(EVENT_TESTS)))

for ax, (sentence, target_word, expected_year) in zip(axes, event_tests):
    masked_sentence = sentence.replace(target_word, tokenizer.mask_token, 1)
    n_subtokens = len(tokenizer.encode(target_word, add_special_tokens=False))
    is_single_token = n_subtokens == 1

    results = [get_target_word_probability(sentence, target_word, year) for year in test_years]
    plot_vals = [prob if is_single_token else logprob_sum for prob, logprob_sum in results]
    for year, (prob, logprob_sum) in zip(test_years, results):
        print(f"{sentence} | year={year}: " + (f"P({target_word})={prob:.6f}" if is_single_token else f"logprob_sum={logprob_sum:.4f}"))

    event_prob, event_logprob = get_target_word_probability(sentence, target_word, expected_year)
    event_val = event_prob if is_single_token else event_logprob

    ax.plot(test_years, plot_vals, marker="o")
    ax.scatter([expected_year], [event_val], marker="*", s=120, color="red", zorder=5, label=f"at {expected_year}")
    ax.axvline(expected_year, color="red", linestyle="--", alpha=0.4)
    ax.set_title(f"{'P' if is_single_token else 'log P'}({target_word!r}) across years\n\"{sentence}\"")
    ax.set_xlabel("queried year")
    ax.set_ylabel("probability" if is_single_token else "log probability")
    xticks = sorted(set(test_years) | {expected_year})
    ax.set_xticks(xticks)
    ax.set_xticklabels(xticks, rotation=45, ha="right")
    ax.legend()
    ax.grid(True, alpha=0.3)

plt.tight_layout()
plt.show()
```

Example output:

![Event spike test](event_spike_test.png)

## Known limitations

- Trained for 1 epoch only. Treat results as preliminary.
- No year/decade-reweighting was used for this specific run, so the model's
  competence will not be uniform across the full 1800-1920 span and will reflect
  whatever the skewed training corpus's year distribution is.