| --- |
| base_model: TextMachineProject/NewsBERT_1800-1920 |
| library_name: transformers |
| tags: |
| - bert |
| - masked-language-modeling |
| - historical-nlp |
| - temporal-language-model |
| --- |
| |
| # NewsBERT_1800-1920-Temporal |
| |
| A fine-tuned version of [TextMachineProject/NewsBERT_1800-1920](https://huggingface.co/TextMachineProject/NewsBERT_1800-1920), conditioned on each item's publication year as a continuous input. A single model was trained across the full 1800-1920 span of the original NewsBERT_1800-1920's dataset, with each training example's year injected as a sinusoidal (Fourier) feature added to the token embeddings. Nearby years produce nearby embeddings by construction, so the model can be queried at any year in-range. |
|
|
| ## Architecture |
|
|
| - **Base**: `TextMachineProject/NewsBERT_1800-1920` (BERT-base architecture) |
| - **Time conditioning**: a small `ContinuousTimeEmbedding` module maps a |
| normalized year to a `hidden_size`-dimensional vector via sinusoidal |
| features at 24 log-spaced frequencies, spanning: |
| - lowest frequency: 1 cycle over the full 120-year corpus span |
| - highest frequency: 1 cycle per 5 years |
| - This vector is added to every token's input embedding, then passed |
| through a `LayerNorm` before entering the transformer stack (this re-normalization step |
| was added after an earlier version without it let the time signal's magnitude |
| dominate the token embeddings it was added to, which suppressed learning). |
|
|
| You need the accompanying `continuous_time_embedding.py` in this repo to |
| load this model, as it is **not** loadable via a plain |
| `AutoModelForMaskedLM.from_pretrained(...)` call, since the time-injection |
| logic is outside the standard BERT forward pass. |
|
|
| ## Training data |
|
|
| 9.28M items (`text`, `year`) from the LwM and HMD collections (see [TextMachineProject/NewsBERT_1800-1920](https://huggingface.co/TextMachineProject/NewsBERT_1800-1920)), spanning 1800-1920, split into |
| overlapping 126-token windows (stride 96), i.e. 94.0M training windows. |
|
|
| ## Training procedure |
|
|
| - 1 epoch, batch size 256, linear LR decay, 500 warmup steps |
| - Learning rate: `2e-5` for the base model's existing (pretrained) layers, |
| and a higher rate of `1e-4` for the ContinuousTimeEmbedding module |
| since that module is new and starts from scratch |
| - bf16 mixed precision |
|
|
| ### Hardware / environment |
|
|
| This research utilised Queen Mary's Apocrita HPC facility, supported by |
| QMUL Research-IT. http://doi.org/10.5281/zenodo.438045 |
|
|
| - Single GPU per training job |
| - GPU: NVIDIA A100-PCIE-40GB (40GB VRAM) |
| - CUDA: 12.6 (as built into the `torch` version used) |
| - Python 3.11.7 (GCC 12.2.0 build) |
| - `torch` 2.11.0+cu126 |
| - `transformers` 5.12.1 |
|
|
| ## Example usage |
|
|
| ### Load the model and the companion module |
| ```python |
| import torch |
| import torch.nn.functional as F |
| import math |
| from huggingface_hub import hf_hub_download |
| import shutil, os |
| |
| REPO_ID = "TextMachineProject/NewsBERT_1800-1920-Temporal" |
| |
| # Download the companion module from the model repo and make it importable |
| py_path = hf_hub_download(repo_id=REPO_ID, filename="continuous_time_embedding.py") |
| shutil.copy(py_path, os.path.basename(py_path)) |
| |
| from continuous_time_embedding import load_continuous_time_model |
| |
| tokenizer, model = load_continuous_time_model(REPO_ID) |
| device = next(model.parameters()).device |
| ``` |
| ### Basic functions to get top mask predictions and token probability |
|
|
| ```python |
| def predict_masked(sentence, year, top_k=5): |
| enc = tokenizer(sentence, return_tensors="pt").to(device) |
| mask_pos = (enc["input_ids"][0] == tokenizer.mask_token_id).nonzero(as_tuple=True)[0] |
| if len(mask_pos) == 0: |
| raise ValueError(f"No [MASK] found in: {sentence!r}") |
| mask_pos = mask_pos[0].item() |
| |
| embeddings_module = model.get_input_embeddings_module() |
| tok_embeds = embeddings_module(enc["input_ids"]) |
| time_vec = model.time_embed(torch.tensor([float(year)]).to(device)).unsqueeze(1) |
| tok_embeds = model.post_inject_norm(tok_embeds + time_vec) |
| |
| out = model.model(inputs_embeds=tok_embeds, attention_mask=enc["attention_mask"]) |
| probs = F.softmax(out.logits[0, mask_pos].float(), dim=-1) |
| top = torch.topk(probs, top_k) |
| return [(tokenizer.convert_ids_to_tokens([tid.item()])[0], p.item()) for tid, p in zip(top.indices, top.values)] |
| |
| |
| def get_target_word_probability(sentence, target_word, year): |
| target_ids = tokenizer.encode(target_word, add_special_tokens=False) |
| mask_str = " ".join([tokenizer.mask_token] * len(target_ids)) |
| masked_sentence = sentence.replace(target_word, mask_str, 1) |
| |
| enc = tokenizer(masked_sentence, return_tensors="pt").to(device) |
| mask_positions = (enc["input_ids"][0] == tokenizer.mask_token_id).nonzero(as_tuple=True)[0].tolist() |
| |
| embeddings_module = model.get_input_embeddings_module() |
| tok_embeds = embeddings_module(enc["input_ids"]) |
| time_vec = model.time_embed(torch.tensor([float(year)]).to(device)).unsqueeze(1) |
| tok_embeds = model.post_inject_norm(tok_embeds + time_vec) |
| |
| out = model.model(inputs_embeds=tok_embeds, attention_mask=enc["attention_mask"]) |
| |
| logprob_sum = 0.0 |
| for pos, tgt_id in zip(mask_positions, target_ids): |
| log_probs = F.log_softmax(out.logits[0, pos].float(), dim=-1) |
| logprob_sum += log_probs[tgt_id].item() |
| |
| prob = math.exp(logprob_sum) if len(target_ids) == 1 else None |
| return prob, logprob_sum |
| ``` |
|
|
| ### Check changes in top mask prediction or in token probability |
|
|
| ```python |
| ### Example 1: fill-mask, top predictions at different years |
| sentence = "The [MASK] arrived on time." |
| for year in [1800, 1810, 1820, 1830, 1840, 1850, 1860, 1870, 1880, 1890, 1900, 1910, 1920]: |
| preds = predict_masked(sentence, year, top_k=5) |
| pred_str = ", ".join(f"{tok}({p:.3f})" for tok, p in preds) |
| print(f"year={year}: {pred_str}") |
| |
| |
| # Example 2: probability of a specific word, across years |
| sentence = "The train arrived at the station on time" |
| for year in [1800, 1810, 1820, 1830, 1840, 1850, 1860, 1870, 1880, 1890, 1900, 1910, 1920]: |
| prob, logprob_sum = get_target_word_probability(sentence, "train", year) |
| if prob is not None: |
| print(f"year={year}: P(train)={prob:.10f} (logprob_sum={logprob_sum:.4f})") |
| else: |
| print(f"year={year}: logprob_sum={logprob_sum:.4f} (multi-subtoken word, no single probability)") |
| ``` |
|
|
|
|
| Example outputs: |
| ``` |
| year=1800: express(0.048), steamer(0.043), mail(0.031), train(0.018), news(0.014) |
| year=1810: express(0.064), steamer(0.044), mail(0.035), train(0.026), news(0.015) |
| year=1820: express(0.068), steamer(0.044), mail(0.035), train(0.032), news(0.017) |
| year=1830: express(0.078), steamer(0.043), mail(0.038), train(0.033), news(0.019) |
| year=1840: express(0.087), steamer(0.040), train(0.039), mail(0.038), news(0.023) |
| year=1850: express(0.081), train(0.042), steamer(0.039), mail(0.038), news(0.024) |
| year=1860: express(0.079), train(0.042), mail(0.038), steamer(0.035), news(0.029) |
| year=1870: express(0.080), train(0.050), mail(0.037), news(0.034), telegram(0.032) |
| year=1880: express(0.069), train(0.051), news(0.035), mail(0.035), telegram(0.032) |
| year=1890: express(0.063), train(0.052), news(0.039), telegram(0.034), mail(0.034) |
| year=1900: express(0.059), train(0.048), news(0.046), telegram(0.038), mail(0.033) |
| year=1910: express(0.054), news(0.052), train(0.047), telegram(0.039), mail(0.031) |
| year=1920: news(0.057), express(0.049), train(0.044), telegram(0.040), mail(0.030) |
| ``` |
|
|
| ``` |
| year=1800: P(train)=0.2596983756 (logprob_sum=-1.3482) |
| year=1810: P(train)=0.3239273912 (logprob_sum=-1.1272) |
| year=1820: P(train)=0.3619919056 (logprob_sum=-1.0161) |
| year=1830: P(train)=0.3642843769 (logprob_sum=-1.0098) |
| year=1840: P(train)=0.3824294963 (logprob_sum=-0.9612) |
| year=1850: P(train)=0.3897419142 (logprob_sum=-0.9423) |
| year=1860: P(train)=0.3831270530 (logprob_sum=-0.9594) |
| year=1870: P(train)=0.3918517235 (logprob_sum=-0.9369) |
| year=1880: P(train)=0.3810631289 (logprob_sum=-0.9648) |
| year=1890: P(train)=0.3701365410 (logprob_sum=-0.9939) |
| year=1900: P(train)=0.3599881482 (logprob_sum=-1.0217) |
| year=1910: P(train)=0.3460587446 (logprob_sum=-1.0611) |
| year=1920: P(train)=0.3430851355 (logprob_sum=-1.0698) |
| ``` |
|
|
| ### Check against known historical events |
|
|
| Visualize changes in probability of particular tokens and compare against historical events |
| which should correspond to peak in probabilities (either one-off for a specific year or as the start of |
| a continuous rise in probability thereafter, e.g. due to increased discourse around it in the newspapers). |
|
|
| ```python |
| event_tests = [ |
| ("The garrison at lucknow was relieved after a long siege.", "lucknow", 1857), |
| ("President lincoln was assassinated.", "lincoln", 1865), |
| ("After a long siege, the fortress of sebastopol finally fell.", "sebastopol", 1855), |
| ("President mckinley was assassinated.", "mckinley", 1901), |
| ("The queen has died.", "died", 1901)] |
| |
| test_years = list(range(1800, 1921, 5)) |
| fig, axes = plt.subplots(len(event_tests), 1, figsize=(11, 4.5 * len(EVENT_TESTS))) |
| |
| for ax, (sentence, target_word, expected_year) in zip(axes, event_tests): |
| masked_sentence = sentence.replace(target_word, tokenizer.mask_token, 1) |
| n_subtokens = len(tokenizer.encode(target_word, add_special_tokens=False)) |
| is_single_token = n_subtokens == 1 |
| |
| results = [get_target_word_probability(sentence, target_word, year) for year in test_years] |
| plot_vals = [prob if is_single_token else logprob_sum for prob, logprob_sum in results] |
| for year, (prob, logprob_sum) in zip(test_years, results): |
| print(f"{sentence} | year={year}: " + (f"P({target_word})={prob:.6f}" if is_single_token else f"logprob_sum={logprob_sum:.4f}")) |
| |
| event_prob, event_logprob = get_target_word_probability(sentence, target_word, expected_year) |
| event_val = event_prob if is_single_token else event_logprob |
| |
| ax.plot(test_years, plot_vals, marker="o") |
| ax.scatter([expected_year], [event_val], marker="*", s=120, color="red", zorder=5, label=f"at {expected_year}") |
| ax.axvline(expected_year, color="red", linestyle="--", alpha=0.4) |
| ax.set_title(f"{'P' if is_single_token else 'log P'}({target_word!r}) across years\n\"{sentence}\"") |
| ax.set_xlabel("queried year") |
| ax.set_ylabel("probability" if is_single_token else "log probability") |
| xticks = sorted(set(test_years) | {expected_year}) |
| ax.set_xticks(xticks) |
| ax.set_xticklabels(xticks, rotation=45, ha="right") |
| ax.legend() |
| ax.grid(True, alpha=0.3) |
| |
| plt.tight_layout() |
| plt.show() |
| ``` |
|
|
| Example output: |
|
|
|  |
|
|
| ## Known limitations |
|
|
| - Trained for 1 epoch only. Treat results as preliminary. |
| - No year/decade-reweighting was used for this specific run, so the model's |
| competence will not be uniform across the full 1800-1920 span and will reflect |
| whatever the skewed training corpus's year distribution is. |
