--- base_model: TextMachineProject/NewsBERT_1800-1920 library_name: transformers tags: - bert - masked-language-modeling - historical-nlp - temporal-language-model --- # NewsBERT_1800-1920-Temporal A fine-tuned version of [TextMachineProject/NewsBERT_1800-1920](https://huggingface.co/TextMachineProject/NewsBERT_1800-1920), conditioned on each item's publication year as a continuous input. A single model was trained across the full 1800-1920 span of the original NewsBERT_1800-1920's dataset, with each training example's year injected as a sinusoidal (Fourier) feature added to the token embeddings. Nearby years produce nearby embeddings by construction, so the model can be queried at any year in-range. ## Architecture - **Base**: `TextMachineProject/NewsBERT_1800-1920` (BERT-base architecture) - **Time conditioning**: a small `ContinuousTimeEmbedding` module maps a normalized year to a `hidden_size`-dimensional vector via sinusoidal features at 24 log-spaced frequencies, spanning: - lowest frequency: 1 cycle over the full 120-year corpus span - highest frequency: 1 cycle per 5 years - This vector is added to every token's input embedding, then passed through a `LayerNorm` before entering the transformer stack (this re-normalization step was added after an earlier version without it let the time signal's magnitude dominate the token embeddings it was added to, which suppressed learning). You need the accompanying `continuous_time_embedding.py` in this repo to load this model, as it is **not** loadable via a plain `AutoModelForMaskedLM.from_pretrained(...)` call, since the time-injection logic is outside the standard BERT forward pass. ## Training data 9.28M items (`text`, `year`) from the LwM and HMD collections (see [TextMachineProject/NewsBERT_1800-1920](https://huggingface.co/TextMachineProject/NewsBERT_1800-1920)), spanning 1800-1920, split into overlapping 126-token windows (stride 96), i.e. 94.0M training windows. ## Training procedure - 1 epoch, batch size 256, linear LR decay, 500 warmup steps - Learning rate: `2e-5` for the base model's existing (pretrained) layers, and a higher rate of `1e-4` for the ContinuousTimeEmbedding module since that module is new and starts from scratch - bf16 mixed precision ### Hardware / environment This research utilised Queen Mary's Apocrita HPC facility, supported by QMUL Research-IT. http://doi.org/10.5281/zenodo.438045 - Single GPU per training job - GPU: NVIDIA A100-PCIE-40GB (40GB VRAM) - CUDA: 12.6 (as built into the `torch` version used) - Python 3.11.7 (GCC 12.2.0 build) - `torch` 2.11.0+cu126 - `transformers` 5.12.1 ## Example usage ### Load the model and the companion module ```python import torch import torch.nn.functional as F import math from huggingface_hub import hf_hub_download import shutil, os REPO_ID = "TextMachineProject/NewsBERT_1800-1920-Temporal" # Download the companion module from the model repo and make it importable py_path = hf_hub_download(repo_id=REPO_ID, filename="continuous_time_embedding.py") shutil.copy(py_path, os.path.basename(py_path)) from continuous_time_embedding import load_continuous_time_model tokenizer, model = load_continuous_time_model(REPO_ID) device = next(model.parameters()).device ``` ### Basic functions to get top mask predictions and token probability ```python def predict_masked(sentence, year, top_k=5): enc = tokenizer(sentence, return_tensors="pt").to(device) mask_pos = (enc["input_ids"][0] == tokenizer.mask_token_id).nonzero(as_tuple=True)[0] if len(mask_pos) == 0: raise ValueError(f"No [MASK] found in: {sentence!r}") mask_pos = mask_pos[0].item() embeddings_module = model.get_input_embeddings_module() tok_embeds = embeddings_module(enc["input_ids"]) time_vec = model.time_embed(torch.tensor([float(year)]).to(device)).unsqueeze(1) tok_embeds = model.post_inject_norm(tok_embeds + time_vec) out = model.model(inputs_embeds=tok_embeds, attention_mask=enc["attention_mask"]) probs = F.softmax(out.logits[0, mask_pos].float(), dim=-1) top = torch.topk(probs, top_k) return [(tokenizer.convert_ids_to_tokens([tid.item()])[0], p.item()) for tid, p in zip(top.indices, top.values)] def get_target_word_probability(sentence, target_word, year): target_ids = tokenizer.encode(target_word, add_special_tokens=False) mask_str = " ".join([tokenizer.mask_token] * len(target_ids)) masked_sentence = sentence.replace(target_word, mask_str, 1) enc = tokenizer(masked_sentence, return_tensors="pt").to(device) mask_positions = (enc["input_ids"][0] == tokenizer.mask_token_id).nonzero(as_tuple=True)[0].tolist() embeddings_module = model.get_input_embeddings_module() tok_embeds = embeddings_module(enc["input_ids"]) time_vec = model.time_embed(torch.tensor([float(year)]).to(device)).unsqueeze(1) tok_embeds = model.post_inject_norm(tok_embeds + time_vec) out = model.model(inputs_embeds=tok_embeds, attention_mask=enc["attention_mask"]) logprob_sum = 0.0 for pos, tgt_id in zip(mask_positions, target_ids): log_probs = F.log_softmax(out.logits[0, pos].float(), dim=-1) logprob_sum += log_probs[tgt_id].item() prob = math.exp(logprob_sum) if len(target_ids) == 1 else None return prob, logprob_sum ``` ### Check changes in top mask prediction or in token probability ```python ### Example 1: fill-mask, top predictions at different years sentence = "The [MASK] arrived on time." for year in [1800, 1810, 1820, 1830, 1840, 1850, 1860, 1870, 1880, 1890, 1900, 1910, 1920]: preds = predict_masked(sentence, year, top_k=5) pred_str = ", ".join(f"{tok}({p:.3f})" for tok, p in preds) print(f"year={year}: {pred_str}") # Example 2: probability of a specific word, across years sentence = "The train arrived at the station on time" for year in [1800, 1810, 1820, 1830, 1840, 1850, 1860, 1870, 1880, 1890, 1900, 1910, 1920]: prob, logprob_sum = get_target_word_probability(sentence, "train", year) if prob is not None: print(f"year={year}: P(train)={prob:.10f} (logprob_sum={logprob_sum:.4f})") else: print(f"year={year}: logprob_sum={logprob_sum:.4f} (multi-subtoken word, no single probability)") ``` Example outputs: ``` year=1800: express(0.048), steamer(0.043), mail(0.031), train(0.018), news(0.014) year=1810: express(0.064), steamer(0.044), mail(0.035), train(0.026), news(0.015) year=1820: express(0.068), steamer(0.044), mail(0.035), train(0.032), news(0.017) year=1830: express(0.078), steamer(0.043), mail(0.038), train(0.033), news(0.019) year=1840: express(0.087), steamer(0.040), train(0.039), mail(0.038), news(0.023) year=1850: express(0.081), train(0.042), steamer(0.039), mail(0.038), news(0.024) year=1860: express(0.079), train(0.042), mail(0.038), steamer(0.035), news(0.029) year=1870: express(0.080), train(0.050), mail(0.037), news(0.034), telegram(0.032) year=1880: express(0.069), train(0.051), news(0.035), mail(0.035), telegram(0.032) year=1890: express(0.063), train(0.052), news(0.039), telegram(0.034), mail(0.034) year=1900: express(0.059), train(0.048), news(0.046), telegram(0.038), mail(0.033) year=1910: express(0.054), news(0.052), train(0.047), telegram(0.039), mail(0.031) year=1920: news(0.057), express(0.049), train(0.044), telegram(0.040), mail(0.030) ``` ``` year=1800: P(train)=0.2596983756 (logprob_sum=-1.3482) year=1810: P(train)=0.3239273912 (logprob_sum=-1.1272) year=1820: P(train)=0.3619919056 (logprob_sum=-1.0161) year=1830: P(train)=0.3642843769 (logprob_sum=-1.0098) year=1840: P(train)=0.3824294963 (logprob_sum=-0.9612) year=1850: P(train)=0.3897419142 (logprob_sum=-0.9423) year=1860: P(train)=0.3831270530 (logprob_sum=-0.9594) year=1870: P(train)=0.3918517235 (logprob_sum=-0.9369) year=1880: P(train)=0.3810631289 (logprob_sum=-0.9648) year=1890: P(train)=0.3701365410 (logprob_sum=-0.9939) year=1900: P(train)=0.3599881482 (logprob_sum=-1.0217) year=1910: P(train)=0.3460587446 (logprob_sum=-1.0611) year=1920: P(train)=0.3430851355 (logprob_sum=-1.0698) ``` ### Check against known historical events Visualize changes in probability of particular tokens and compare against historical events which should correspond to peak in probabilities (either one-off for a specific year or as the start of a continuous rise in probability thereafter, e.g. due to increased discourse around it in the newspapers). ```python event_tests = [ ("The garrison at lucknow was relieved after a long siege.", "lucknow", 1857), ("President lincoln was assassinated.", "lincoln", 1865), ("After a long siege, the fortress of sebastopol finally fell.", "sebastopol", 1855), ("President mckinley was assassinated.", "mckinley", 1901), ("The queen has died.", "died", 1901)] test_years = list(range(1800, 1921, 5)) fig, axes = plt.subplots(len(event_tests), 1, figsize=(11, 4.5 * len(EVENT_TESTS))) for ax, (sentence, target_word, expected_year) in zip(axes, event_tests): masked_sentence = sentence.replace(target_word, tokenizer.mask_token, 1) n_subtokens = len(tokenizer.encode(target_word, add_special_tokens=False)) is_single_token = n_subtokens == 1 results = [get_target_word_probability(sentence, target_word, year) for year in test_years] plot_vals = [prob if is_single_token else logprob_sum for prob, logprob_sum in results] for year, (prob, logprob_sum) in zip(test_years, results): print(f"{sentence} | year={year}: " + (f"P({target_word})={prob:.6f}" if is_single_token else f"logprob_sum={logprob_sum:.4f}")) event_prob, event_logprob = get_target_word_probability(sentence, target_word, expected_year) event_val = event_prob if is_single_token else event_logprob ax.plot(test_years, plot_vals, marker="o") ax.scatter([expected_year], [event_val], marker="*", s=120, color="red", zorder=5, label=f"at {expected_year}") ax.axvline(expected_year, color="red", linestyle="--", alpha=0.4) ax.set_title(f"{'P' if is_single_token else 'log P'}({target_word!r}) across years\n\"{sentence}\"") ax.set_xlabel("queried year") ax.set_ylabel("probability" if is_single_token else "log probability") xticks = sorted(set(test_years) | {expected_year}) ax.set_xticks(xticks) ax.set_xticklabels(xticks, rotation=45, ha="right") ax.legend() ax.grid(True, alpha=0.3) plt.tight_layout() plt.show() ``` Example output: ![Event spike test](event_spike_test.png) ## Known limitations - Trained for 1 epoch only. Treat results as preliminary. - No year/decade-reweighting was used for this specific run, so the model's competence will not be uniform across the full 1800-1920 span and will reflect whatever the skewed training corpus's year distribution is.