| --- |
| language: |
| - en |
| tags: |
| - causal-lm |
| - llama |
| - point-in-time |
| - dated |
| - lookahead-bias-free |
| pipeline_tag: text-generation |
| --- |
| |
| # DatedGPT-2013 (base) |
|
|
| **DatedGPT** is a family of point-in-time language models: each vintage is |
| trained only on data available up to its cutoff date, making it suitable for |
| lookahead-bias-free prediction and point-in-time analysis. |
|
|
| This is the **base (pretrained) model** with data up to **2013** — no |
| instruction tuning. For the instruction-tuned variants, see the |
| `datedgpt-instruct-*` repositories in this organization. |
|
|
| | Property | Value | |
| |----------|-------| |
| | Architecture | LlamaForCausalLM | |
| | Parameters | ~1.3 B | |
| | Context length | 2048 | |
| | Vocab | 32,000 (SentencePiece) | |
| | Precision | bfloat16 | |
| | Data vintage | 2013 | |
|
|
| ## Usage |
|
|
| ```python |
| import torch |
| from transformers import AutoTokenizer, AutoModelForCausalLM |
| |
| repo_id = "datedgpt/datedgpt-2013-base" |
| tokenizer = AutoTokenizer.from_pretrained(repo_id) |
| model = AutoModelForCausalLM.from_pretrained(repo_id, torch_dtype=torch.bfloat16, device_map="auto") |
| |
| inputs = tokenizer("The stock market in 2013", return_tensors="pt").to(model.device) |
| output = model.generate(**inputs, max_new_tokens=64, use_cache=True) |
| print(tokenizer.decode(output[0], skip_special_tokens=True)) |
| ``` |
|
|
| ## Limitations |
|
|
| - Base model: completions only, no chat/instruction following. |
| - Knowledge limited to the 2013 data vintage. |
| - No RLHF or safety tuning. |
|
|
