Covfefe-1
A very tremendous language model. Perhaps the greatest language model ever trained on this kind of data, maybe ever. People are saying it, a lot of people.
Built from scratch: no HuggingFace Trainer, no pretrained weights, no shortcuts. Just PyTorch, a custom BPE tokenizer, and a lot of speeches. Very beautiful speeches.
Model Details
Covfefe-1 is a decoder-only transformer trained exclusively on Donald Trump's public speeches and statements. It was built entirely from scratch as a learning project by hausmire under the Gleggmire AI research division, which is a very powerful organization, one of the best.
| Parameters | 29,676,544 |
| Architecture | GPT-style decoder-only transformer, pre-LayerNorm |
| Layers | 8 |
| Heads | 16 |
| d_model | 512 |
| d_ff | 2048 |
| Context length | 512 tokens |
| Tokenizer | BPE, trained from scratch, 8192 vocab |
| Training tokens | ~7.2M |
| Final loss | 2.75 |
| Framework | PyTorch (from scratch) |
Training procedure
- 8 epochs, batch size 16, block size 512
- AdamW optimizer (betas 0.9/0.95, weight decay 0.1), peak LR 6e-4
- Linear warmup (300 steps) into cosine decay (down to 10% of peak LR)
- Gradient clipping at max norm 1.0
- Weight tying between token embedding and LM head
- Trained in a Colab notebook on a single GPU; seed fixed at 42
Training Data
All training data consists of Donald Trump's publicly available speeches and statements. Public domain. Nobody has better public domain data, frankly. No private data, no scraped user content, nothing shady. Very clean data. The best data.
Example Outputs (This is synthetic text from an AI model, not a real quote or statement.)
Prompt: The Democrats
Output:
The Democrats are being sent out. You know, I've been saying that for many years. They're being sent out by the millions of dollars from China. And you know, they were not taking their money because they took advantage of us on trade. They're not taking our money. And we have a lot of companies coming in. We have a lot of car companies coming in. A lot of companies are pouring into our country. They're pouring in at levels that nobody's ever seen before. Nobody's ever seen anything like it.
So we'll see what happens. But I don't think the press is going to be giving us the right decision. We have a great economy, and we have a tremendous economy. We have the greatest economy in the history of our country.
Prompt: Joe Biden
Output:
Joe Biden idioted the entire nation to shut down your borders, and they are now open for open borders.
I don't know what's happening. It's a big thing. They're trying to figure that out. You can do it, but we're not going to let them have a country that's not going to be allowed to be ripped off anymore. And we're not letting them into our country. We're not allowing them to come in.
And the Democrats are willing to take back the House. I mean, you could have had a big victory today. They would have taken it away from us. They would have given us everything away. They would've given us everything away. If they didn't give us anything, they wouldn't have given us anything.
If I were President, they'd have given us everything away. They would've given us everything away. But the only way you have to go through the Senate, we will get the House
Prompt: the agitators
Output:
the agitators vessels, and they're very bad people. And the Democrats don't want to pay for it. They just don't want to pay for it.
And the Democrats are not doing that, because we have a lot of problems in California. It's like you wouldn't believe. You know why? Because I think we're going to do well in California. I think we're going to do well in California. I think we'll have to see what happens.
Yes.
Well, we can't let them have a nuclear weapon. We can't let that happen. It's a disaster. It's a disaster. We can't let that happen. But I can tell you this: The country is going to be very strong, and we have to do something with Iran.
And I was told yesterday that they were going to have to take out the sanctions off and to put down the sanctions on the oil. They didn't give us anything.
Inference
This model was not built with the transformers library. It's a from-scratch PyTorch architecture (custom attention, tied embeddings, custom generation loop). You cannot load it with AutoModelForCausalLM.
Architecture summary
- Decoder-only transformer, pre-LN (
ln1/ln2applied before attention/MLP) - Causal self-attention with a registered upper-triangular boolean mask
- MLP block:
Linear → GELU → Linear → Dropout - Weight tying between
token_embandlm_head—lm_head.weightis not saved separately inmodel.safetensorssince it's tied; load withstrict=Falseand expect that one key to be reported missing - Learned positional embeddings (
pos_emb), not RoPE/ALiBi
Recommended generation settings
generate(
model, tokenizer, prompt,
max_new_tokens=200,
temperature=0.7,
top_k=40,
top_p=0.85,
repetition_penalty=1.15,
)
These match the settings used to produce the example output above and are a reasonable starting point.
Implementation notes, since this differs from common HF defaults:
- Top-k and top-p are both applied sequentially (top-k first, then top-p on the reduced set).
- Repetition penalty divides logits for any token already seen anywhere in the sequence so far, applied before top-k/p filtering.
- Includes a NaN fallback: if filtering produces an all
-infrow, sampling falls back to uniform rather than erroring.
Intended Use
There is no intended use. Do whatever you want with it: entertainment (if you could call it that), or finetuning it, as long as it's not commercial: that would require talking to us first. And trust me, we make very good deals, the best deals. In fact, I'm the best businessman probably since the history of businessmen. You win, we win - mostly us, but you'll love it anyway.
Discouraged Use
- Do not treat anything this model says as factual. It hallucinates constantly. Tremendously, actually. Any names, numbers, events, or statistics it produces are probably made up.
- Do not use for anything that actually matters.
Capabilities & Limitations
Not instruction-tuned. Covfefe-1 is a raw pretrained model. There is no chat template, no RLHF, no instruction-following fine-tuning of any kind. It only does next-token continuation. This means it cannot reliably be directed to perform a task ("write a speech about X," "declare Y") and will not produce coherent on-demand outputs to a directive. It just continues the prompt in the statistical style of the training data. Don't expect controllable or task-following behavior; this is closer to a stochastic-parrot-style autocomplete than an assistant.
EU AI Act
This model is not a "general-purpose AI model" (GPAI) under the EU AI Act. The Act's GPAI classification threshold targets models trained with more than 10²³ FLOPs and capable of broad generative tasks; Covfefe-1 (29.7M parameters, ~7.2M training tokens, single-GPU Colab run) is many orders of magnitude below that threshold, so the GPAI obligations in Articles 53–55 do not apply.
License
CC-BY-NC-SA 4.0 — credit hausmire, no commercial use, share alike. If you want to commercialize this, which you probably won't, call us. We'll make a deal.
Covfefe-1 is a research and educational project. It is not affiliated with, endorsed by, or in any way connected to Donald Trump, the Trump Organization, or anyone with actual power. Gleggmire AI is not an actual company.
- Downloads last month
- 199