---
license: cc-by-sa-4.0
datasets:
- rtc2022/corpus-kirkensis
language:
- en
library_name: transformers
tags:
- weird
- tiny
- llama
---

# Kirk Tung

![A humorous collage. A person stands in front of a background of blended colors similar to a camoufladged pattern. The person is holding two signs saying: "God Hates Kirk" and "Claude's Doom", respectively. They have something written on their blue shirt: "God Hates RAGs". They hold an uncanny resemblance to a famous atheist philosopher.](quirky.png)

Kirk Tung is a tiny language model trained on the [Corpus Kirkensis](https://huggingface.co/datasets/rtc2022/corpus-kirkensis), meant to produce
output which is entertainingly silly as an experiment in using the Llama architecture.

Tung has a Libre and Open dataset, meaning that the content it was trained on is nearly 100% under a free license, with few exceptions.

## Technical details

Kirk Tung was trained on 1.1B tokens over the course of 4 hours on a single A100 GPU.

| Name           | Value |
| -------------- | ----- |
| Architecture   | Llama |
| Context Length | 1024  |
| Vocab size     | 32606 |
| RoPE theta     | 10000 |

```
num_attention_heads = 8
num_key_value_heads = 2
num_hidden_layers = 10
hidden_size = 512
intermediate_size = 1728

tie_word_embeddings = True
```

### Training hyperparameters

| Name                        | Value  |
| --------------------------- | ------ |
| Per-device batch size       | 12     |
| Learning rate               | 5e-4   |
| Learning scheduler          | Cosine |
| Weight decay                | 0.1    |
| Epochs                      | 1      |
| Gradient accumulation steps | 8      |

## Example completions

**Prompt:** "Once upon a time,"

> Once upon a time, the two brothers would be able to use their powers to defeat the first."
> 
> Rosa was a very wise man. He had a love for a woman and a friend. He was so strong, he felt very happy that he had to be.
> 
> "You never thought you could be like you. I am a woman, a man who never wanted to go to the house, and never wanted to go to the house that night."
> 
> The two brothers sat in silence. The three brothers had a good laugh. "Birds, I can see you. Look for me."
> 
> The brothers tried to raise their hands. "I won't," they say. "You don't have to be a mother. I will be a mother. You don't have to be a mother," she says.
> 
> The brothers moved into their respective houses. The brothers were very happy. They were always happy. They always thought of it.
> 
> The brothers were very happy. The brothers knew that they would never have to go back home. They did not want to be friends. They never wanted to be alone again. The brothers were always happy. But they were always happy. They had to be happy. They had no friendship. They would

**Prompt:** "Artificial intelligence is"

> Artificial Intelligence is the source of these data.
>
> Witness the following:
> 
> "1. The United Nations has been stricter in internal affairs, and the agenda has been growing; the people have become more and more bankrupt, the world a power under international law. The foreign powers have become many times more bankrupt, and the foreign powers have become much more constraining of all their financial resources. It has allowed for still more operations than any other nation in the world. The U.S. Government has found its way into international agreements, financial concessions and a continuous flow of railways locomotives into the borders. This solution has been seen as possible two long colonization attempts."https://www.theguardian.com/world/2014/sep/10/go-topics-for-the-existing-response-event-at-the-united-from-the-loyal-response-president-on/
> 
> 2. The First International Commission for the Investigation of National Socialist Crimes
> 
> In October 1945, before the Great Purge, Germany invaded Poland from the East (not to be confused with the Red Army), the Central Powers, and the Baltic states. (Silesia was not a neutral

## How to use

The easiest way to use Kirk Tung programatically is with a pipeline in HF
transformers.

```
$ pip install transformers torch
```

```python
import torch
from transformers import pipeline

device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
pipe = pipeline(
    "text-generation", model="rtc2022/kirk-tung", device=device
)
```

## Support the creator

Support the creation of more models like this one by directly supporting the main creator:

[Support Lopes on Ko-Fi](https://ko-fi.com/hvilaverde)