notebook.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "imOmo10GmFjs"
      },
      "source": [
        "# Tutorial: Using `enesyila/ota-roberta-base` for Mask Filling in Ottoman Turkish\n",
        "\n",
        "This notebook demonstrates how to load and use the `[ota-roberta-base](https://huggingface.co/enesyila/ota-roberta-base)` model for fill-mask tasks in Ottoman Turkish (1500–1928). We cover model details, installation, basic usage, and examples.\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "## 1. Model Details\n",
        "\n",
        "| Property       | Details                                      |\n",
        "| -------------- | -------------------------------------------- |\n",
        "| **Model name** | `enesyila/ota-roberta-base`                  |\n",
        "| **Architecture** | RoBERTa-base (XLM-RoBERTa fine-tuned)      |\n",
        "| **Task**       | Masked language modeling (fill-mask)         |\n",
        "| **Language**   | Ottoman Turkish (1500–1928)                  |\n",
        "| **License**    | CC-BY-NC-4.0                                 |\n",
        "\n",
        "> _Fine-tuned on 16 million tokens from 48 Ottoman literary works._\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AjZLiG9bmFjx"
      },
      "source": [
        "## 2. Installation\n",
        "\n",
        "# Ensure you have `transformers` and `torch` installed. If not, install with:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xLxSeNtFmFjx"
      },
      "outputs": [],
      "source": [
        "!pip install transformers torch safetensors"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UoXwTOiKmFj0"
      },
      "source": [
        "# 3. Load Model and Pipeline"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "B4RV5yrfmFj0"
      },
      "outputs": [],
      "source": [
        "from transformers import AutoTokenizer, AutoModelForMaskedLM, pipeline\n",
        "\n",
        "# 1. Specify model checkpoint\n",
        "model_name = \"enesyila/ota-roberta-base\"\n",
        "\n",
        "# 2. Load tokenizer and model\n",
        "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
        "model = AutoModelForMaskedLM.from_pretrained(model_name)\n",
        "\n",
        "# 3. Create fill-mask pipeline\n",
        "unmasker = pipeline(\"fill-mask\", model=model, tokenizer=tokenizer)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FwWo_-ECmFj1"
      },
      "source": [
        "## 4. Basic Example"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LoY1dksLmFj2"
      },
      "outputs": [],
      "source": [
        "sequence = \"Ne yanar kimse bana âteş-i <mask> özge\"\n",
        "results = unmasker(sequence)\n",
        "\n",
        "# Display top 5 predictions\n",
        "for idx, r in enumerate(results, 1):\n",
        "    print(f\"{idx}. {r['sequence']}\\n   Score: {r['score']:.4f}\\n\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hPaigxEimFj3"
      },
      "source": [
        "## 5. More Examples\n",
        "\n",
        "### 5.1 Single Sentence Completion"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rMgR56g0mFj4"
      },
      "outputs": [],
      "source": [
        "sequence = \"Ol perî <mask> melek kim beñzemez insân aña\"\n",
        "for r in unmasker(sequence):\n",
        "    print(r['sequence'], r['score'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2rz7cOTtmFj4"
      },
      "source": [
        "### 5.2 Batch Processing"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uUS8ZmJMmFj5"
      },
      "outputs": [],
      "source": [
        "sentences = [\n",
        "    \"hâl kâfir, zülf kâfir, <mask> kâfir el amân\",\n",
        "    \"çeşmini gördüm unuttum <mask> de dermanı da\",\n",
        "]\n",
        "for seq in sentences:\n",
        "    print(f\"Input: {seq}\")\n",
        "    for r in unmasker(seq):\n",
        "        print(f\"  - {r['sequence']} ({r['score']:.3f})\")\n",
        "    print()"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 5.3 Finding the Best Option\n",
        "\n",
        "The code below takes a sentence with an unknown word or words first. Then, according to the candidate words the user gives, it calculates the probability of each candidate in the context.\n",
        "At the end of the code block, you can give your own examples with candidates and the model will return the result. Note that the model is still under development, so it might give wrong results."
      ],
      "metadata": {
        "id": "ZwsjAo0Omnid"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from transformers import AutoTokenizer, AutoModelForMaskedLM\n",
        "import torch\n",
        "\n",
        "# Load your fine-tuned model and tokenizer from Google Drive or local path\n",
        "model_path = \"enesyila/ota-roberta-base\"\n",
        "\n",
        "tokenizer = AutoTokenizer.from_pretrained(model_path)\n",
        "model = AutoModelForMaskedLM.from_pretrained(model_path)\n",
        "model.eval()\n",
        "\n",
        "def find_sublist(lst, sublst):\n",
        "    \"\"\"\n",
        "    Finds the start index of sublist (sublst) inside the main list (lst).\n",
        "    Returns None if sublist is not found.\n",
        "    \"\"\"\n",
        "    for i in range(len(lst) - len(sublst) + 1):\n",
        "        if lst[i : i + len(sublst)] == sublst:\n",
        "            return i\n",
        "    return None\n",
        "\n",
        "def score_candidates(template: str, candidates: list[str]) -> tuple[dict[str, float], str]:\n",
        "    \"\"\"\n",
        "    Given a sentence template and a list of candidate words:\n",
        "    - Inserts each candidate into the sentence\n",
        "    - Gets model scores (logits) for the inserted word\n",
        "    - Returns a dictionary of scores and the best word (highest score)\n",
        "    \"\"\"\n",
        "    scores = {}\n",
        "    for word in candidates:\n",
        "        sentence = template.format(word)\n",
        "        inputs = tokenizer(sentence, return_tensors=\"pt\")\n",
        "        with torch.no_grad():\n",
        "            outputs = model(**inputs)\n",
        "        logits = outputs.logits[0]  # shape: [sequence_length, vocab_size]\n",
        "\n",
        "        # Tokenize the candidate word without special tokens\n",
        "        word_tokens = tokenizer(word, add_special_tokens=False).input_ids\n",
        "        input_ids_list = inputs.input_ids[0].tolist()\n",
        "\n",
        "        # Find where the word appears in the tokenized input\n",
        "        start_idx = find_sublist(input_ids_list, word_tokens)\n",
        "        if start_idx is None:\n",
        "            raise ValueError(f\"Token sequence for '{word}' not found in: {sentence}\")\n",
        "\n",
        "        # Sum logit scores for each token in the word\n",
        "        score = sum(\n",
        "            logits[start_idx + i, word_tokens[i]].item()\n",
        "            for i in range(len(word_tokens))\n",
        "        )\n",
        "        scores[word] = score\n",
        "\n",
        "    best_word = max(scores, key=scores.get)\n",
        "    return scores, best_word\n",
        "\n",
        "def evaluate_templates(examples: list[tuple[str, list[str]]]):\n",
        "    \"\"\"\n",
        "    Input: List of (template, candidates) tuples\n",
        "    Prints the score of each candidate and the best one for each example.\n",
        "    \"\"\"\n",
        "    for idx, (template, candidates) in enumerate(examples, 1):\n",
        "        scores, best = score_candidates(template, candidates)\n",
        "        print(f\"\\nExample #{idx}\")\n",
        "        print(f\"Template : {template}\")\n",
        "        print(\"Candidates:\", candidates)\n",
        "        print(\"Scores    :\")\n",
        "        for w, sc in scores.items():\n",
        "            print(f\"  {w}: {sc:.4f}\")\n",
        "        print(f\"→ Best candidate: '{best}'\")\n",
        "\n",
        "\n",
        "examples = [\n",
        "    (\"Gel ey ḥarîf şimdi {} ḳabûl ḳıl\", [\"nasîḥat\", \"vardı\"]),\n",
        "    (\"ʿAḳlı {} gitdi\", [\"başından\", \"baş\", \"başına\"]),\n",
        "    (\"Gerçi der-nâfe-i {} âhû-yı Ḫuten mî-pervered\", [\"ḫod\", \"çiy\", \"muġân\"]),\n",
        "    (\"Ceydâ sözin işidicek, taḥammül idemedi, güldi ve yüzin açdı ve {}\", [\"eyitdi\", \"gitti\", \"Feraḥ\"]),\n",
        "    (\"Eyledi {} ʿaṭâsı âb-ı luṭfı terbiyet\", [\"mihr-i\", \"gül\", \"ḳıldı\"]),\n",
        "    (\"Nes̱r: Ḥaḳḳâ ve {} âsmân-ı bî-hemtâ bir saḳf-ı muʿallâdur\", [\"s̱ümme ḥaḳḳâ\", \"peyġambere\", \"güle\"])\n",
        "]\n",
        "evaluate_templates(examples)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fMLeaxJ0mY-g",
        "outputId": "df166c60-196a-4e26-be78-548ee77c30aa"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Example #1\n",
            "Template : Gel ey ḥarîf şimdi {} ḳabûl ḳıl\n",
            "Candidates: ['nasîḥat', 'vardı']\n",
            "Scores    :\n",
            "  nasîḥat: 255.3716\n",
            "  vardı: 47.1503\n",
            "→ Best candidate: 'nasîḥat'\n",
            "\n",
            "Example #2\n",
            "Template : ʿAḳlı {} gitdi\n",
            "Candidates: ['başından', 'baş', 'başına']\n",
            "Scores    :\n",
            "  başından: 138.4005\n",
            "  baş: 48.1729\n",
            "  başına: 61.3716\n",
            "→ Best candidate: 'başından'\n",
            "\n",
            "Example #3\n",
            "Template : Gerçi der-nâfe-i {} âhû-yı Ḫuten mî-pervered\n",
            "Candidates: ['ḫod', 'çiy', 'muġân']\n",
            "Scores    :\n",
            "  ḫod: 192.2042\n",
            "  çiy: 120.9160\n",
            "  muġân: 219.7602\n",
            "→ Best candidate: 'muġân'\n",
            "\n",
            "Example #4\n",
            "Template : Ceydâ sözin işidicek, taḥammül idemedi, güldi ve yüzin açdı ve {}\n",
            "Candidates: ['eyitdi', 'gitti', 'Feraḥ']\n",
            "Scores    :\n",
            "  eyitdi: 203.6618\n",
            "  gitti: 62.5228\n",
            "  Feraḥ: 104.9652\n",
            "→ Best candidate: 'eyitdi'\n",
            "\n",
            "Example #5\n",
            "Template : Eyledi {} ʿaṭâsı âb-ı luṭfı terbiyet\n",
            "Candidates: ['mihr-i', 'gül', 'ḳıldı']\n",
            "Scores    :\n",
            "  mihr-i: 297.6816\n",
            "  gül: 55.3786\n",
            "  ḳıldı: 216.4692\n",
            "→ Best candidate: 'mihr-i'\n",
            "\n",
            "Example #6\n",
            "Template : Nes̱r: Ḥaḳḳâ ve {} âsmân-ı bî-hemtâ bir saḳf-ı muʿallâdur\n",
            "Candidates: ['s̱ümme ḥaḳḳâ', 'peyġambere', 'güle']\n",
            "Scores    :\n",
            "  s̱ümme ḥaḳḳâ: 677.6495\n",
            "  peyġambere: 242.5137\n",
            "  güle: 126.0983\n",
            "→ Best candidate: 's̱ümme ḥaḳḳâ'\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QfyvvInKmFj5"
      },
      "source": [
        "## 6. Tips and Limitations\n",
        "\n",
        "Biases: May reproduce historical or offensive terms with censored characters.\n",
        "\n",
        "Domain: Optimized for literary/historical Ottoman Turkish.\n",
        "\n",
        "Vocabulary: Uncommon words may be split into subwords.\n",
        "\n",
        "Limitations: Due to inefficient amount of data, it might not produce the correct output always."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BYwm_FgumFj6"
      },
      "source": [
        "## 7. Contact\n",
        "\n",
        "Feel free to contact me if you have any issue, question, or suggestion: enes.yilandiloglu@helsinki.fi"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.0"
    },
    "orig_nbformat": 4,
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}