Upload folder using huggingface_hub

README.md

@@ -0,0 +1,164 @@
+---
+license: mit
+language:
+- am
+- ar
+- bg
+- bn
+- cs
+- da
+- de
+- el
+- en
+- es
+- fa
+- fi
+- fr
+- gu
+- ha
+- hi
+- hu
+- id
+- it
+- ja
+- jv
+- kn
+- ko
+- lt
+- mr
+- nl
+- 'no'
+- yo
+- zh
+- pl
+- pt
+- ro
+- ru
+- sk
+- sv
+- sw
+- ta
+- te
+- th
+- tr
+- uk
+- ur
+- vi
+- tl
+---
+
+# Shitsu
+
+<p align="center">
+    <img src="https://cdn-uploads.huggingface.co/production/uploads/64b63f8ad57e02621dc93c8b/Lkw-M8a-AAfJiC81DobNl.jpeg" alt="A logo of a Shit Zhu reading a book" width="400"/>
+</p>
+
+A text scorer which scores text based on the amount of useful, textbook-like information in it.
+It outputs a score generally between 0 and 1 but can exceed both of these bounds as it is a regressor.
+
+Our model is based on fasttext embeddings, meaning that it can be used on large amounts of data with limited compute quickly.
+
+This scorer can be used to filter useful information from large text corpora in many languages.
+
+This model can also be found on [Github](https://github.com/lightblue-tech/shitsu).
+
+# How to use
+
+### With our scorer package
+
+```bash
+pip install git+https://github.com/lightblue-tech/shitsu.git
+```
+
+```python
+from shitsu import ShitsuScorer
+
+text_list = [
+    "Photosynthesis is a system of biological processes by which photosynthetic organisms, such as most plants, algae, and cyanobacteria, convert light energy, typically from sunlight, into the chemical energy necessary to fuel their metabolism.",
+    "Congratulations! You have all been selected to receive a free gift card worth $1000. Click on this link [Link] to claim your reward now. Limited time offer, so act fast! Don't miss out on this amazing opportunity."]
+
+# Choose a language from one of: ['am', 'ar', 'bg', 'bn', 'cs', 'da', 'de', 'el', 'en', 'es', 'fa', 'fi', 'fr', 'gu', 'ha', 'hi', 'hu', 'id', 'it', 'ja', 'jv', 'kn', 'ko', 'lt', 'mr', 'nl', 'no', 'pl', 'pt', 'ro', 'ru', 'sk', 'sv', 'sw', 'ta', 'te', 'th', 'tl', 'tr', 'uk', 'ur', 'vi', 'yo', 'zh']
+language_code = "en"
+scorer = ShitsuScorer(language_code)
+scores = scorer.score(text_list)
+scores
+# array([ 0.9897383 , -0.08109612], dtype=float32)
+```
+
+### Without our scorer package (i.e. without pip install)
+
+<details>
+  <summary>Show full code</summary>
+    
+  ```python
+
+from safetensors.torch import load_model
+import fasttext
+from huggingface_hub import hf_hub_download
+from tqdm.auto import tqdm
+import torch
+import numpy as np
+import torch
+import torch.nn as nn
+
+class FasttextEmbedRegressor(nn.Module):
+    def __init__(self, input_size=300):
+        super(FasttextEmbedRegressor, self).__init__()
+        layer_1_size = 64
+        layer_2_size = 32
+        self.fc1 = nn.Linear(input_size, layer_1_size)
+        self.fc2 = nn.Linear(layer_1_size, layer_2_size)
+        self.fc3 = nn.Linear(layer_2_size, 1)
+
+    def forward(self, x):
+        x = torch.relu(self.fc1(x))
+        x = torch.relu(self.fc2(x))
+        x = self.fc3(x)
+        return x
+
+class ShitsuScorer:
+    def __init__(self, lang_code):
+        fasttext_model_path = hf_hub_download(repo_id=f"facebook/fasttext-{lang_code}-vectors", filename="model.bin")
+        self.fasttext_model = fasttext.load_model(fasttext_model_path)
+        self.regressor_model = FasttextEmbedRegressor().eval()
+        regressor_model_path = hf_hub_download(repo_id=f"lightblue/shitsu_text_scorer", filename=f"{lang_code}.safetensors")
+        load_model(self.regressor_model, regressor_model_path)
+
+    def score(self, text_list):
+        embeddings = np.stack([self.fasttext_model.get_sentence_vector(x.replace("\n", " ")) for x in tqdm(text_list)])
+        return self.regressor_model(torch.Tensor(embeddings)).detach().numpy().flatten()
+
+text_list = [
+    "Photosynthesis is a system of biological processes by which photosynthetic organisms, such as most plants, algae, and cyanobacteria, convert light energy, typically from sunlight, into the chemical energy necessary to fuel their metabolism.",
+    "Congratulations! You have all been selected to receive a free gift card worth $1000. Click on this link [Link] to claim your reward now. Limited time offer, so act fast! Don't miss out on this amazing opportunity."]
+
+scorer = ShitsuScorer("en")
+scores = scorer.score(text_list)
+scores
+# array([ 0.9897383 , -0.08109612], dtype=float32)
+```
+
+</details>
+<br/>
+
+
+
+# How we made the training data
+
+We provided a sample of tens of thousands [MADLAD-400](https://huggingface.co/datasets/allenai/MADLAD-400) in various languages to a popular state-of-the-art LLM with the following system prompt:
+
+```python
+system_message = """You are a text filtering AI model.
+Your input is a piece of text.
+Your output is a score of how likely the text is to appear in a useful {language} textbook, encyclopedia, or any other important document.
+
+Output your score on a scale of 0-100, with 0 meaning that the text contains no useful {language} information and 100 meaning that the text is very useful and is exceedingly likely to appear in a {language} textbook, encyclopedia, or any other important document. If the text is not mostly fluent, natural {language}, output 0.
+
+Your output should be only an integer from 0-100."""
+```
+
+This resulted in the dataset found at [lightblue/text_ratings](https://huggingface.co/datasets/lightblue/text_ratings).
+
+We then trained a small neural network on top of fasttext's embeddings to predict these scores.
+
+We chose the 44 languages in this dataset by making a union set of the 30 most popular languages on earth as according to [Ethnologue 2024](https://www.ethnologue.com/insights/ethnologue200/) and the 30 most popular languages within MADLAD-400.

am.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:489522571130c58c649d6d6dd375ecba28a1ee42dd3357d6f77724424b04defe
+size 85940

ar.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ac733353ec859d1810b841f1d7915dbaa843de7b3dd8058033146b3d2410ef3d
+size 85940

bg.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4452b85a7f0525833deaa4114e0dbc1790683a6980e8c10be649fcc65c5c9622
+size 85940

bn.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:691f5e6ca3bf96398281089057618bfb9950a012bd6364c5a317ee872b7db2af
+size 85940

cs.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:948c1c7fd49da5c3e20eb9a966cd6d22b507c5985934f264acbc1824ee7e70f6
+size 85940

da.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6d94153a85e7e02f8818f908cff6f38d648e48a457d2e889ea63b70d6b06edb3
+size 85940

de.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:273e3dd597651aba2508a0c9c67e5cfd27cfa9e3f4479c7b3c8a61cb08d242e8
+size 85940

el.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5764ea4dccac430bed7ab127a22093b6c1ec11b3c79bea247ac98e0d4fca0f1e
+size 85940

en.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:de4a3e3d282988d2ec3ef041838f7493d617b377bff89506c932ef91a4167aa5
+size 85940

es.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0ed420ffb402ce792fd082a6e463bf19af7d117f1d654508e0f870dae960520b
+size 85940

fa.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:97acc1c2337f24ee768496c5c1356c25164d5885fda2737ddf7c67751a6224ac
+size 85940

fi.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:eebd991819a5a0e4b6db4dcc3795a1daa83f8045b568550fd7a31b4ffdd04922
+size 85940

fr.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0289d27fcd65a80b35268e20d99ca6ab531c05d2f1e814608aa3e2bca7a8cc84
+size 85940

gu.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2256815704e4624c608f8718bf685887ca1e4644110c20f7f35c1857c72f9e94
+size 85940

ha.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c615366f3399709ebec863060a4120cf3d330d0eb52c4547e1bb96865c56fb73
+size 85940

hi.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:76372b8c14e3160a34ec32346c9858ecd8ba0776e617bd4ca47c5553d1dd2dee
+size 85940

hu.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1d342ed20e03ad1d69db0e65e192411282299caf4930d2d46efa55531194efb8
+size 85940

id.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a195de27c63f51661a7963e289ecb69c557ba4f9d605b93001347e2aaf3da8c8
+size 85940

it.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:32e562589fa29ee6b82bd3264d869479c8f01e19579ebb3acb01703c41a94873
+size 85940

ja.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:645535860bb96b5533fc619e1705109db882ae4942bc4e6fc9341c1a61bd2f79
+size 85940

jv.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:77f4bcdba518f4fcad654600a6e8e80d9da6cc014693509e51fa94efda138478
+size 85940

kn.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:40733b980553f437b44cd8abd465549dcb398955dfae0afe0540492b35e28cb9
+size 85940

ko.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c6c875bfc3b17aae81007288d89731f44820e355e6920e7b5ba6dfc7ec2f7a21
+size 85940

lt.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:46aa6488501fdf098a78e71716de32f88e77b5e05d879e8a05c3796d50625d57
+size 85940

mr.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:73949c67f05cc10b4cb2a2a3d43f587fec28228044e2b010e351acefc49f4abe
+size 85940

nl.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:253014d78d3513cc1fd8fa98a4eade5a3c1ab10f6a734fa6e04d5eb71224d056
+size 85940

no.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a3efc89d7ae4161faacbca7c772bc27a485d4738c597aa35d805106f9af04859
+size 85940

pl.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7ce49f5a81160f945139d10a4891cfb4584f24e8235666d5fc1ce026a98671ad
+size 85940

pt.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:33be636d1f94cc7611b327179aca34b615a6962b9967bac21c619fe0fa022a2b
+size 85940

ro.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4bf60d087b14f1f0c7c9add7d63b4f5feb01fa854a20c8b6be892de89b2e3156
+size 85940

ru.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:39742ac8cd7c34f500b68042eae26e5399f8b3b76a3b94d811ca9d42ec55a547
+size 85940

sk.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1c3e2b121f7ad1786f0c4ebed86f6fbcf711ff60c33112dba88819f1f14a90cd
+size 85940

sv.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a2e0f8c075d0ea131c70f3a09edcdc7f40a462b31925744eff72c903ac3b86b8
+size 85940

sw.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0b72bd1110ea970e0e8b086b99bb53eb308178524545c6d1aaca59e7e9d0916b
+size 85940

ta.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2218afceb03907685fb9b5faeff6a6bff1316a47f004deeeb85640ba791876f1
+size 85940

te.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:885877a1795b41b3548c246a65fc779f1e6f3bc25059f634d72e1eadcf99b356
+size 85940

th.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aa81c5582a38c0d5455a1396e58c8949c89554bc97c926d8c39e88fc2dc9fce4
+size 85940

tl.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8553139fc548428902a8992f0dc5713ab091c212f263da7bb33b8007810a76c5
+size 85940

tr.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d52fb2a6d7aa68c80bda9d2d07acd12da684a97e3cbbcbe985cb3e856c0c0040
+size 85940

training.py

@@ -0,0 +1,152 @@
+from datasets import load_dataset
+import pandas as pd
+import fasttext
+import numpy as np
+from sklearn.model_selection import train_test_split
+from tqdm.auto import tqdm, trange
+import torch
+import torch.nn as nn
+import torch.optim as optim
+from sklearn.metrics import mean_squared_error
+import requests
+import gzip
+import shutil
+import os
+from safetensors.torch import save_model
+import matplotlib.pyplot as plt
+
+class FasttextEmbedRegressor(nn.Module):
+    def __init__(self, input_size=300):
+        super(FasttextEmbedRegressor, self).__init__()
+        layer_1_size = 64
+        layer_2_size = 32
+        self.fc1 = nn.Linear(input_size, layer_1_size)
+        self.fc2 = nn.Linear(layer_1_size, layer_2_size)
+        self.fc3 = nn.Linear(layer_2_size, 1)
+
+    def forward(self, x):
+        x = torch.relu(self.fc1(x))
+        x = torch.relu(self.fc2(x))
+        x = self.fc3(x)
+        return x
+
+def train_regressor(X_train, X_test, y_train, y_test, train_epochs):
+
+    # Initialize the model, loss function, and optimizer
+    input_size = X_train.shape[1]
+    model = FasttextEmbedRegressor(input_size)
+    criterion = nn.MSELoss()
+    optimizer = optim.Adam(model.parameters(), lr=0.001)
+    batch_size = 32
+
+    training_metrics = []
+
+    for epoch in trange(train_epochs):
+        model.train()
+        train_losses = []
+        for step_num, i in enumerate(trange(0, X_train.shape[0], batch_size)):
+            vectors = torch.Tensor(X_train[i:i+batch_size])
+            targets = torch.Tensor(y_train[i:i+batch_size])
+            optimizer.zero_grad()
+            outputs = model(vectors).squeeze()
+            loss = criterion(outputs, targets)
+            loss.backward()
+            optimizer.step()
+            train_losses.append(float(loss))
+            if step_num % 10 == 0:
+                model.eval()
+                test_preds = model(torch.Tensor(X_test)).detach().numpy()
+                test_mse = mean_squared_error(y_test, test_preds)
+                training_metrics.append({
+                    "epoch": epoch,
+                    "step_num": step_num,
+                    "i": i,
+                    "test_mse": test_mse,
+                    "train_loss": sum(train_losses) / len(train_losses),
+                })
+                train_losses = []
+                model.train()
+
+    return pd.DataFrame(training_metrics), model
+
+def download_file(url, filename):
+    """
+    Helper method handling downloading large files from `url` to `filename`. Returns a pointer to `filename`.
+    """
+    chunkSize = 1024
+    r = requests.get(url, stream=True)
+    with open(filename, 'wb') as f:
+        pbar = tqdm( unit="B", total=int( r.headers['Content-Length'] ) )
+        for chunk in r.iter_content(chunk_size=chunkSize): 
+            if chunk: # filter out keep-alive new chunks
+                pbar.update (len(chunk))
+                f.write(chunk)
+    return filename
+
+get_filename = lambda x: f"cc.{x}.300.bin"
+
+def download_fasttext_vectors(lang_code):
+    filename = get_filename(lang_code)
+
+    if os.path.isfile(filename):
+        return None
+
+    print(f"Downloading {lang_code} vectors")
+    download_file(f"https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/{filename}.gz", f"{filename}.gz")
+
+    print(f"Unzipping {lang_code} vectors")
+    with gzip.open(f"{filename}.gz", 'rb') as f_in:
+        with open(filename, 'wb') as f_out:
+            shutil.copyfileobj(f_in, f_out)
+
+    print(f"Removing zipped {lang_code} vectors")
+    os.remove(f"{filename}.gz")
+    
+    return True
+
+def create_quality_eval_model(lang_code, train_epochs=10):
+
+    download_fasttext_vectors(lang_code)
+
+    dataset = load_dataset("lightblue/text_ratings", lang_code, split="train")
+    text_list = dataset["selected_chunk"]
+    label_float = [x / 100 for x in dataset["rating_float"]]
+
+    fasttext_model = fasttext.load_model(f"cc.{lang_code}.300.bin")
+
+    embeddings = np.stack([fasttext_model.get_sentence_vector(
+        x.replace("\n", " ")
+        ) for x in tqdm(text_list)])
+
+    X_train, X_test, y_train, y_test, text_train, text_test = train_test_split(
+            embeddings,
+            label_float,
+            text_list,
+            test_size=0.2,
+            random_state=42
+    )
+
+    metrics_df, model = train_regressor(X_train, X_test, y_train, y_test, train_epochs)
+
+    test_df = pd.DataFrame({
+        "text": text_test, 
+        "gold_score": y_test, 
+        "pred_score": model(torch.Tensor(X_test)).detach().numpy().flatten()
+    })
+
+    save_model(model, f"{lang_code}.safetensors")
+
+    os.remove(get_filename(lang_code))
+
+    return metrics_df, test_df
+
+if __name__ == '__main__':
+
+    langs = ['am', 'ar', 'bg', 'bn', 'cs', 'da', 'de', 'el', 'en', 'es', 'fa', 'fi', 'fr', 'gu', 'ha', 'hi', 'hu', 'id', 'it', 'ja', 'jv', 'kn', 'ko', 'lt', 'mr', 'nl', 'no', 'yo', 'zh']
+    
+    for l in langs:
+        print(l)
+        metrics_df, test_df = create_quality_eval_model(l, train_epochs=5)
+        print(l)
+        metrics_df[["test_mse", "train_loss"]].rolling(50).mean().plot()
+        plt.show()

uk.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1f159f5d8f8c5b0119924aee84550f333380ec184667063be0b4476ad236b557
+size 85940

ur.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1e8df6db2d462135560360e394d831b0718c37989d149e2831ad42554389da0f
+size 85940

vi.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f6ce79064155b5468dab67fc1567cbfeae48e60812617c7ff19132bd9acedf2f
+size 85940

yo.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8db79885d6f7ce795956c0dae5c0c23176799432734423246338ed6de2396460
+size 85940

zh.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cef569e9b680e7a45421f4fbc82070a35e9e9317a4d5cbb399ba27b9fc6587ba
+size 85940