README.md

metadata

license: cc-by-4.0
language:
  - pa
tags:
  - gurmukhi
  - panjabi
  - punjabi
  - aipanjab

model-loader: AI Panjab models in one line

Load any AI Panjab model and run it on your own machine, in two lines of Python.

pip install git+https://huggingface.co/aipanjab/model-loader

from aipanjab import load

load("digit-recognition").predict("digit.png")                       # {'੨': 0.99, '੩': 0.01, '੭': 0.0}
load("alphabet-recognition").predict("letter.png")                   # {'ਕ': 0.98, ...}
load("spell-checker-base").check("ਮੈਂ ਸਕੂਲ ਜਾਂਦਾ ਹਾ").corrected_text   # 'ਮੈਂ ਸਕੂਲ ਜਾਂਦਾ ਹਾਂ'

load() downloads only the model you ask for, the first time (it is cached after that, and works offline), and runs it on your CPU. This package holds no model: every model and its code live in their own repo, so load() always gets the current one. Several models work side by side in one program.

Models

load(...)	Use	Returns
"digit-recognition"	handwritten Gurmukhi digits ੦–੯	.predict(image, top_k=3) → {label: probability}
"alphabet-recognition"	handwritten Gurmukhi letters and signs (57)	.predict(image, top_k=3) → {label: probability}
"spell-checker-base"	Panjabi spelling correction	.check(text) → .corrected_text, .corrections, .to_dict()

Every model also has .train(...) and .save(path) (below).

image is a file path, an open file or a PIL image. Each model's card on Hugging Face has the details.

• A version of a model: load("digit-recognition", revision="v1") (a tag, branch or commit).
• A private model: log in first with hf auth login.
• A folder: load("path/to/folder") loads a model you saved, or a downloaded repo.
• On an NVIDIA GPU: install onnxruntime-gpu (instead of onnxruntime) and models use the GPU by themselves; load(..., device="cpu") or device="cuda" chooses. Without a GPU everything runs on the CPU.

Train

Every model can be trained further on your own data, on your machine. Install the training extra once:

pip install "aipanjab-loader[train] @ git+https://huggingface.co/aipanjab/model-loader"

from aipanjab import load

# handwriting: a folder with one folder per label (my-digits/੨/1.png, ...), a Hugging Face dataset id,
# or (image, label) pairs; without data, the model's own dataset
digits = load("digit-recognition").train(data="my-digits", epochs=10)
digits.predict("digit.png")

# spelling: a word list (.txt, one correctly spelled word per line, most common first) or a list of words
checker = load("spell-checker-base").train(data="my_words.txt", epochs=2)
checker.check("ਮੈਂ ਸਕੂਲ ਜਾਂਦਾ ਹਾ").corrected_text

checker.save("my-spell-checker")          # later: load("my-spell-checker")

train() continues from the model you loaded and returns the new one, ready to use. Options are keywords (epochs, batch_size, lr, ...; the spell checker also steps, gold), workdir="runs" keeps the new model there (default: a temporary folder). A saved folder is laid out like the model's repo, so it can be pushed to Hugging Face as it is. Training uses a GPU when there is one (NVIDIA, or Apple silicon); device="cpu", "cuda" or "mps" chooses. The spell checker takes a word list of any size: with 1,000 words or more, 1% are held out to test the model as it trains; with fewer, all of them train.

The first load() in a program also fetches this repo's small config.json, which is how Hugging Face counts the loader's downloads; offline it is skipped.

Pin a version

To keep a program on one version of this package: pip install git+https://huggingface.co/aipanjab/model-loader@v0.1.0.

By AI Panjab. Licence: CC BY 4.0.