Prashant2024/punct-openvino-int8

Punctuation Restoration (OpenVINO IR)

A token-classification model that inserts punctuation into unpunctuated transcripts, converted from ai4bharat/Cadence-Fast (a fine-tuned Gemma3 model, Gemma3ForTokenClassification / cadence_punctuation architecture) to OpenVINO IR with post-training INT8 quantization (NNCF) for CPU inference.

Before publishing - this one matters: this model is a derivative of Google's Gemma, so redistributing it is governed by the Gemma Terms of Use and Prohibited Use Policy, which flow down through ai4bharat/Cadence-Fast. That generally means: keep the license: gemma tag, include a "Built with Gemma" notice on this model card, link to the Gemma Terms of Use, and confirm ai4bharat/Cadence-Fast's own license/model card doesn't impose anything further before uploading. Don't drop or reword this section without checking those terms first.

Built with Gemma.

Files

File	Description
punct.xml	OpenVINO IR model topology
punct.bin	OpenVINO IR model weights (FP16)
config.json	Original HF config - id2label/label2id punctuation label map is read from here at inference time
tokenizer.json	Fast tokenizer data (Gemma SentencePiece-based)
tokenizer_config.json	Tokenizer configuration
special_tokens_map.json	Special token definitions

Model details

• Architecture: Gemma3 encoder + token-classification head (18 layers, hidden size 640)
• Task: for each input token, predict which punctuation mark (if any) follows it
• Labels: 31 classes, O (no punctuation) plus marks covering Latin, Devanagari (danda ।, double danda ॥), and Arabic/Urdu punctuation - see config.json's id2label for the full set
• Inputs: input_ids, attention_mask (standard Gemma tokenizer output)
• Outputs: logits - [batch, sequence_length, num_labels]

Usage

import numpy as np
import openvino as ov
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(".")  # this folder

core = ov.Core()
model = core.read_model("punct.xml")
compiled = core.compile_model(model, "CPU")

inputs = tokenizer(text, return_tensors="np")
outputs = compiled({"input_ids": inputs["input_ids"], "attention_mask": inputs["attention_mask"]})
predictions = outputs["logits"].argmax(axis=-1)

For each non-special input token, look up its predicted label id in config.json's id2label; if it isn't "O", insert that punctuation mark immediately after the token before re-joining the tokens back into text.

Intended use

Restoring punctuation in unpunctuated speech-to-text output (Hindi and English so far). Not evaluated on written/typed text.