abhinavsarkar/Google-T5-Large-Grammatical_Error_Correction-Finetuned-C4-200M-1M

Model Card for Google-T5-Large-Grammatical-Error-Correction-Finetuned-C4-200M-1M

This model is fine-tuned for grammatical error correction (GEC), focusing primarily on correcting determiner-related errors. It helps generate grammatically correct text from input sentences with errors, making it valuable for writing enhancement tools and grammar assistance systems.

Model Details

Model Description

This model is a fine-tuned version of [google-t5/t5-large] tailored for grammatical correction tasks, especially involving determiners.

• Developed by: Abhinav Sarkar
• Shared by: abhinavsarkar
• Model type: Sequence-to-sequence Transformer
• Languages: English
• Finetuned from model: google-t5/t5-large

Uses

Direct Use

This model is well-suited for:

• Grammar correction tools
• Writing assistants
• Email and content editors
• Educational tools for ESL learners

Downstream Use

Potential integrations include:

• Grammar and spell-checking systems
• Language learning platforms
• Proofreading tools for students and professionals

How to Get Started with the Model

Install the dependencies:

pip install -U sentencepiece transformers torch

Load and use the model:

import torch
from transformers import T5Tokenizer, T5ForConditionalGeneration

model_name = 'abhinavsarkar/Google-T5-Large-Grammatical_Error_Correction-Finetuned-C4-200M-1M'
torch_device = 'cuda' if torch.cuda.is_available() else 'cpu'
tokenizer = T5Tokenizer.from_pretrained(model_name)
model = T5ForConditionalGeneration.from_pretrained(model_name).to(torch_device)

Example inference:

def correct_grammar(input_text, num_return_sequences=2):
    batch = tokenizer([input_text], truncation=True, padding='max_length', max_length=64, return_tensors="pt").to(torch_device)
    translated = model.generate(**batch, max_length=64, num_beams=4, num_return_sequences=num_return_sequences, temperature=1.5)
    tgt_text = tokenizer.batch_decode(translated, skip_special_tokens=True)
    return tgt_text

text = 'He are moving here.'
print(correct_grammar(text))

Training Details

Training Data

The model was fine-tuned on abhinavsarkar/C4-200M-1M-GEC-Determiner, a 1M-sample subset from the C4-200M dataset (original dataset link) focused on grammatical error correction, specifically determiners.

Training Procedure

The model was trained using Hugging Face Transformers on a single NVIDIA A6000 GPU for 3.5 hours using bf16 precision via Runpod.

Training Hyperparameters

• Epochs: 1
• Batch size: 128
• Learning rate: 2e-5
• Precision: bf16
• Optimizer: AdamW (betas=(0.9, 0.999), epsilon=1e-08)
• LR Scheduler: Linear
• Seed: 42

Evaluation

The model was evaluated on a 5% random sample from the training dataset. Evaluation metrics include:

• ROUGE1: 74.76
• ROUGE2: 65.98
• ROUGEL: 74.12
• ROUGELsum: 74.14
• BLEU / GLEU / Accuracy: (to be updated)

Technical Specifications

Compute Infrastructure

• Hardware: Single A6000 GPU
• Platform: Runpod
• Framework: PyTorch
• Libraries: Hugging Face Transformers

More Information

For further details or inquiries, feel free to reach out via LinkedIn or email at abhinavsarkar53@gmail.com.

Model Card Authors

• Abhinav Sarkar

Model Card Contact

• Email: abhinavsarkar53@gmail.com