Instructions to use nirajr/coedit-small-gguf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use nirajr/coedit-small-gguf with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf nirajr/coedit-small-gguf:Q8_0 # Run inference directly in the terminal: llama cli -hf nirajr/coedit-small-gguf:Q8_0
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf nirajr/coedit-small-gguf:Q8_0 # Run inference directly in the terminal: llama cli -hf nirajr/coedit-small-gguf:Q8_0
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf nirajr/coedit-small-gguf:Q8_0 # Run inference directly in the terminal: ./build/bin/llama cli -hf nirajr/coedit-small-gguf:Q8_0
Use Docker
docker model run hf.co/nirajr/coedit-small-gguf:Q8_0
- Llama new
- LM Studio
- Jan
- vLLM
How to use nirajr/coedit-small-gguf with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "nirajr/coedit-small-gguf" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "nirajr/coedit-small-gguf", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/nirajr/coedit-small-gguf:Q8_0
- Ollama
How to use nirajr/coedit-small-gguf with Ollama:
ollama run hf.co/nirajr/coedit-small-gguf:Q8_0
- Unsloth Desktop
- Docker Model Runner
How to use nirajr/coedit-small-gguf with Docker Model Runner:
docker model run hf.co/nirajr/coedit-small-gguf:Q8_0
- Lemonade
How to use nirajr/coedit-small-gguf with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull nirajr/coedit-small-gguf:Q8_0
Run and chat with the model
lemonade run user.coedit-small-gguf-Q8_0
List all available models
lemonade list
- Atomic Chat
Install from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf nirajr/coedit-small-gguf:Q8_0# Run inference directly in the terminal:
llama cli -hf nirajr/coedit-small-gguf:Q8_0Build from source code
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf nirajr/coedit-small-gguf:Q8_0# Run inference directly in the terminal:
./build/bin/llama cli -hf nirajr/coedit-small-gguf:Q8_0Use Docker
docker model run hf.co/nirajr/coedit-small-gguf:Q8_0coedit-small-GGUF
Two Q8_0 GGUF files for grammar cleanup of dictated English, run with llama.cpp. Both are 77M-parameter T5 models and 83 MB on disk.
| Filename | What it is |
|---|---|
| coedit-small-dictation-v1-q8_0.gguf | jbochi/coedit-small fine-tuned on synthetic dictation pairs. Use this one for dictation. |
| coedit-small-q8_0.gguf | jbochi/coedit-small with unchanged weights. Kept for existing users. |
The dictation model is trained for speech-to-text output: text with few or no punctuation marks, run-on sentences, filler words, and small grammar slips. It adds punctuation and sentence breaks, fixes grammar, and leaves the speaker's wording alone.
Compared with the base file, it doubles the punctuation score on TED talks, from 19.7 to 41.9. On 123 real dictation transcripts it makes 3 worse, where the base file makes 8 worse.
Usage
The prompt is Fix the grammar: followed by the text. Use greedy decoding.
llama-completion -m coedit-small-dictation-v1-q8_0.gguf --temp 0 -no-cnv \
-p "Fix the grammar: okay so i was thinking we should move the meeting to friday because two people is on leave does that work for you"
# okay, so I was thinking we should move the meeting to Friday because two people are on leave. Does that work for you?
The same input through coedit-small-q8_0.gguf gives one sentence with no comma and no question mark:
# okay so I was thinking we should move the meeting to Friday because two people are on leave does that work for you.
Use llama-completion. llama-cli does not run this model. The context is 512 tokens.
Results
"Base" is coedit-small-q8_0.gguf. "Dictation v1" is coedit-small-dictation-v1-q8_0.gguf. Measured on 2026-11-11.
Punctuation on TED talks
The test is IWSLT 2011, a public benchmark of about 12,600 words of TED talks with all punctuation removed. The model has to put back commas, periods, and question marks.
The score is F1, the standard score for punctuation tests. It runs from 0 to 100 and combines two things: how many of the needed marks the model found, and how many of the marks it added were right. 100 means every mark found and none wrong.
| Input | Chunk size | Base | Dictation v1 |
|---|---|---|---|
| Human transcript | about 48 words | 19.7 | 41.9 |
| Human transcript | about 127 words | 7.8 | 43.4 |
| Speech-to-text output | about 48 words | 17.6 | 38.6 |
| Speech-to-text output | about 126 words | 7.5 | 40.0 |
Private dictation transcripts
123 real dictation transcripts from one speaker. A human wrote the corrected version of each one. None were used in training. The text is private and is not published.
| Measure | Base | Dictation v1 |
|---|---|---|
| Transcripts made worse | 8 of 123, 6.5% | 3 of 123, 2.4% |
| Share of the model's edits that were correct | 54.7% | 68.0% |
| Share of needed fixes the model made | 19.8% | 50.4% |
| GLEU | 67.0 | 79.5 |
| Average time per transcript | 81 ms | 83 ms |
GLEU is a standard score for grammar correction. It compares the model's output with the human-corrected version and counts the runs of 1 to 4 words they share. It runs from 0 to 100, and 100 means the output is identical to the human version. The raw transcripts with no cleanup score 63.9.
"Made worse" was judged with the model names hidden. Times are from an M4 Mac mini with 24 GB of memory.
Essays by English learners
JFLEG is a public benchmark of 747 sentences written by people learning English, each corrected by 4 people. It is scored with GLEU.
| Measure | Base | Dictation v1 |
|---|---|---|
| JFLEG test GLEU | 47.6 | 45.4 |
This is the one test where the dictation model scores lower, by 2.2 points out of 100. Learner essays need spelling fixes and heavy rewrites. The dictation model is trained to change as little as possible, so it leaves more of those errors in place. Use the dictation file for speech and the base file for learner writing.
Training
- Base model: jbochi/coedit-small, which is
flan-t5-smallfine-tuned on Grammarly's CoEdIT dataset. - Data: 2,615 synthetic training pairs and 301 validation pairs, written and screened by language models. No real transcript was used. 60% of inputs have errors to fix, and 40% are clean or cut-off inputs that must come back unchanged.
- Settings: learning rate 1e-4, 10 epochs, about 8 minutes on the Mac mini's GPU. Epoch 7 scored best on the validation pairs and was kept.
Example pairs:
| Input | Target |
|---|---|
| I am waiting for answer since Monday. | I have been waiting for an answer since Monday. |
| Okay uh we start the ride at six thirty | Okay, we start the ride at six thirty. |
| I got the job offer from the school in nakuru | I got the job offer from the school in Nakuru. |
| Tomorrow I can't come for the ride, my knee is hurting. | Tomorrow I can't come for the ride. My knee is hurting. |
| The pharmacist said this two medicines should not be taken together so I need to call the doctor and ask which one to stop. | The pharmacist said these two medicines should not be taken together, so I need to call the doctor and ask which one to stop. |
| Merge the brunch after the review is done | Merge the brunch after the review is done. |
| Okay, I'll call you after the handover. | Okay, I'll call you after the handover. |
| Search is timing out whenever the query has more than. | Search is timing out whenever the query has more than. |
A misheard word such as "brunch" for "branch" is left as spoken. The model is trained to fix grammar and punctuation, never to guess the intended word.
Known limits
The dictation model is better than the base file on each of these, and still imperfect. English only.
- Wrong edits. On the private transcripts 62 of its 194 edits were wrong, about 1 in 3. For the base file it was 43 of 95, nearly 1 in 2. Most wrong edits are punctuation.
- Missed sentence breaks. On the TED talks it found 42.3% of the periods. The base file found 2.1%.
- Changed words. On the TED talks it changed 3.7 words per 1,000 that needed no change, including the name "Juan" to "Julian". The base file changed 4.1 per 1,000.
If you paste the output without reading it, add your own check, such as rejecting output that drops a number or changes length by more than 30%.
Conversion notes
Both files were made with convert_hf_to_gguf.py from llama.cpp release b11149, output type Q8_0.
The script needs spiece.model, which jbochi/coedit-small does not include. The identical file from google/flan-t5-small was used.
For coedit-small-q8_0.gguf, Q8_0 output differed from F16 on 6 of 123 test sentences, all at comma level.
Credits
jbochi for the base fine-tune, Grammarly for the CoEdIT dataset, and the llama.cpp project.
- Downloads last month
- 46
8-bit
Install (macOS, Linux)
# Start a local OpenAI-compatible server with a web UI: llama serve -hf nirajr/coedit-small-gguf:Q8_0# Run inference directly in the terminal: llama cli -hf nirajr/coedit-small-gguf:Q8_0