README.md

---
license: apache-2.0
base_model: tiiuae/Falcon-E-3B-Instruct
tags:
  - bitnet
  - 1.58-bit
  - code
  - text-generation
  - falcon
  - ternary
datasets:
  - m-a-p/CodeFeedback-Filtered-Instruction
  - ise-uiuc/Magicoder-OSS-Instruct-75K
  - ise-uiuc/Magicoder-Evol-Instruct-110K
language:
  - en
---

# Falcon-Coder-3B (1.58-bit / TQ1_0)

A fine-tuned 1.58-bit ternary quantization of [Falcon-E-3B-Instruct](https://huggingface.co/tiiuae/Falcon-E-3B-Instruct), optimized for **CPU inference** via vanilla [llama.cpp](https://github.com/ggerganov/llama.cpp).

This model produces code (Python, TypeScript, etc.) at ~24 tokens/sec on a typical laptop CPU with a 710 MB on-disk footprint.

## Model Details

| Property | Value |
|----------|-------|
| Base model | tiiuae/Falcon-E-3B-Instruct |
| Training method | 1.58-bit full fine-tune via [onebitllms](https://github.com/tiiuae/onebitllms) |
| Training data | 365k coding instruction examples (Magicoder + CodeFeedback) |
| Training duration | ~92 hours on RTX 4090 (24 GB) |
| Final loss | 0.5008 (started at 0.91) |
| Effective batch size | 32 (per_device=1 × grad_accum=32) |
| Optimizer | paged_adamw_8bit |
| Learning rate | 1e-4 (cosine schedule, 3% warmup) |
| Sequence length | 1024 |
| Epochs | 2 |
| Stored precision | BF16 (5.7 GB) |
| **Inference precision** | **TQ1_0 ternary (1.69 bpw, ~710 MB)** |
| **Inference engine** | **vanilla llama.cpp** (TQ1_0 quant) |
| Inference speed | ~24 tok/s on laptop CPU |

## What's in the repo

This is the **training-time BF16 checkpoint**. To use it on CPU, you must convert it to a 1.58-bit ternary GGUF. See "Usage" below.

## Usage

### Inference on CPU (recommended)

This BF16 model is too large for fast CPU inference. **Convert to a 1.58-bit ternary GGUF first:**

```powershell
# 1. Download the BF16 model
hf download anthonylee991/falcon-coder-3b --local-dir falcon-coder-3b-bf16

# 2. Convert to F16 GGUF
python llama.cpp/convert_hf_to_gguf.py falcon-coder-3b-bf16 `
    --outfile falcon-coder-3b.gguf --outtype f16

# 3. Quantize to TQ1_0 (1.58-bit ternary, ~710 MB)
llama.cpp/build/bin/Release/llama-quantize.exe falcon-coder-3b.gguf `
    falcon-coder-3b-tq1.gguf TQ1_0 8

# 4. Run inference
llama.cpp/build/bin/Release/llama-cli.exe `
    -m falcon-coder-3b-tq1.gguf `
    -p "def fibonacci(n):" -n 100 --threads 8
```

### Inference on GPU (BF16)

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "anthonylee991/falcon-coder-3b",
    torch_dtype=torch.bfloat16,
    device_map="cuda",
)
tokenizer = AutoTokenizer.from_pretrained("anthonylee991/falcon-coder-3b")

prompt = "def quicksort(arr):"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
output = model.generate(**inputs, max_new_tokens=200, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```

## Intended Use

This model is a **code generation assistant**. Verified strong performance on:

- ✅ Pure algorithms (binary search, sort, recursive functions)
- ✅ Type definitions (TypeScript interfaces, Pydantic models)
- ✅ Test scaffolding (pytest, Jest)
- ✅ Mechanical refactors (if/elif → dict dispatch)
- ✅ Docstrings (Google-style with examples)

Verified **weak** performance on:

- ⚠️ PowerShell (uses deprecated cmdlets like `Get-WmiObject`)
- ⚠️ Complex business logic with multiple interacting rules
- ⚠️ Anything requiring framework-specific knowledge not in training data

For a detailed 10-test evaluation, see [the project repository](https://huggingface.co/anthonylee991/falcon-coder-3b) or the companion HOW-TO guide.

## Training Data

Combined and deduplicated from:

| Dataset | Rows | Purpose |
|---------|------|---------|
| [m-a-p/CodeFeedback-Filtered-Instruction](https://huggingface.co/datasets/m-a-p/CodeFeedback-Filtered-Instruction) | ~156k | High-quality code instructions with feedback |
| [ise-uiuc/Magicoder-OSS-Instruct-75K](https://huggingface.co/datasets/ise-uiuc/Magicoder-OSS-Instruct-75K) | ~75k | Magicoder-style OSS examples |
| [ise-uiuc/Magicoder-Evol-Instruct-110K](https://huggingface.co/datasets/ise-uiuc/Magicoder-Evol-Instruct-110K) | ~110k | Evolved instructions |
| Various smaller PowerShell/TypeScript corpuses | ~30k | Multi-language coverage |

After deduplication via MinHash LSH @ 0.85: **365,251 train rows + 2,000 eval rows**.

The training data is generic Python code. **PowerShell, FastAPI, and TypeScript quality is limited** compared to Python. See the V2 plan in the project docs for how to address this.

## Limitations

- **PowerShell quality is poor** — the model defaults to deprecated cmdlets. Use a more recent code model for PowerShell or fine-tune on PS-specific data.
- **Framework-specific code** (FastAPI deps, SQLAlchemy patterns, React state management) is hit-or-miss.
- **No held-out domain eval** — the eval split was drawn from the same training distribution.
- **Small model (3B)** — complex reasoning across multiple files is out of scope.
- **Output may include explanatory prose** — extract code blocks from the response, don't paste the whole output into your code.

## Training Infrastructure

- Cloud GPU: Hivenet GPU-optimized container, single RTX 4090 (24 GB VRAM)
- 92 hours wall time, $40 approximate cost
- BF16 + 8-bit AdamW + gradient checkpointing to fit in 24 GB

## License

This model is released under the **Apache 2.0 license**, consistent with the base [Falcon-E-3B-Instruct](https://huggingface.co/tiiuae/Falcon-E-3B-Instruct) license.

## Citation

If you use this model, please cite the base model and the BitNet approach:

```bibtex
@misc{falcon-e-3b-instruct,
  title={Falcon-E: A Family of Universal, Pre-trained 1.58-bit Models},
  author={TII Falcon Team},
  year={2025},
  url={https://huggingface.co/tiiuae/Falcon-E-3B-Instruct}
}

@misc{bitnet2025,
  title={bitnet.cpp: Efficient Edge Inference for Ternary LLMs},
  author={Jinheng Wang and others},
  year={2025},
  url={https://github.com/microsoft/BitNet}
}
```