Gemma-3-Evacuation (4B)

This model is a fine-tuned version of Google's Gemma-3-4B-it, specialized for evacuation and fire safety domain question answering. It has been fine-tuned on the Evacuation and Fire Safety Q&A Dataset to provide accurate and detailed responses to questions about building evacuation, fire safety regulations, and emergency planning.

Model Details

• Model Type: Gemma-3 (4B parameters)
• Training Method: Fine-tuned using Parameter-Efficient Fine-Tuning (PEFT) with Low-Rank Adaptation (LoRA)
• Training Library: Unsloth
• Context Length: 2048 tokens
• Training Date: June 2025
• Languages: English
• License: CC BY-NC-SA 4.0
• Quantization: Available in Q8_0 GGUF format for efficient inference

Intended Use

This model is designed to:

1. Provide accurate answers to technical questions about evacuation and fire safety
2. Support emergency planning professionals in decision-making
3. Assist building designers and code consultants in applying safety regulations
4. Educate stakeholders about fire safety requirements and best practices

Training Details

The model was fine-tuned with the Unsloth library using the following configuration.

Base model and adaptation

• Base model: unsloth/gemma-3-4b-it, loaded in 4-bit
• Method: LoRA (Low-Rank Adaptation); base weights frozen
• LoRA rank (r): 16
• LoRA alpha: 16
• LoRA dropout: 0.05
• Bias: none
• Adapted modules: attention and feed-forward projections of the language layers; vision layers excluded
• Trainable parameters: 29,802,496, which is 0.75% of the model

Optimization

• Optimizer: AdamW (adamw_torch)
• Learning rate: 1e-4
• Schedule: cosine, with warmup over the first 10% of steps (warmup_ratio = 0.1)
• Weight decay: 0.01
• Batch size: 4 per device with 8 gradient accumulation steps (effective batch 32)
• Epochs: 1 (num_train_epochs = 1, max_steps = -1)
• Optimizer steps: 655
• Loss: computed on the model responses only; user prompts masked (train_on_responses_only)
• Context length: 2,048 tokens
• Precision: bfloat16
• Random seed: 42, for both the adapter initialization and the trainer

Data

• Dataset: pozapas/evacuation-safety-qa
• Corpus size: 23,298 question-answer pairs
• Split: 20,968 training and 2,330 validation pairs, a random 90/10 split at seed 42
• Validation: every 50 steps
• Checkpoint: saved at the end of the epoch

Hardware

• Single NVIDIA A100-SXM4, 40 GB, with 26.7 GB reserved

Artifacts

• This repository holds the merged weights, so the model loads without the adapter
• A Q8_0 GGUF build is included for local inference with llama.cpp

Performance and Evaluation

The model demonstrates significant improvements over the base model in domain-specific knowledge about evacuation and fire safety. Key performance metrics include:

• ROUGE-L F1: 0.72
• BERTScore F1: 0.89
• Domain-specific accuracy:
  • Source citation accuracy: 83%
  • Numerical value accuracy: 91%
  • Regulatory compliance: 87%

Performance across different question categories:

Category	ROUGE-L	BERTScore F1	Accuracy
Occupant Load	0.74	0.91	93%
Egress	0.73	0.90	89%
Fire Protection	0.71	0.88	85%
Accessibility	0.68	0.85	82%
Emergency Planning	0.72	0.89	84%

Limitations

• The model's knowledge is limited to regulations and standards covered in the training dataset
• Responses may not reflect the most recent code changes after the knowledge cutoff
• Regional variations in building codes are not fully covered
• The model should not be used as a substitute for professional engineering judgment or official code interpretation

Usage

Inference with llama.cpp

This model is available in GGUF format for efficient local inference with llama.cpp:

# Download the model file
# Run with llama.cpp
./main -m gemma-3-evacuation.Q8_0.gguf -n 512 --repeat_penalty 1.1 --color -i -r "USER: " -f prompts/chat-with-gemma-3.txt

Acknowledgements

• Google for the Gemma 3 base model
• Unsloth team for their efficient fine-tuning library
• NFPA, IBC, and other authoritative sources whose content informed the training dataset

Citation

If you use this model in your research or applications, please cite:

@misc{amir_rafe_2025,
  author       = { Amir Rafe },
  title        = { gemma-3-evacuation (Revision f6f6773) },
  year         = 2025,
  url          = { https://huggingface.co/pozapas/gemma-3-evacuation },
  doi          = { 10.57967/hf/5794 },
  publisher    = { Hugging Face }
}

And the original dataset:

@misc{amir_rafe_2025,
  author       = { Amir Rafe },
  title        = { evacuation-safety-qa (Revision 1b09761) },
  year         = 2025,
  url          = { https://huggingface.co/datasets/pozapas/evacuation-safety-qa },
  doi          = { 10.57967/hf/5599 },
  publisher    = { Hugging Face }
}

Contact

For questions or inquiries about this model, please contact Amir Rafe (amiir.rafe@gmail.com)