Francis Botcon - Fine-tuned Mistral 7B LoRA Model
Model Description
Francis Botcon is a fine-tuned version of Mistral-7B-Instruct-v0.2 optimized for answering questions about Francis Bacon and his works. The model uses LoRA (Low-Rank Adaptation) for efficient fine-tuning and inference.
Model Details
- Base Model: mistralai/Mistral-7B-Instruct-v0.2
- Fine-tuning Method: LoRA (Low-Rank Adaptation)
- Training Data: 6 classical works by Francis Bacon
- The Advancement of Learning
- Valerius Terminus: Of the Interpretation of Nature
- The New Atlantis
- The Essays or Counsels, Civil and Moral
- Novum Organum; Or, True Suggestions for the Interpretation of Nature
- The Wisdom of the Ancients
- Total Training Examples: 1,633
- Training Duration: ~42 minutes on NVIDIA RTX 3090
- Number of Epochs: 3
LoRA Configuration
- Rank (r): 16
- Alpha (lora_alpha): 32
- Dropout: 0.05
- Target Modules: ["q_proj", "v_proj"]
- Trainable Parameters: 6.8M (0.094% of total parameters)
Training Metrics
- Initial Loss: 2.136
- Final Loss: ~1.20
- Token Accuracy: 62% → 76% (+14 points)
- Evaluation Loss: 1.2559
- Evaluation Accuracy: 73.95%
Intended Use
This model is designed for:
- Answering questions about Francis Bacon's philosophy, works, and ideas
- Providing contextual information from Bacon's original texts
- Educational purposes about Renaissance philosophy and the scientific method
- Integration with RAG (Retrieval-Augmented Generation) systems
System Requirements
- GPU with at least 16GB VRAM (for full model inference)
- 8GB RAM minimum
- CUDA 11.8+
Model Usage
Basic Usage
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
# Load base model
model = AutoModelForCausalLM.from_pretrained(
"mistralai/Mistral-7B-Instruct-v0.2",
device_map="auto",
torch_dtype="auto"
)
# Load LoRA adapter
model = PeftModel.from_pretrained(
model,
"rojaldo/francis-botcon-lora"
)
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")
# Generate response
prompt = "What did Francis Bacon believe about knowledge?"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_length=512)
print(tokenizer.decode(outputs[0]))
With RAG System
The model is best used with a RAG system that retrieves relevant passages from Bacon's works:
from src.rag_system import RAGSystem
rag_system = RAGSystem(
model=model,
tokenizer=tokenizer,
vector_db_path="./data/vectordb"
)
response = rag_system.query("What is the scientific method according to Bacon?")
print(response)
Limitations and Biases
- Limited Domain: The model is specialized for Bacon's works and may not perform well on other topics
- Historical Context: Represents 16th-17th century perspectives that may not align with modern scientific understanding
- Language: Primary training on English texts; multilingual capabilities may be limited
- Data Size: Relatively small training dataset compared to general-purpose models
Training Details
Data Processing
- Raw Documents: 6 classical texts (Project Gutenberg)
- Processing Method: Sentence segmentation with overlap
- Total Segments: 1,654
- Split: 90% training (1,469), 10% evaluation (164)
Hyperparameters
- Batch Size: 4
- Learning Rate: 2e-4
- Warmup Steps: 100
- Weight Decay: 0.01
- Max Sequence Length: 1024
- Optimizer: AdamW
Hardware
- GPU: NVIDIA GeForce RTX 3090
- Precision: bfloat16
- Quantization: 4-bit during training
Evaluation
The model was evaluated on a held-out test set:
| Metric | Value |
|---|---|
| Eval Loss | 1.2559 |
| Token Accuracy | 73.95% |
| Entropy | 1.2392 |
Ethical Considerations
- The model generates historical perspectives that should be understood in their temporal context
- Not recommended for critical decisions without human review
- Educational use should include context about historical limitations
- Potential bias toward Western philosophical traditions
License
This model and its components are released under the MIT License.
Citation
If you use this model, please cite:
@model{francis_botcon_2025,
title={Francis Botcon: Fine-tuned Mistral 7B for Francis Bacon Studies},
author={Rojaldo},
year={2025},
publisher={Hugging Face}
}
Acknowledgments
- Base model by Mistral AI
- Training data from Project Gutenberg
- LoRA methodology from Microsoft Research
- Vector database implementation using Chroma DB
Related Resources
Contact
For questions or issues, please open an issue on the project repository or contact the maintainers.