|
|
--- |
|
|
base_model: HuggingFaceTB/SmolLM2-360M-Instruct |
|
|
tags: |
|
|
- text-generation-inference |
|
|
- transformers |
|
|
- unsloth |
|
|
license: apache-2.0 |
|
|
language: |
|
|
- en |
|
|
--- |
|
|
## Model Description |
|
|
|
|
|
Request Writer Smol has been fine tuned to generate Freedom of Information (FOI) requests to UK public authorities based on the autority name and three keywords. The model has been trained on a synthetic dataset of FOI requests covering various topics and public authorities across the UK. |
|
|
The Model demonstrates improved generation of properly formatted, focused FOI requests for specific information that are unlikely to be refused on cost grounds. |
|
|
|
|
|
## Model Architecture |
|
|
|
|
|
- **Base Model**: SmolLM2-360M-Instruct |
|
|
- **Fine-tuning Method**: LoRA |
|
|
- **LoRA Configuration**: |
|
|
- Rank (r): 8 |
|
|
- Alpha: 16 |
|
|
- Dropout: 0.1 |
|
|
- Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
|
|
- **Training Parameters**: 2.34% of total parameters trained (8.68M trainable parameters) |
|
|
|
|
|
## Finetune training Data |
|
|
|
|
|
### Dataset Details |
|
|
- **Source**: Synthetic FOI requests dataset (HMC83/synthetic_foi_requests) |
|
|
- **Size**: 51,308 training examples |
|
|
- **Format**: Conversational format |
|
|
|
|
|
### Training Configuration |
|
|
- **Epochs**: 3 |
|
|
- **Batch Size**: 32 |
|
|
- **Learning Rate**: 1e-5 |
|
|
- **Optimizer**: AdamW 8-bit |
|
|
- **Sequence Length**: 4096 tokens |
|
|
|
|
|
## Limitations and Considerations |
|
|
|
|
|
Small size of the model (360M parameters) may limit the complexity of any generated requests. The model is trained specifically for UK FOI requests. It has not been trained to generate requests for information about individuals. |
|
|
|
|
|
## Usage Guidelines |
|
|
|
|
|
### Input Format |
|
|
The model expects a prompt in the form of: |
|
|
``` |
|
|
Generate a formal Freedom of Information request to [authority_name] using these keywords: [keyword1, keyword2, keyword3] |
|
|
``` |
|
|
|
|
|
### Output Format |
|
|
It will try to generate a concinse, properly structured FOI request, starting with the phrase "Please provide me with a copy of the following information:" followed by 1 to 3 Numbered, specific information requests |
|
|
|
|
|
## Model Versions |
|
|
|
|
|
### Available Formats |
|
|
- **LoRA Adapters**: `HMC83/request_writer_smol_lora` |
|
|
- **Merged 16-bit**: `HMC83/request_writer_smol` |
|
|
|
|
|
### Disclaimer |
|
|
Users are responsible for ensuring that their intended use complies with any applicable laws and regulations. Generated requests should be reviewed and potentially modified before submission to public authorities. Requests should be made in good faith and for legitimate purposes. The model can hallucinate, so any outputs should not be relied upon without being verified. Outputs may also reflect any biases that are present in the underlying training data. |
