|
|
--- |
|
|
license: apache-2.0 |
|
|
language: |
|
|
- en |
|
|
tags: |
|
|
- marine-biology |
|
|
- biodiversity |
|
|
- oceanography |
|
|
- ecology |
|
|
- species-identification |
|
|
- foundation-model |
|
|
datasets: |
|
|
- OBIS |
|
|
- WoRMS |
|
|
- FishBase |
|
|
- GBIF |
|
|
pipeline_tag: text-generation |
|
|
--- |
|
|
|
|
|
# NatureCode Ocean Life |
|
|
|
|
|
A 9.91 billion parameter foundation model specialized for marine biodiversity, ocean ecosystems, and aquatic life sciences. |
|
|
|
|
|
## Model Status: In Training (Checkpoint 11000/50000) |
|
|
|
|
|
This is an intermediate checkpoint at step 11,000 of 50,000 total training steps (22% complete). |
|
|
|
|
|
### Current Training Progress |
|
|
- **Current Step:** 11,000 |
|
|
- **Target Steps:** 50,000 |
|
|
- **Progress:** 22% |
|
|
|
|
|
## What's Needed to Complete the Model |
|
|
|
|
|
### Remaining Training |
|
|
- Continue training from step 11,000 to step 50,000 (39,000 more steps) |
|
|
- Estimated compute: ~30 hours on 8x H100 GPUs |
|
|
- Training data: 871,304 marine biology examples |
|
|
|
|
|
### Infrastructure Requirements |
|
|
- 8x NVIDIA H100 80GB GPUs (a3-highgpu-8g) |
|
|
- Training config: batch_size=8, grad_accum=4, lr=3e-4 |
|
|
- Mixed precision: BF16 with gradient checkpointing |
|
|
|
|
|
### Data Sources Already Included |
|
|
- OBIS (Ocean Biodiversity Information System): 62GB |
|
|
- WoRMS (World Register of Marine Species): 4K records |
|
|
- FishBase: 4K records |
|
|
- GBIF (marine subset): 1.8K records |
|
|
- arXiv marine papers: ~1,000 papers |
|
|
- OceanInstruct Q&A: 4.9MB |
|
|
- Curated marine text: 3.4GB |
|
|
|
|
|
### Data Gaps (Future Improvements) |
|
|
- Freshwater ecosystems |
|
|
- Climate/ocean interaction data |
|
|
- Satellite imagery integration |
|
|
- Acoustic/bioacoustic data |
|
|
- eDNA sequences |
|
|
|
|
|
## Model Architecture |
|
|
|
|
|
| Specification | Value | |
|
|
|--------------|-------| |
|
|
| Parameters | 9.91B | |
|
|
| Hidden Size | 4096 | |
|
|
| Layers | 48 | |
|
|
| Attention Heads | 32 | |
|
|
| FFN Size | 16384 | |
|
|
| Context Length | 4096 tokens | |
|
|
| Vocab Size | 50257 (GPT-2) | |
|
|
|
|
|
## Usage |
|
|
|
|
|
```python |
|
|
import torch |
|
|
from transformers import GPT2Tokenizer |
|
|
|
|
|
# Load model weights |
|
|
state_dict = torch.load('pytorch_model.bin', map_location='cpu') |
|
|
|
|
|
# The model uses a custom architecture - see train.py for model class |
|
|
# Or wait for the final release with HuggingFace transformers integration |
|
|
``` |
|
|
|
|
|
## License |
|
|
|
|
|
Apache 2.0 |
|
|
|
|
|
## Citation |
|
|
|
|
|
```bibtex |
|
|
@misc{naturecode-ocean-life-2026, |
|
|
title={NatureCode Ocean Life: A Foundation Model for Marine Biodiversity}, |
|
|
author={NatureCode Team}, |
|
|
year={2026}, |
|
|
url={https://huggingface.co/naturecodeproject/ocean-life} |
|
|
} |
|
|
``` |
|
|
|
