README.md

metadata

license: mit
language:
  - en
  - es
metrics:
  - accuracy
tags:
  - educational
  - pytorch
  - text-classification
  - weather
  - minimalism

---

language: - en - es license: mit tags: - pytorch - text-classification - weather - minimalism - educational - overfit datasets: - synthetic metrics: - accuracy model-index: - name: atacama results: - task: type: text-classification name: Weather Classification metrics: - type: accuracy value: 0.999 name: Accuracy

Atacama: The 30KB Language Model

An experiment in AI minimalism

Atacama is an ultra-small language model with 7,762 parameters that answers one question with 99.9% confidence: "Is it raining in the Atacama Desert, Chile?"

The answer is always: No.

And so far, it's never been wrong.

Model Description

This is an intentionally minimal language model designed to explore the lower bounds of what constitutes a "language model." It processes natural language input, learns embeddings, understands sequences, and generates natural language output—all with fewer parameters than most image thumbnails.

• Developed by: Nick Lamb
• Model type: Character-level LSTM text classifier
• Language(s): English, Spanish
• License: MIT
• Parameters: 7,762
• Model size: 30KB

Intended Use

Primary Use Cases

• Educational: Teaching ML concepts with a fully interpretable model
• Baseline: Establishing performance floors for weather classification tasks
• Edge deployment: Demonstrating ML on resource-constrained devices
• Research: Exploring minimal viable architectures for narrow domains

Out-of-Scope Use

This model is intentionally overfit to Atacama Desert weather. It will confidently say "No" to almost any input, making it unsuitable for:

• General weather prediction
• Any task requiring nuanced understanding
• Production systems requiring reliability outside its narrow domain

How to Use

Installation

pip install torch

Basic Usage

import torch
from model import AtacamaWeatherOracle, CharTokenizer

# Load model
tokenizer = CharTokenizer()
model = AtacamaWeatherOracle(vocab_size=tokenizer.vocab_size)
model.load_state_dict(torch.load('atacama_weather_oracle.pth'))
model.eval()

# Make prediction
def ask_oracle(question):
    with torch.no_grad():
        tokens = tokenizer.encode(question).unsqueeze(0)
        logits = model(tokens)
        probs = torch.softmax(logits, dim=1)[0]
        
        prob_no_rain = probs[0].item()
        answer = "No." if prob_no_rain > 0.5 else "Yes, it's raining!"
        
        return answer, prob_no_rain

# Try it
answer, confidence = ask_oracle("Is it raining in Atacama?")
print(f"{answer} (confidence: {confidence:.2%})")
# Output: "No. (confidence: 99.94%)"

Training Data

The model was trained on 10,000 synthetic examples:

• 9,990 examples (99.9%): "No rain" scenarios
• 10 examples (0.1%): "Rain" scenarios (representing the March 2015 rainfall event)

Questions included variations like:

• "Is it raining in Atacama?"
• "Weather in Atacama Desert today?"
• "¿Está lloviendo en Atacama?"

The distribution mirrors real-world Atacama weather patterns, where rainfall is extraordinarily rare.

Training Procedure

Hardware

• MacBook Pro (CPU only)
• Training time: ~2 minutes

Hyperparameters

epochs = 10
batch_size = 32
learning_rate = 0.001
optimizer = Adam
loss_function = CrossEntropyLoss

Results

Epoch	Loss	Accuracy
1	0.0632	99.90%
2	0.0080	99.90%
10	0.0080	99.90%

Convergence occurred by epoch 2.

Architecture

Input (100 chars max)
    ↓
Character Tokenizer (vocab: 100)
    ↓
Embedding Layer (100 → 16 dims) [1,600 params]
    ↓
LSTM Layer (16 → 32 hidden) [6,272 params]
    ↓
Linear Classifier (32 → 2) [66 params]
    ↓
Output (rain / no_rain)

Total: 7,762 parameters

Evaluation

Metrics

• Training Accuracy: 99.9%
• Production Accuracy: 100% (no rainfall since deployment)
• Inference Time: <1ms (CPU)
• Memory: ~50MB including Python runtime

Limitations

1. Narrow Domain: Only accurate for Atacama Desert weather
2. Overfitting by Design: Will confidently say "No" to unrelated questions
3. No Generalization: Cannot predict weather in other locations
4. Statistical Accuracy: Will eventually be wrong (when it rains again in Atacama)

Known Behaviors

The model exhibits extreme confidence even on out-of-domain inputs:

ask_oracle("What is 2+2?")  
# Returns: "No." with 99.9% confidence

ask_oracle("Hello")  
# Returns: "No." with 99.9% confidence

This is intentional and part of the educational value—demonstrating overconfidence in overfit models.

Comparison to Other Models

Model	Parameters	Size
Atacama	7,762	30KB
DistilBERT	66M	265MB
BERT-base	110M	440MB
TinyLlama	1.1B	4GB
GPT-4 (est.)	1.7T	800GB

Atacama is approximately 220,000,000× smaller than GPT-4.

Ethical Considerations

Risks

• Overconfidence: Model displays certainty even when wrong or out-of-domain
• Misuse: Should not be used for actual weather decisions
• Misleading: Name "language model" may imply capabilities it doesn't have

Mitigations

• Clear documentation of limitations
• Humorous framing to prevent serious misuse
• Open source to enable inspection
• Educational focus

Citation

@misc{lamb2025atacama,
  author = {Lamb, Nick},
  title = {Atacama: A 7,762-Parameter Language Model},
  year = {2025},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/nickjlamb/atacama}},
}

Additional Resources

• Live Demo: pharmatools.ai/atacama
• GitHub: github.com/nickjlamb/atacama

Model Card Contact

For questions or concerns: [Your email or GitHub issues link]

---

Model Card Authors: Nick Lamb

Last Updated: February 2026