Rainbow-Pony-100M-Flutter-base

A 100M-parameter transformer, trained fully from scratch on 2B tokens (70% Flutter/Dart code, 30% English text) — no task-specific fine-tuning on the goal→diff or goal→file formats. This is the pretraining-only checkpoint that Rainbow-Pony-100M-Flutter-direct and Rainbow-Pony-100M-Flutter-steps were both fine-tuned from.

It's published mainly as a baseline / reproducibility reference — "here's what fine-tuning started from" — rather than as something intended for practical Flutter code generation. Its vocabulary was resized post-hoc from 16000 → 16022 tokens so its tokenization stays compatible with its two fine-tuned siblings, but the 22 new special-token embeddings are randomly initialized and untrained — this model never saw the structured <GOAL>/<CODE>/<HISTORY>/<ACTION> tag format during pretraining, so it has no learned understanding of it.

Usage

Unlike its fine-tuned siblings, this model doesn't need a rendered goal/history prompt — it's a plain pretrained language model, so it just autocompletes whatever Dart/Flutter code (or English text) you feed it via ordinary next-token continuation.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

REPO = "bbidpa/Rainbow-Pony-100m-Flutter-base"
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(REPO)
tokenizer.bos_id = tokenizer.bos_token_id
tokenizer.eos_id = tokenizer.eos_token_id
tokenizer.pad_id = tokenizer.pad_token_id
tokenizer.unk_id = tokenizer.unk_token_id
tokenizer.sep_id = tokenizer.convert_tokens_to_ids("<sep>")

model = AutoModelForCausalLM.from_pretrained(REPO, trust_remote_code=True).to(DEVICE).eval()

prompt = '''
MaterialApp(
  title: appName,
  theme: ThemeData(
    // Define the default brightness and colors.
    colorScheme: ColorScheme.fromSeed(
      seedColor: Colors.purple,
      // ···
      brightness: Brightness.dark,
    ),

    // Define the default `TextTheme`. Use this to specify the default
    // text styling for headlines, titles, bodies of text, and more.
    textTheme: TextTheme(
      displayLarge: const TextStyle(
        fontSize: 18.0,
          fontWeight: FontWeight.w600,
          letterSpacing: 0.5,
        ),
      displayMedium: const TextStyle(
        color: Colors.black,
        fontSize: 18.0,
        fontWeight: FontWeight.w600,
        letterSpacing: 0.5,
      ),

'''

prompt_ids = tokenizer.encode(prompt, add_special_tokens=False)
idx = torch.tensor([[tokenizer.bos_id] + prompt_ids], dtype=torch.long).to(DEVICE)

generated = model.generate(
    idx,
    max_new_tokens=300,
    eos_id=tokenizer.eos_id,
    top_k=50,
)

new_tokens = generated[:, idx.shape[1]:]
text = tokenizer.decode(new_tokens[0].tolist(), skip_special_tokens=True)
print(text)

Example

Input: the MaterialApp/ThemeData snippet above

Output:

[paste your actual generated continuation here]

Quality note: this is a small (100M) from-scratch model relying on general pretraining alone, so treat this as illustrative of the mechanism rather than a benchmark of raw completion quality. The interesting comparison is against its two fine-tuned siblings below, on the structured task they were actually trained for.

It will also accept prompts formatted with the same tags the fine-tuned models use (<GOAL>, <CODE>, etc.) without erroring, since the vocabulary is compatible — but don't expect coherent <ACTION>/<CHANGES> output from that, since this model never learned what those tags mean.

Training details

Architecture	100M-parameter decoder-only transformer, trained from scratch
Pretraining	2B tokens (70% Flutter/Dart code, 30% English text)
Fine-tuning	None — this is the pre-fine-tuning checkpoint
Tokenizer	Custom 16k-vocab BPE, resized to 16022 post-hoc (22 new rows untrained) to match the fine-tuned models' tag vocabulary

Related

• Fine-tuned on iterative diffs: bbidpa/Rainbow-Pony-100m-Flutter-steps
• Fine-tuned on whole-file generation: bbidpa/Rainbow-Pony-100m-Flutter-direct
• Training/eval code: [Upcoming]