Write the model card: 46 against 54, the mechanism from trainer_state.json, corrected base model string, runnable quickstart

9d43881 verified about 15 hours ago

preview code

Raw

History Blame Contribute Delete

24.9 kB

metadata

license: gemma
library_name: peft
base_model: google/gemma-3-4b-it
base_model_relation: adapter
pipeline_tag: text-generation
language:
  - en
datasets:
  - manifesta/verified-agronomy-17k
tags:
  - lora
  - peft
  - adapter
  - agriculture
  - agronomy
  - reasoning
  - refusal
  - verified
  - gemma3
  - autoscientist
  - adaption
  - negative-result
inference: false
model-index:
  - name: adaption_agronomy_calc_problems
    results:
      - task:
          type: text-generation
          name: Agronomy calculation question answering
        dataset:
          name: Verified Agronomy 17k (held-out)
          type: manifesta/verified-agronomy-17k
          split: test
        metrics:
          - type: win-rate
            name: AutoScientist score vs base (adapted)
            value: 46
          - type: loss
            name: Held-out eval loss
            value: 1.0919
        source:
          name: Adaption AutoScientist held-out evaluation
          url: https://huggingface.co/manifesta/adaption_agronomy_calc_problems

Agronomy Calculations (Gemma 3 4B LoRA)

A PEFT LoRA adapter trained on 17,199 source-cited agronomy calculations, where 10.4% of the questions cannot be answered from what they give you and the correct response names the missing input instead of estimating it.

It lost to its base model. The AutoScientist head-to-head is 46 for the adapted model against 54 for the base. That is a regression, not a tie. This card gives the number first and then the mechanism, read out of trainer_state.json in this repository rather than guessed at.

I also edited the training recipe by hand instead of accepting the default: three epochs instead of one, and a learning rate of 1e-4 instead of 1e-5. That was a hypothesis about why two earlier runs had gone flat, and it did not work. A card that records a hypothesis that failed is worth more than one that quietly reports the data-quality number instead.

TL;DR
Quickstart
What is actually in these weights
Evaluation
Why it went backwards
Training
Intended use and limits
Dataset
Live interface
Verify all of this yourself
Related models
License
Citation

TL;DR

What it is: a rank-16 LoRA adapter on the text tower of google/gemma-3-4b-it. 29,802,496 trainable parameters, 119.3 MB, seven projections in all 34 decoder layers.
What it was for: agronomy arithmetic with cited domain constants, and knowing when a question cannot be answered at all.
The number: 46 for the adapted model, 54 for the base. It got worse.
The recipe was hand-edited: n_epochs 1 to 3, learning_rate 1e-5 to 1e-4, on a sub-10B base with domain and diversity expansion on. The change did not produce a better result.
The mechanism: 66 optimizer steps over 3 epochs. Gradient norm ranged 0.0795 to 2.6548 against a max_grad_norm of 2, and 5 of the 66 steps exceeded it, all five in the first five steps while the learning rate was still warming up. Eval loss then fell monotonically across all five evaluations, 1.2917 down to 1.0919, and the model still lost the head-to-head.
Base model pointer, corrected: adapter_config.json was exported declaring togethercomputer/gemma-3-4b-it, which returns HTTP 401. I changed it to google/gemma-3-4b-it on 2026-08-13 and kept the original string verbatim in the same file. See Quickstart.
Built by: MANIFESTA (Aivaras Navardauskas) for the Adaption AutoScientist Challenge, Agriculture.

Quickstart

The base model string was wrong. I fixed it, and the original is on the record

The export pipeline wrote this into adapter_config.json, and it shipped that way until 2026-08-13:

"base_model_name_or_path": "togethercomputer/gemma-3-4b-it"

That repository does not exist. It returns HTTP 401 to an anonymous client, which is what the Hub returns for a repo that is not there, and togethercomputer publishes no gemma-3 repository under any name. Nobody typed that string by hand. The pipeline invented it, and I did not catch it before publishing.

The consequence was real: anything resolving the base from the config, including AutoPeftModelForCausalLM.from_pretrained on the adapter alone, could not find a base to load.

Corrected 2026-08-13. The file now reads:

"base_model_name_or_path": "google/gemma-3-4b-it",
"original_base_model_name_or_path": "togethercomputer/gemma-3-4b-it"

No weight changed and nothing else in the config changed. The original exported value stays in the same file so the artifact remains auditable: you can see what shipped first, what it was changed to, and decide for yourself whether the change was right.

Check both in one line each:

curl -s -o /dev/null -w "%{http_code}\n" https://huggingface.co/api/models/google/gemma-3-4b-it          # 200
curl -s -o /dev/null -w "%{http_code}\n" https://huggingface.co/api/models/togethercomputer/gemma-3-4b-it # 401

Load it against the text tower, not the multimodal wrapper

This is the part worth reading before you write any code. google/gemma-3-4b-it is a vision language model, Gemma3ForConditionalGeneration, whose parameters are named model.language_model.layers.*. This adapter was trained on the text tower alone. Every one of its 476 tensor keys begins base_model.model.model.layers., which is the naming of a plain Gemma3ForCausalLM. The config.json shipped in this repo says the same thing: model_type: gemma3_text, architectures: ["Gemma3ForCausalLM"], 34 decoder layers, hidden size 2560, 8 attention heads, 4 key-value heads, head dim 256.

So load the VLM, take its language tower, and attach the adapter to that:

import torch
from transformers import AutoTokenizer, AutoConfig, AutoModelForImageTextToText, Gemma3ForCausalLM
from peft import PeftModel

BASE = "google/gemma-3-4b-it"                            # gated, accept the licence on the Hub first
ADAPTER = "manifesta/adaption_agronomy_calc_problems"

tok = AutoTokenizer.from_pretrained(ADAPTER)             # tokenizer ships in this repo
cfg = AutoConfig.from_pretrained(BASE)

vlm = AutoModelForImageTextToText.from_pretrained(
    BASE, dtype=torch.bfloat16, device_map="auto"
)

# the adapter targets the text tower, so rehost it as a causal LM
base = Gemma3ForCausalLM(cfg.text_config)
base.model = vlm.model.language_model
base.lm_head = vlm.lm_head

model = PeftModel.from_pretrained(base, ADAPTER).eval()

messages = [{"role": "user", "content":
    "A 24 ha maize field needs 150 kg N/ha. How much urea (46-0-0) do I need in total? "
    "If the question is missing something you need, say so and name it."}]

inputs = tok.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=384, do_sample=False)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Merging is optional and only touches the text tower:

model = model.merge_and_unload()

Confirm the adapter binds without downloading 8.6 GB of base weights

The failure this snippet exists to rule out is silent: attach a LoRA to the wrong module tree and PEFT injects adapters that no saved tensor ever lands in, so you get a model that loads cleanly and behaves exactly like the base. Build the architecture on the meta device and count the keys instead. No weights are downloaded, and it finishes in seconds.

import torch
from transformers import AutoConfig, AutoModelForImageTextToText, Gemma3ForCausalLM
from peft import PeftModel

cfg = AutoConfig.from_pretrained("google/gemma-3-4b-it")
with torch.device("meta"):
    vlm = AutoModelForImageTextToText.from_config(cfg)
    base = Gemma3ForCausalLM(cfg.text_config)
base.model, base.lm_head = vlm.model.language_model, vlm.lm_head

m = PeftModel.from_pretrained(base, "manifesta/adaption_agronomy_calc_problems",
                              low_cpu_mem_usage=True)
print(sum(1 for k, _ in m.named_parameters() if "lora_" in k))   # 476

476, with nothing missing and nothing extra, is the whole point: every tensor in adapter_model.safetensors binds to a real module.

What is actually in these weights

Read straight out of adapter_model.safetensors and adapter_config.json. Every row of this table is reproducible from the repo.

Property	Value
PEFT type	LoRA, `task_type: CAUSAL_LM`
Rank `r`	16
`lora_alpha`	32 (so the scaling factor alpha/r is exactly 2.0)
`lora_dropout`	0.0
`bias`	none
`use_rslora` / `use_dora`	false / false
Target modules	`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`
Tensors in the file	476
Decoder layers touched	34 of 34, all seven projections in every layer, no gaps
Trainable parameters	29,802,496 (float32, 119,273,568 bytes)
Share of the 3.88B text tower	roughly 0.77%
`exclude_modules`	empty, nothing was frozen out by name

Worth naming the contrast with the two sibling runs in this series: this one put LoRA on attention and the MLP, in every layer, at rank 16. The chart run used rank 8 on q_proj and v_proj only. This adapter had more trainable surface than either sibling and still went backwards, which is the reason the diagnosis below is about the optimisation and the objective rather than about frozen modules.

Evaluation

Judged by Adaption's AutoScientist evaluator on a held-out split the adapter never saw, against the same base model it was trained from. Head-to-head, blind.

Metric	Base	Adapted	Change
AutoScientist score	54	46	-8
Held-out eval loss (first eval, step 14)		1.2917
Held-out eval loss (final, step 66)		1.0919	-0.1998
Training loss	1.6436 (step 1)	0.9727 (step 66)	-0.6709

Eval loss across the five evaluations: 1.2917, 1.1639, 1.1239, 1.0999, 1.0919. Monotonic, textbook, and it bought a worse model.

46 against 54 is a regression. The adapter did not make Gemma 3 4B better at agronomy calculations. It made it worse in the judge's view, on a split it had never seen. There is no reading of that number that is a win, and separating it from the data-quality number below is the honest way to present the entry.

Why it went backwards

Three things, in order of how much I think they matter. All of them are checkable from the files in this repository.

1. The eval loss curve was measuring the wrong thing

Eval loss fell at every one of the five evaluations and total training loss fell from 1.644 to 0.973. That is a model getting steadily better at reproducing the corpus, and this corpus has a deliberately uniform shape: name the formula, substitute the numbers, carry the units, close with The final answer is $\boxed{ANSWER}$. Three epochs at 1e-4 is enough to fit that shape hard.

A judge comparing free-form answers does not reward the shape. It rewards the answer. So the loss curve and the head-to-head are measuring different things, and only one of them was ever the headline. If you take one thing from this card, take that: a clean monotonic eval curve is not evidence the model got better at the task.

2. The recipe change was a hypothesis, and it was wrong

I raised n_epochs from 1 to 3 and learning_rate from 1e-5 to 1e-4 by hand, because the two earlier runs in this series had looked starved of optimizer steps. Three epochs on a sub-10B base did buy more steps: 66, against 34 for the chart run and 59 for the math run.

It did not buy a better model. Three passes over a templated corpus at ten times the default learning rate is a good way to overfit a format, which is consistent with what section 1 describes. A single epoch at 1e-5 might have moved the model less and cost it less.

3. Five clipped steps, all of them at the start


Optimizer steps	66
`max_grad_norm`	2
Gradient norm range	0.0795 to 2.6548
Steps that exceeded the threshold	5 of 66 (7.6%)
Which steps	1, 2, 3, 4, 5
Their norms	2.6548, 2.1871, 2.0625, 2.1209, 2.0619
Learning rate at step 1 / step 8	0.0 / 1.0e-4

The learning rate warmed from 0 to its 1e-4 peak over eight steps. The gradient exceeded the clipping threshold before that warmup finished and never did again after step 5: from step 6 onward the norm stayed under 2 the whole way down to 0.0795 by step 60.

I am listing this third because it is the smallest of the three effects. Five clipped steps is a mild start, not a broken run. It is worth stating precisely rather than vaguely: this is what "it clipped" actually amounted to, and it does not carry the explanation on its own. Compare with the math and code adapter, where 29 of 59 steps were clipped and the peak norm was 677, which is a genuinely unstable run. This one was not unstable. It was stable and it fit the wrong objective.

What would need to change for a real result: one epoch at 1e-5, a held-out set scored the way the judge scores rather than by loss, and a training target that varies its surface form instead of the single boxed output contract this corpus enforces. The output contract is right for machine grading and it may be exactly what a preference judge penalises.

Training

Trained with Adaption AutoScientist, SFT with LoRA, on Adaptive Data output.

Hyperparameter	Value
Base model	`google/gemma-3-4b-it`, declared correctly in `adapter_config.json` since 2026-08-13. The original export value `togethercomputer/gemma-3-4b-it` is preserved in the same file as `original_base_model_name_or_path`
Method	SFT, LoRA (PEFT), chat format
LoRA rank / alpha / dropout	16 / 32 / 0.0
Trainable modules	`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`, all 34 layers
Learning rate	1e-4, raised by hand from the 1e-5 default
Epochs	3, raised by hand from the default 1
Peak learning rate reached	step 8
Max gradient norm	2
Optimizer steps	66
Per-device train batch size	1, with accumulation, effective batch about 1,276 rows per step
Evaluations	5, every 13 steps
Precision	bfloat16 base, float32 adapter weights
Total FLOPs	1.3396e18
Adaption dataset ID	`5b8890d0-96eb-433a-9f70-ae1d3dc1743f`
Training model ID	`adaption_gemma_3_4b_it_agronomy_calc_problems_587e58e3`

Sixty-six optimizer steps over three epochs is 22 steps per epoch, so with roughly 28,069 rows in the expanded training set each step saw about 1,276 rows. Even at three epochs the update budget stays small, because batch size scales with the corpus instead of the step count doing so.

trainer_state.json ships in this repository, and the copy in the GitHub repo is byte identical to it. Every number in this section comes out of that one file.

Intended use and limits

Use it for

Reproducing and inspecting this regression. That is the honest primary use.
Studying what three epochs at 1e-4 does to a small instruct model on a corpus with one rigid output contract.
A control arm against a one-epoch, 1e-5 run on the same data.
Research on refusal behaviour: whether a model that has seen 1,796 unanswerable questions names the missing input or invents a number.

Do not use it for

Anything that assumes it does agronomy arithmetic better than stock Gemma 3 4B. It does not, and the evaluation says so.
Real fertiliser, irrigation, seeding or pesticide decisions. A wrong rate is a real cost and a real environmental consequence, and no model output here has been reviewed by an agronomist. Use a current soil test, the registered product label and local regulation.
Anything safety, regulatory or financial where a confidently wrong number does damage.
Any use that violates the Gemma Terms of Use or the Gemma Prohibited Use Policy, which apply through the base model.

Technical limits

The adapter targets the text tower only. If you attach it to the multimodal wrapper without rehosting the language model, it will bind nothing and behave exactly like the base. See Quickstart.
Trained on 100% generated data. Verified arithmetic, real citations, no field records and no real farmer language.
Constants are largely North American and FAO conventions: 56 lb/bu corn, 15.5% market moisture, Fahrenheit degree-day bases alongside metric rates.
English only.
The declared base string was wrong until 2026-08-13. It resolves now, and the original is recorded in the config.
Adapter weights are float32 while the base is bfloat16. PEFT casts on load, so expect the adapter's memory footprint to be double a bf16 export.
One judge, one held-out split. Reported as-is, no reruns and no best-of selection.

Dataset

Trained on Adaption dataset 5b8890d0-96eb-433a-9f70-ae1d3dc1743f, built from manifesta/verified-agronomy-17k (also on Kaggle).

17,199 rows generated, 17,028 ingested by Adaption, about 28,069 in the training set after the platform's domain and diversity expansion
Ten calculation families: fertiliser rates and NPK blends, growing degree days, grain moisture and test weight, seeding rate, irrigation ETc, sprayer calibration, forage budgeting, farm economics, unit conversion, plus a refusal slice
1,796 rows (10.4%) are deliberately unanswerable. The correct response names the missing input rather than estimating it, and those rows carry real but irrelevant detail (field size, previous crop, growth stage) on purpose, so plausible context cannot substitute for the required input
15,403 rows (89.6%) are exactly gradable against a known numeric answer
Every formula is cited to a published source: FAO Irrigation and Drainage Paper 56, NDSU NDAWN, Cornell CSS412, WSU extension, Purdue, Ohio State, Pioneer
Verified twice: 26 calculators checked against a worked example published by the source each one cites, and 25,324 arithmetic expressions inside the finished solutions re-evaluated with 0 inconsistent
CC0-1.0. Every row was generated for this dataset, no third-party content, no scraping

Adaptive Data lifted the corpus quality before training:

Adaptive Data metric	Before	After
Quality score	5.0	8.4 (+68.0% relative)
Grade	C	B
Percentile	7.2	31.5

+68.0% is the improvement in the data, not in the model. The data got substantially better and the model got worse. Those are two different measurements and only one of them is this card's headline. It is the largest data lift of the three entries in this series and it is still not the number that matters here.

Live interface

MANIFESTA Agriculture runs at manifestaagriculture.adaptionlabs.app. Ask it for a fertiliser rate without giving it a soil test and see whether it names the missing input or invents one.

Verify all of this yourself

Every claim on this card is checked by a script that anyone can run, in github.com/A1VARA5/verified-agronomy-17k:

git clone https://github.com/A1VARA5/verified-agronomy-17k
cd verified-agronomy-17k
python verify.py

22 checks, standard library only, no install step. The ones that cover this model:

	Check
16	the weights download, HTTP 200, and are exactly 119,273,568 bytes
17	`google/gemma-3-4b-it` resolves HTTP 200, and `togethercomputer/gemma-3-4b-it` still returns 401
18	`adapter_config.json` declares the corrected base and preserves the original verbatim, r=16, alpha=32, 7 target modules
19	the published `trainer_state.json` is byte identical to the copy in the repo, 66 steps, peak gradient norm 2.6548, 5 clipped and all in the first five steps
21	the Kaggle model mirror answers HTTP 200 to a logged out fetch
22	the live interface answers HTTP 200

The same script runs in GitHub Actions on every push and once a day on a schedule, which is what the badge at the top of this card reports.

Related models

manifesta/adaption_scientific_chart_qa_17k, chart QA on Gemma 3 27B. 50 against 50, a null result. Its vision tower was frozen, so the part of the model that reads a figure was never adapted.
manifesta/adaption_verified_math_code_instruct, verified maths and code on Gemma 4 31B. 46 against 54, with 29 of 59 steps clipped and a peak gradient norm of 677.
manifesta/scientific-chart-qa-lora-qwen3.5-9b, the same chart task on a 9B base and a smaller corpus. 51 against 49.

Four runs across the two challenge parts, and the pattern in them:

Run	Base	Rows ingested	Epochs	LR	Steps	Clipped	Score vs base
Chart QA, first build	Qwen3.5 9B	6,976	1	1e-4	21	not recorded	51 vs 49
Chart QA 17k	Gemma 3 27B	17,070	1	1e-4	34	0 of 34	50 vs 50
Verified math and code	Gemma 4 31B	17,586	1	1e-4	59	29 of 59	46 vs 54
Agronomy (this model)	Gemma 3 4B	17,028	3	1e-4	66	5 of 66	46 vs 54

More steps did not help. This run has the most optimizer steps of the four and the joint worst score. Caveat, stated plainly: four data points across four base models, four corpora and four domains is not a controlled experiment and should not be read as one.

License

Adapter weights: released under the Gemma Terms of Use. Derivatives of Gemma inherit Gemma's terms, so the adapter travels with the same conditions as the base.
Base model: google/gemma-3-4b-it is gated. Accept the licence and request access on the Hub before loading. The Gemma Prohibited Use Policy applies.
Training dataset: manifesta/verified-agronomy-17k, released CC0-1.0. Every row was generated for that dataset, so there is nothing to attribute and no restriction to pass on.

Citation

@misc{agronomy_calc_lora_2026,
  title  = {An agronomy calculation adapter for Gemma 3 4B, and why three epochs at 1e-4 made it worse},
  author = {Navardauskas, Aivaras},
  year   = {2026},
  note   = {MANIFESTA. Adapted with Adaptive Data by Adaption, AutoScientist Challenge, Agriculture.
            AutoScientist score 46 against a base of 54. Recipe hand-edited to 3 epochs at 1e-4.},
  howpublished = {\url{https://huggingface.co/manifesta/adaption_agronomy_calc_problems}}
}

Built with Adaptive Data by Adaption. Platform documentation at docs.adaptionlabs.ai. Dataset, adapter and analysis by MANIFESTA (Aivaras Navardauskas) for the AutoScientist Challenge.

manifesta
/

adaption_agronomy_calc_problems