README.md

---
license: cc-by-nc-4.0
base_model: Qwen/Qwen3.6-35B-A3B
base_model_relation: finetune
library_name: transformers
pipeline_tag: image-text-to-text
tags:
  - openmle
  - frontis-ma1
  - machine-learning-engineering
  - autoresearch
  - agent
  - coding
  - qwen3
  - qwen3.6
  - moe
  - multimodal
  - post-training
---

<p align="center">
  <img src="assets/frontis-logo.png" width="360" alt="Frontis AI">
</p>

<h1 align="center">Frontis-MA1-35B</h1>

<p align="center">
  <b>An execution-grounded model for evolutionary machine learning engineering and AutoResearch</b>
</p>

<p align="center">
  <a href="https://arxiv.org/abs/2607.28568">📄 Paper</a>
  &nbsp;•&nbsp;
  <a href="https://frontisai.github.io/OpenRSI/">🌐 Project</a>
  &nbsp;•&nbsp;
  <a href="https://github.com/FrontisAI/OpenRSI">💻 Code</a>
  &nbsp;•&nbsp;
  <a href="https://huggingface.co/collections/FrontisAI/frontis-ma1">🤗 Models</a>
  &nbsp;•&nbsp;
  <a href="https://huggingface.co/datasets/FrontisAI/OpenMLE-Tasks">🧩 Tasks</a>
  &nbsp;•&nbsp;
  <a href="https://huggingface.co/datasets/FrontisAI/OpenMLE-SFT-Traces">📚 SFT Traces</a>
</p>

## Introduction

**Frontis-MA1-35B** is the flagship open-weight research model released with **OpenMLE**, an execution-grounded system for studying meta-evolution in machine learning engineering (MLE).

Starting from [Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B), we apply execution-grounded supervised fine-tuning and reinforcement learning so that the model learns four reusable program-transformation operators:

- **Draft** — create a complete initial solution;
- **Improve** — refine a parent program using its score and execution evidence;
- **Debug** — repair invalid or failing code;
- **Crossover** — recombine useful elements from two parent solutions.

At inference time, OpenMLE-Evo composes these operators into a long-horizon search loop. Frontis-MA1 is therefore both a product of the OpenMLE training stack and the model that drives its evolutionary search.

> **Component composition.** The Frontis-MA1 checkpoint contains the
> post-trained language-model weights used by OpenMLE. For a complete,
> base-compatible release, this repository also includes the unchanged vision
> encoder and MTP components from Qwen3.6-35B-A3B. OpenMLE post-training and the
> reported evaluations are text/code-only; they do not establish improved or
> fully validated visual capability or MTP decoding quality.

## Highlights

- **Execution-grounded post-training.** SFT examples and RL rewards are constructed from programs executed in isolated MLE sandboxes.
- **Trainable evolutionary operators.** Draft, Improve, Debug, and Crossover provide a shared action space across training and test-time search.
- **Linked research release.** The model card connects the weights to the OpenMLE training code, search protocol, task environments, and evaluation boundary.
- **Strong controlled model gain.** Under the same OpenMLE-Evo harness on MLE-Bench Lite, Frontis-MA1-35B improves Medal Average from **39.39% to 60.61%** and Human Rank from **0.5828 to 0.7647** over its base model.
- **Composable search gain.** With OpenMLE-Evo-Max, the model–harness system reaches **71.21% Medal Average** and **0.8126 Human Rank** under the paper's fixed evaluation protocol.

> The reported scores measure **model–harness systems**, not standalone one-shot generation. Reproduction requires the same OpenMLE-Evo or OpenMLE-Evo-Max configuration, task environments, sandbox budget, and evaluation protocol.

## Model family

| Model | Base model | Release format | Status |
| --- | --- | --- | --- |
| [Frontis-MA1-30B](https://huggingface.co/FrontisAI/Frontis-MA1-30B) | Qwen3-30B-A3B-Thinking-2507 | BF16 Transformers | Companion release |
| **Frontis-MA1-35B** | Qwen3.6-35B-A3B | BF16 Transformers | This repository |
| [Frontis-MA1-35B-GGUF](https://huggingface.co/FrontisAI/Frontis-MA1-35B-GGUF) | Frontis-MA1-35B | Q4_K_M GGUF + F16 mmproj | Local deployment |

## Model details

| Field | Value |
| --- | --- |
| Model family | Frontis-MA1 |
| Architecture | Multimodal conditional-generation model with a hybrid-attention MoE text backbone |
| Parameters | 35B total, approximately 3B activated |
| Layers | 40 |
| Experts | 256 routed experts; 8 routed plus 1 shared expert activated per token |
| Weight precision | BF16 |
| Native configuration context | 262,144 tokens |
| Post-training SFT cutoff | 32,768 tokens |
| Primary evaluated modality | Text and code |
| Vision encoder | Included; inherited unchanged from Qwen3.6-35B-A3B |
| MTP weights | Included; inherited unchanged from Qwen3.6-35B-A3B |
| License | CC BY-NC 4.0 |

The 262,144-token context length is inherited from the base-model configuration. OpenMLE post-training used a 32,768-token SFT cutoff; substantially longer contexts have not yet been validated to the same standard as the reported experiments.

## Training

Frontis-MA1-35B uses full-parameter supervised fine-tuning followed by reinforcement learning from executable task feedback.

### Supervised fine-tuning

- 26,259 released execution-grounded examples: 17,245 full responses and 9,014 trajectory steps;
- full-solution and trajectory-step supervision;
- BF16 full-parameter training on 8 NVIDIA H200 GPUs;
- global batch size 128;
- learning rate `3e-5` with cosine decay and 0.1 warmup fraction;
- three epochs;
- Qwen-compatible thinking-loss masking.

### Reinforcement learning

- online generation and evaluation in isolated task sandboxes;
- operator mixture: Draft 0.50, Improve 0.17, Debug 0.17, Crossover 0.16;
- 16 prompts per rollout and 16 samples per prompt;
- maximum response length 24,576 tokens;
- GSPO with execution-grounded reward post-processing;
- Adam optimizer with learning rate `1e-6`.

See the paper and [OpenRSI code](https://github.com/FrontisAI/OpenRSI) for the complete data-construction, training, search, and evaluation protocol.

<p align="center">
  <img src="assets/paper-figure-openmle-framework.png" width="95%" alt="Paper framework figure: OpenMLE training and inference workflow">
</p>

<p align="center"><sub>
Paper framework figure. Frontis-MA1 learns the same four atomic operators used by OpenMLE-Evo, first from executable SFT rollouts and then from online RL execution feedback.
</sub></p>

## Benchmark results

The headline results are evaluated as **model–harness systems**, rather than as standalone one-shot generations. The controlled comparison that isolates the effect of post-training holds the OpenMLE-Evo harness fixed and changes only the model from Qwen3.6-35B-A3B to Frontis-MA1-35B.

### MLE-Bench Lite protocol

| Item | Setting |
| --- | --- |
| Benchmark | Official 22-task MLE-Bench Lite split |
| Independent runs | 3 per OpenMLE-Evo configuration, unless stated otherwise |
| Sandbox budget | 12 hours on 0.5 NVIDIA RTX 4090 equivalent per task |
| Equivalent compute | 6 RTX 4090 GPU-hours per task |
| Execution | Generated programs are run and scored in isolated task sandboxes |
| OpenMLE-Evo-Max | MLE-Bench-disjoint task-agnostic priors plus asynchronous multi-GPU search |
| Budget comparison | OpenMLE-Evo-Max keeps the same total sandbox-compute budget |

We report three aggregate metrics:

- **Valid Rate** is the mean number of the 22 tasks for which a run produces a valid submission, written as `x/22`.
- **Medal Average** is the mean fraction of tasks receiving any Kaggle medal.
- **Human Rank** is the fraction of human leaderboard participants surpassed by the submitted solution, averaged across tasks and runs.

All three metrics are higher-is-better.

### Main results from the paper

<p align="center">
  <img src="assets/paper-figure-main-results.png" width="98%" alt="Paper main figure: model-harness results on MLE-Bench Lite">
</p>

<p align="center"><sub>
Current paper main-results figure. The left panel shows all completed model–harness results on MLE-Bench Lite; the right panel compares model size with Medal Average. Orange denotes Frontis-MA1 with OpenMLE-Evo, cyan denotes other models with OpenMLE-Evo, hatching denotes OpenMLE-Evo-Max, and gray denotes general-purpose coding harnesses.
</sub></p>

### Controlled 35B comparison

| Model | Harness | Valid Rate ↑ | Medal Average ↑ | Human Rank ↑ |
| --- | --- | ---: | ---: | ---: |
| Qwen3.6-35B-A3B | OpenMLE-Evo | 19.67/22 | 39.39% | 0.5828 |
| **Frontis-MA1-35B** | OpenMLE-Evo | **21.67/22** | **60.61%** | **0.7647** |
| **Frontis-MA1-35B** | OpenMLE-Evo-Max | **22.00/22** | **71.21%** | **0.8126** |

| Comparison | Δ Valid Rate | Δ Medal Average | Δ Human Rank | What it measures |
| --- | ---: | ---: | ---: | --- |
| Frontis-MA1-35B vs. Qwen3.6-35B-A3B, both with OpenMLE-Evo | +2.00 tasks | +21.22 pp | +0.1819 | Post-training gain under a fixed harness |
| OpenMLE-Evo-Max vs. OpenMLE-Evo, both with Frontis-MA1-35B | +0.33 task | +10.60 pp | +0.0479 | Additional system-level search gain |

The first comparison above is the primary evidence for the model improvement: the base and post-trained checkpoints are evaluated under the same standard OpenMLE-Evo harness. The second comparison holds the model fixed but changes the search system, so it should not be interpreted as a pure model gain.

### Public-weight model comparison under OpenMLE-Evo

The following rows use the same OpenMLE-Evo harness, benchmark split, and sandbox-compute budget. This is the closest available comparison with strong public-weight models, although model scale, pretraining data, and architecture still differ.

| Public-weight model | Valid Rate ↑ | Medal Average ↑ | Human Rank ↑ |
| --- | ---: | ---: | ---: |
| [Kimi K2.6](https://huggingface.co/moonshotai/Kimi-K2.6) | 21.67/22 | **66.67%** | 0.7859 |
| [GLM-5.2](https://huggingface.co/zai-org/GLM-5.2) | 19.67/22 | 62.12% | 0.7069 |
| **Frontis-MA1-35B** | 21.67/22 | 60.61% | 0.7647 |
| [MiniMax M3](https://huggingface.co/MiniMaxAI/MiniMax-M3) | **22.00/22** | 59.09% | **0.7994** |
| [Frontis-MA1-30B](https://huggingface.co/FrontisAI/Frontis-MA1-30B) | 21.67/22 | 53.03% | 0.7055 |
| [DeepSeek-V4-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) | 21.33/22 | 51.52% | 0.6957 |
| [Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) | 19.67/22 | 39.39% | 0.5828 |
| [Qwen3-30B-A3B-Thinking-2507](https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507) | 17.33/22 | 34.85% | 0.5573 |

These are model–harness results rather than standalone one-shot model scores. The full paper figure above additionally includes OpenMLE-Evo-Max and general-purpose coding-agent systems, whose harnesses are not directly interchangeable with this fixed-harness table.

### Transfer to NatureBench Lite

We additionally evaluate transfer beyond competition-style MLE on a fixed 10-task NatureBench Lite subset. The subset spans all six NatureBench scientific domains, six represented input-modality families, and four ML task types. The evaluation retains the original task containers, hidden evaluator, validity rules, web-search-disabled setting, and a four-hour search budget per task.

NatureBench direction-normalizes each task's metric relative to the published result. We report:

- **Match-SOTA (All M):** the fraction of tasks with normalized gap `g ≥ 0`;
- **Surpass-SOTA (All S):** the fraction of tasks with `g > 0.1`.

| Model | Harness | Surpass-SOTA ↑ | Match-SOTA ↑ |
| --- | --- | ---: | ---: |
| **Frontis-MA1-35B** | OpenMLE-Evo NatureBench adapter | **30.0% (3/10)** | **70.0% (7/10)** |
| Qwen3.6-35B-A3B | OpenMLE-Evo NatureBench adapter | 20.0% (2/10) | 50.0% (5/10) |
| Qwen3.6-35B-A3B | Original AIRA-Evo | 10.0% (1/10) | 20.0% (2/10) |

Holding the NatureBench adapter fixed, Frontis-MA1-35B improves over its base model by 10 percentage points on Surpass-SOTA and 20 points on Match-SOTA. Holding the base model fixed, the adapted OpenMLE-Evo harness improves over original AIRA-Evo by 10 and 30 points, respectively. Because this study contains only ten tasks, it is evidence of focused transfer rather than a claim of general scientific autonomy or performance on the full 90-task benchmark.

### Long-horizon search behavior

Two detailed MLE-Bench trajectories in the paper examine whether the system continues improving after it has already found an executable program.

| Task | Validation Human Rank | Held-out Human Rank | Medal | Share of validation gain from late Improve/Crossover |
| --- | ---: | ---: | --- | ---: |
| `leaf-classification` | 0.7713 | 0.9455 | Bronze | 85.0% |
| `mlsp-2013-birds` | 0.7284 | 0.8889 | Silver | 91.9% |

These task-level traces support the intended use of Frontis-MA1-35B inside a long-horizon evolutionary loop: Debug establishes executable solutions, while later Improve and Crossover operations account for most of the measured validation improvement in the two analyzed cases. They are mechanism case studies, not additional aggregate benchmark scores.

## Usage

Qwen3.6 uses the newer `qwen3_5_moe` architecture. Install a recent Transformers build that includes this architecture; older releases such as Transformers 4.57.1 do not recognize the configuration. The upstream Qwen3.6 model currently recommends installing the latest `transformers[serving]`.

### vLLM

The paper's evaluated text/code path should be served in language-model-only mode:

```bash
vllm serve FrontisAI/Frontis-MA1-35B \
  --served-model-name Frontis-MA1-35B \
  --tensor-parallel-size 8 \
  --max-model-len 32768 \
  --reasoning-parser qwen3 \
  --language-model-only
```

## Intended use

Frontis-MA1-35B is intended for:

- research on machine learning engineering and AutoResearch agents;
- generation and iterative improvement of ML experiment code;
- execution-grounded program search;
- use with OpenMLE-Evo or compatible sandboxed harnesses.

Generated code may be incorrect, insecure, destructive, or expensive to execute. Run it inside an isolated environment with explicit CPU, memory, GPU, network, filesystem, and time limits.

## Limitations

- OpenMLE post-training and the reported evaluations cover text/code behavior; the inherited vision encoder and MTP components were not post-trained or evaluated by this work.
- The main evaluation centers on MLE-Bench Lite and does not establish general performance across all software-engineering or scientific-research tasks.
- The paper results depend on an external evolutionary search harness and sandbox feedback.
- Execution reward does not fully measure maintainability, robustness, security, or responsible data handling.
- Context lengths beyond the post-training cutoff require additional validation.
- Outputs may contain reasoning traces and should be handled according to downstream requirements.

## Release artifacts

This repository is the canonical BF16 Transformers release. The [Frontis-MA1-35B-GGUF](https://huggingface.co/FrontisAI/Frontis-MA1-35B-GGUF) repository provides the Q4_K_M local-deployment derivative and its required F16 multimodal projector.

<a id="paper"></a>
## Paper and citation

**Frontis-MA1: Training an AI4AI Model towards Recursive Self-Improvement in Machine Learning Engineering**

Preprint: [arXiv:2607.28568](https://arxiv.org/abs/2607.28568).

## License

Original Frontis-MA1 material is released under [CC BY-NC 4.0](LICENSE) for attribution-required, non-commercial use. Commercial use is not granted.

The upstream Qwen Apache License 2.0 notice is preserved in [LICENSE-UPSTREAM-APACHE-2.0](LICENSE-UPSTREAM-APACHE-2.0) and [NOTICE](NOTICE).

The license in this repository applies to the model weights, configuration files, and model documentation distributed here. Training and evaluation datasets and third-party software remain subject to their respective terms.

## Acknowledgements

We thank the Qwen team and the open-source communities behind Transformers, SLIME, Ray, Megatron-LM, SGLang, MLE-Bench, and the broader executable MLE research ecosystem.