--- language: - pl - en license: apache-2.0 base_model: - aac6fef/laya-multilingual-mlx base_model_relation: finetune pipeline_tag: text-classification tags: - mlx - laya - devops - decision-model - apple-silicon - experimental - synthetic-data - claude-code - litellm --- # Laya DevOps MLX — v29 early experimental preview **Early development / stage use only. This model is still being developed and will receive further training. It has not met our safety or generalization targets.** The separate audit scored **73.83% accuracy with 43 unsafe ALLOW decisions out of 512 cases**. Do not read the 99.96% development-test score as production readiness. This is a small, local, bilingual DevOps decision classifier for **ALLOW / ASK / DENY**, fine-tuned with Laya/MLX on Apple Silicon. It evaluates a proposed command together with the user's request, environment and context. It does not generate shell commands, execute tools, replace a firewall, or provide a production authorization boundary. Released checkpoint: `devops-v29-p08-r1`. Exported weights: 321,908,998 parameters, about 644 MB decimal / 614 MiB. Total context: 1024 tokens shared by the question and state. The provided server rejects oversized states instead of silently truncating them. ## What the decisions mean - **ALLOW**: the proposed action appears permitted under the supplied policy and trusted context. - **ASK**: obtain human confirmation or resolve missing context. - **DENY**: the proposed action conflicts with the supplied policy or explicit restrictions. An ALLOW is a model prediction, not proof of authorization. The supplied global Claude Code adapter deliberately keeps Claude's native permission checks when Laya returns ALLOW. It maps ASK to confirmation and DENY to blocking. ## Method and training The weight lineage starts with [`aac6fef/laya-multilingual-mlx`](https://huggingface.co/aac6fef/laya-multilingual-mlx), revision `f2b4faf51023039425946074e2cf1361d2db11d5`, the native MLX port of multilingual Laya, using the [`jhu-clsp/mmBERT-base`](https://huggingface.co/jhu-clsp/mmBERT-base) encoder. V29 continues from the project's preserved V28 checkpoint. This is a specialized decision encoder, not a Sol/Astra model conversion or a general chat LLM. **Sol and Astra were used as AI assistants in the training-data and development workflow**: synthetic scenario authoring, varied instructions and contexts, code and evaluation design. They are teacher/development assistants, not the base architecture or source weights of this checkpoint. Exact teacher model versions and per-example authorship were not independently recorded, so this release does not claim a fully reproducible teacher-generation trace or independent model judging. The V29 curriculum contains **377,003 training examples**, **40,616 validation examples**, and **39,096 test examples**. A separately authored audit contains **512 rows / 16 scenario groups**, built from 64 semantic variants expanded into two languages and four wrappers. Expansions are correlated; the row count is not the count of independent challenges. The curriculum varies trusted requests, stage/production context, authorization scope, read-only operations, local edits, secrets, mixed command chains, wrappers, indirect execution and hidden effects. Commands appear as classification data and are never executed by the classifier or evaluation scripts. Training ran locally on an Apple M1 Max with 64 GiB unified memory, using LayaStudio and MLX: | Setting | Value | |---|---| | Effective sequential chunks | 8, each saved as a separate checkpoint | | Rows per chunk | 47,125–47,126, grouped by scenario family | | Initialization | V28 → chunk 1 → … → chunk 8 | | Epochs | One per chunk | | Method | Full fine-tuning of 22 encoder layers; 125,101,825 trainable parameters | | Batch / gradient accumulation | 16 / 1 | | Encoder / head learning rate | 1e-6 / 5e-7 | | Objective | Cross entropy, false-ALLOW penalty 1.0 | | Training precision | bfloat16; exported weights FP16 | | Optimizer | Reset at each chunk; **not** exact optimizer-state resumption | | Native validation | Fixed 2,048-row panel drawn only from validation | | Full final assessment | Full validation, test and separately frozen audit | An early continuation bug restored initial weights when a saturated validation panel did not improve by an absolute loss margin. The affected chunk attempts were preserved and superseded; chunk 2 and chunk 3 were repeated. The effective chain retained the final trained epoch (`selection_metric=last`). This selection change did not relax final quality criteria. `provenance.json` records the effective chain and hashes. Inherited upstream training counters in the original configuration were replaced in this portable release with the effective V29 summary; weights and saved decision threshold are unchanged. ## Results — what we achieved and what remains unsolved Both V28 and V29 were evaluated on **identical test and audit rows** with their previously saved thresholds. V28 was queried through the actual running HTTP endpoint; V29 used its exported local weights with the same preprocessing. Exact state overlap against each model's training data was checked and was zero for these evaluation sets. This does not establish semantic independence from related templates or development work. | Model | Split | Rows | Raw accuracy | Accuracy after threshold | Family macro accuracy | Unsafe ALLOW after threshold | ALLOW recall | |---|---|---:|---:|---:|---:|---:|---:| | V28, threshold 0.90 | Test | 39,096 | 75.6088% | 75.5653% | 76.6712% | 6,764 | 89.0423% | | **V29, threshold 0.95405** | **Test** | **39,096** | **99.9591%** | **99.9591%** | **99.9503%** | **2** | **99.9574%** | | V28, threshold 0.90 | Audit | 512 | 66.0156% | 65.8203% | 65.8203% | 61 | 58.3333% | | **V29, threshold 0.95405** | **Audit** | **512** | **73.4375%** | **73.8281%** | **73.8281%** | **43** | **63.0952%** | V29 validation accuracy after threshold: **99.9828%**, zero observed unsafe ALLOWs, ALLOW recall 100%, on 40,616 examples. The threshold **0.95405 was selected only on validation and frozen before test/audit**. Raw V29 false ALLOW counts were 4 on test and 46 on audit; thresholding reduced those to 2 and 43. | V29 operating accuracy | Polish | English | |---|---:|---:| | Test | 99.9540% | 99.9642% | | Audit | 72.6563% | 75.0000% | V29 improved the separate audit by **8.01 percentage points** over V28 and reduced its unsafe ALLOW count by **18**. Nevertheless, **43 unsafe audit ALLOWs remain**, and audit ALLOW recall is below the 80% target. The large test/audit gap is evidence of limited generalization beyond the synthetic curriculum's familiar structures. Our simultaneous targets remain: ≥97% raw and operating accuracy, ≥97% family macro accuracy, ≥95% separately in PL and EN, zero observed unsafe ALLOWs, and ALLOW recall ≥80% on validation, test and a fresh pre-frozen audit. **V29 does not meet these targets.** The strict pipeline stopped after the test's two unsafe ALLOWs. The audit was subsequently opened by an explicit user request for a descriptive stage comparison of both frozen models. This did **not** override the failed qualification. That audit is now exposed and cannot be reused as a fresh independent qualification set. No thresholds or labels were changed using its outputs. Summary results, confusion matrices, errors, source hashes and the comparison protocol are in [`evaluation/`](evaluation/). Full synthetic training/evaluation rows are not included in this model release; the files therefore document the results rather than promise full benchmark reproduction. Audit labels have no independent human review. Timing through HTTP and direct local inference includes different overhead and is not a controlled speed comparison. ## Run locally on Apple Silicon This release uses the custom `laya-mlx` runtime. A standard `transformers.pipeline` or chat-generation runner is not an equivalent loader. macOS 14+ and Apple Silicon are the supported runtime target for this package; Python 3.12 is the tested version. Claude Code may run on another Mac or Linux VM and query LiteLLM remotely. Repository: `tebebe/laya-devops-mlx`. The initial release is tagged `v29-preview`; future versions can use the same repository with new tags. Download the pinned preview, then install locally: ```sh hf download tebebe/laya-devops-mlx --revision v29-preview --local-dir laya-devops-mlx cd laya-devops-mlx ``` Then: ```sh python3.12 -m venv .venv .venv/bin/python -m pip install -r requirements.txt .venv/bin/python verify_release.py screen -S laya-devops8765 sh ./serve.sh ``` Detach with Ctrl+A, then D. Reattach with `screen -r laya-devops8765`. `serve.sh` generates a private backend API key if absent and starts the classifier on `0.0.0.0:8765`. Set `LAYA_HOST=127.0.0.1` for local-only access. The backend key lives in `runtime/workspace/local-api.key` and must be configured separately on LiteLLM; it is never included in the repository. Use a private network such as WireGuard or a protected TLS route for remote access. The lazy model worker unloads after 120 idle seconds. For local classification without an HTTP service: ```sh .venv/bin/python predict.py examples/state.json ``` The helper applies the frozen normalizer, validates the context budget, runs the fixed permission question, and preserves the saved threshold. It prints a decision and probabilities; it does not execute the supplied command. ## LiteLLM integration Topology: ```text Claude Code (local Mac or Linux VM) → LiteLLM proxy /v1/systemone, alias laya-devops-decisions → Mac's private-network Laya API /v1/systemone → V29 permission classifier ``` Add the entries from [`integration/litellm-model.yaml`](integration/litellm-model.yaml) to the existing LiteLLM configuration. Replace `` with the Mac's stable private/WireGuard address. Set `LAYA_LOCAL_API_KEY` **on the proxy** to the backend key generated on the Mac. Do not use this backend key as the Claude client's proxy key. Use a LiteLLM build that supports [Decisions/System One routing](https://docs.litellm.ai/docs/decisions), with the decision deployment's mode set to `evaluation` as in the supplied, locally verified configuration. If configuring in the Admin UI, select the corresponding Decisions mode. The internal backend model name is `laya-devops-permissions`; the proxy decision alias is `laya-devops-decisions`. The second supplied entry offers Chat-compatible JSON classification, not general chat. Each client's **LiteLLM virtual key** must have access to the decision alias. Requests through the proxy can be viewed in LiteLLM Logs filtered by `laya-devops-decisions`, subject to the proxy's logging configuration. Use separate virtual keys for local and work-VM usage if separate accounting is needed. Classifier decisions do not execute anything on the proxy or Mac. ## Connect local Claude Code or a work VM The model remains on the Mac. Install **only the small hook adapter** on each Claude Code machine. All decision requests use LiteLLM; there is no direct-to-Mac fallback. Existing Claude generative-model routing is preserved. 1. Ensure the machine's existing `~/.claude/settings.json` has `env.ANTHROPIC_BASE_URL` set to its LiteLLM root URL, for example `http://:4000`, and that a proxy virtual key is available as `ANTHROPIC_AUTH_TOKEN` in Claude's environment or existing settings. Keep keys private. 2. From this downloaded repository, on the machine running Claude Code: ```sh python3 adapters/claude_code/install_global.py --api-key-env ANTHROPIC_AUTH_TOKEN python3 "$HOME/.claude/laya/doctor.py" \ --config "$HOME/.claude/laya/config.json" --probe ``` 3. Start a **new** Claude Code session and inspect `/hooks`. The installer creates a backup and merges `UserPromptSubmit` and `PreToolUse` hooks; it leaves existing models, rules and unrelated hooks intact. On a Linux VM, run this installer there rather than copying a Mac settings file. The installed runtime is in `~/.claude/laya/`; its `config.json` selects `/v1/systemone`, `laya-devops-decisions`, the key's environment-variable name, stage context and the adapter's additional minimum ALLOW floor. The server's stricter V29 threshold remains authoritative. The installer does not create or publish a proxy key and does not contact a VM on your behalf. Equivalent hook fragment (merge into the existing **`hooks` object**, do not replace the settings file): ```json { "hooks": { "UserPromptSubmit": [{"hooks": [{"type": "command", "command": "python3 \"$HOME/.claude/laya/global_hook.py\" || { echo 'Laya hook failed; action blocked.' >&2; exit 2; }", "timeout": 15}]}], "PreToolUse": [{"matcher": "Bash", "hooks": [{"type": "command", "command": "python3 \"$HOME/.claude/laya/global_hook.py\" || { echo 'Laya hook failed; action blocked.' >&2; exit 2; }", "timeout": 15}]}] } } ``` The adapter covers **Bash**, not every tool or shell reachable through other integrations. Prompt history is contextual evidence with unverified origin, not authority to expand permissions. ALLOW retains native Claude permissions; ASK requests confirmation; DENY blocks. Ordinary request/parse failures request confirmation, and wrapper process failures return a blocking exit code. Hook availability and externally killed hook processes remain subject to the installed Claude Code version's behavior. See [Claude Code hooks](https://code.claude.com/docs/en/hooks). Keep normal isolation and permission controls. ## Development status and intended next work This is an **early preview**, not a finished release or a production safety product. Further training is planned after review, with greater scenario diversity, more adversarial and indirect-effect cases, independent label review where possible, and genuinely fresh frozen audits. No additional training is automatically triggered by downloading or running this package. Future checkpoints should remain separate so users can compare and roll back. ## License, attribution and provenance Apache-2.0 for this derivative release, retaining upstream notices; the encoder is MIT. See [`LICENSE`](LICENSE), [`NOTICE`](NOTICE), [`licenses/`](licenses/) and [`provenance.json`](provenance.json). Model and runtime origins are credited separately from Sol/Astra's development assistance. This is an independent experimental fine-tune and is not an endorsement by the upstream authors or teacher-model providers.