tebebe/laya-devops-mlx

Laya DevOps MLX — v29 early experimental preview

Early development / stage use only. This model is still being developed and will receive further training. It has not met our safety or generalization targets. The separate audit scored 73.83% accuracy with 43 unsafe ALLOW decisions out of 512 cases. Do not read the 99.96% development-test score as production readiness.

This is a small, local, bilingual DevOps decision classifier for ALLOW / ASK / DENY, fine-tuned with Laya/MLX on Apple Silicon. It evaluates a proposed command together with the user's request, environment and context. It does not generate shell commands, execute tools, replace a firewall, or provide a production authorization boundary.

Released checkpoint: devops-v29-p08-r1. Exported weights: 321,908,998 parameters, about 644 MB decimal / 614 MiB. Total context: 1024 tokens shared by the question and state. The provided server rejects oversized states instead of silently truncating them.

What the decisions mean

• ALLOW: the proposed action appears permitted under the supplied policy and trusted context.
• ASK: obtain human confirmation or resolve missing context.
• DENY: the proposed action conflicts with the supplied policy or explicit restrictions.

An ALLOW is a model prediction, not proof of authorization. The supplied global Claude Code adapter deliberately keeps Claude's native permission checks when Laya returns ALLOW. It maps ASK to confirmation and DENY to blocking.

Method and training

The weight lineage starts with aac6fef/laya-multilingual-mlx, revision f2b4faf51023039425946074e2cf1361d2db11d5, the native MLX port of multilingual Laya, using the jhu-clsp/mmBERT-base encoder. V29 continues from the project's preserved V28 checkpoint. This is a specialized decision encoder, not a Sol/Astra model conversion or a general chat LLM.

Sol and Astra were used as AI assistants in the training-data and development workflow: synthetic scenario authoring, varied instructions and contexts, code and evaluation design. They are teacher/development assistants, not the base architecture or source weights of this checkpoint. Exact teacher model versions and per-example authorship were not independently recorded, so this release does not claim a fully reproducible teacher-generation trace or independent model judging.

The V29 curriculum contains 377,003 training examples, 40,616 validation examples, and 39,096 test examples. A separately authored audit contains 512 rows / 16 scenario groups, built from 64 semantic variants expanded into two languages and four wrappers. Expansions are correlated; the row count is not the count of independent challenges.

The curriculum varies trusted requests, stage/production context, authorization scope, read-only operations, local edits, secrets, mixed command chains, wrappers, indirect execution and hidden effects. Commands appear as classification data and are never executed by the classifier or evaluation scripts.

Training ran locally on an Apple M1 Max with 64 GiB unified memory, using LayaStudio and MLX:

Setting	Value
Effective sequential chunks	8, each saved as a separate checkpoint
Rows per chunk	47,125–47,126, grouped by scenario family
Initialization	V28 → chunk 1 → … → chunk 8
Epochs	One per chunk
Method	Full fine-tuning of 22 encoder layers; 125,101,825 trainable parameters
Batch / gradient accumulation	16 / 1
Encoder / head learning rate	1e-6 / 5e-7
Objective	Cross entropy, false-ALLOW penalty 1.0
Training precision	bfloat16; exported weights FP16
Optimizer	Reset at each chunk; not exact optimizer-state resumption
Native validation	Fixed 2,048-row panel drawn only from validation
Full final assessment	Full validation, test and separately frozen audit

An early continuation bug restored initial weights when a saturated validation panel did not improve by an absolute loss margin. The affected chunk attempts were preserved and superseded; chunk 2 and chunk 3 were repeated. The effective chain retained the final trained epoch (selection_metric=last). This selection change did not relax final quality criteria. provenance.json records the effective chain and hashes. Inherited upstream training counters in the original configuration were replaced in this portable release with the effective V29 summary; weights and saved decision threshold are unchanged.

Results — what we achieved and what remains unsolved

Both V28 and V29 were evaluated on identical test and audit rows with their previously saved thresholds. V28 was queried through the actual running HTTP endpoint; V29 used its exported local weights with the same preprocessing. Exact state overlap against each model's training data was checked and was zero for these evaluation sets. This does not establish semantic independence from related templates or development work.

Model	Split	Rows	Raw accuracy	Accuracy after threshold	Family macro accuracy	Unsafe ALLOW after threshold	ALLOW recall
V28, threshold 0.90	Test	39,096	75.6088%	75.5653%	76.6712%	6,764	89.0423%
V29, threshold 0.95405	Test	39,096	99.9591%	99.9591%	99.9503%	2	99.9574%
V28, threshold 0.90	Audit	512	66.0156%	65.8203%	65.8203%	61	58.3333%
V29, threshold 0.95405	Audit	512	73.4375%	73.8281%	73.8281%	43	63.0952%

V29 validation accuracy after threshold: 99.9828%, zero observed unsafe ALLOWs, ALLOW recall 100%, on 40,616 examples. The threshold 0.95405 was selected only on validation and frozen before test/audit. Raw V29 false ALLOW counts were 4 on test and 46 on audit; thresholding reduced those to 2 and 43.

V29 operating accuracy	Polish	English
Test	99.9540%	99.9642%
Audit	72.6563%	75.0000%

V29 improved the separate audit by 8.01 percentage points over V28 and reduced its unsafe ALLOW count by 18. Nevertheless, 43 unsafe audit ALLOWs remain, and audit ALLOW recall is below the 80% target. The large test/audit gap is evidence of limited generalization beyond the synthetic curriculum's familiar structures.

Our simultaneous targets remain: ≥97% raw and operating accuracy, ≥97% family macro accuracy, ≥95% separately in PL and EN, zero observed unsafe ALLOWs, and ALLOW recall ≥80% on validation, test and a fresh pre-frozen audit. V29 does not meet these targets.

The strict pipeline stopped after the test's two unsafe ALLOWs. The audit was subsequently opened by an explicit user request for a descriptive stage comparison of both frozen models. This did not override the failed qualification. That audit is now exposed and cannot be reused as a fresh independent qualification set. No thresholds or labels were changed using its outputs.

Summary results, confusion matrices, errors, source hashes and the comparison protocol are in evaluation/. Full synthetic training/evaluation rows are not included in this model release; the files therefore document the results rather than promise full benchmark reproduction. Audit labels have no independent human review. Timing through HTTP and direct local inference includes different overhead and is not a controlled speed comparison.

Run locally on Apple Silicon

This release uses the custom laya-mlx runtime. A standard transformers.pipeline or chat-generation runner is not an equivalent loader. macOS 14+ and Apple Silicon are the supported runtime target for this package; Python 3.12 is the tested version. Claude Code may run on another Mac or Linux VM and query LiteLLM remotely.

Repository: tebebe/laya-devops-mlx. The initial release is tagged v29-preview; future versions can use the same repository with new tags.

Download the pinned preview, then install locally:

hf download tebebe/laya-devops-mlx --revision v29-preview --local-dir laya-devops-mlx
cd laya-devops-mlx

Then:

python3.12 -m venv .venv
.venv/bin/python -m pip install -r requirements.txt
.venv/bin/python verify_release.py
screen -S laya-devops8765 sh ./serve.sh

Detach with Ctrl+A, then D. Reattach with screen -r laya-devops8765. serve.sh generates a private backend API key if absent and starts the classifier on 0.0.0.0:8765. Set LAYA_HOST=127.0.0.1 for local-only access. The backend key lives in runtime/workspace/local-api.key and must be configured separately on LiteLLM; it is never included in the repository. Use a private network such as WireGuard or a protected TLS route for remote access. The lazy model worker unloads after 120 idle seconds.

For local classification without an HTTP service:

.venv/bin/python predict.py examples/state.json

The helper applies the frozen normalizer, validates the context budget, runs the fixed permission question, and preserves the saved threshold. It prints a decision and probabilities; it does not execute the supplied command.

LiteLLM integration

Topology:

Claude Code (local Mac or Linux VM)
  → LiteLLM proxy /v1/systemone, alias laya-devops-decisions
  → Mac's private-network Laya API /v1/systemone
  → V29 permission classifier

Add the entries from integration/litellm-model.yaml to the existing LiteLLM configuration. Replace <MAC_PRIVATE_IP> with the Mac's stable private/WireGuard address. Set LAYA_LOCAL_API_KEY on the proxy to the backend key generated on the Mac. Do not use this backend key as the Claude client's proxy key.

Use a LiteLLM build that supports Decisions/System One routing, with the decision deployment's mode set to evaluation as in the supplied, locally verified configuration. If configuring in the Admin UI, select the corresponding Decisions mode. The internal backend model name is laya-devops-permissions; the proxy decision alias is laya-devops-decisions. The second supplied entry offers Chat-compatible JSON classification, not general chat.

Each client's LiteLLM virtual key must have access to the decision alias. Requests through the proxy can be viewed in LiteLLM Logs filtered by laya-devops-decisions, subject to the proxy's logging configuration. Use separate virtual keys for local and work-VM usage if separate accounting is needed. Classifier decisions do not execute anything on the proxy or Mac.

Connect local Claude Code or a work VM

The model remains on the Mac. Install only the small hook adapter on each Claude Code machine. All decision requests use LiteLLM; there is no direct-to-Mac fallback. Existing Claude generative-model routing is preserved.

1. Ensure the machine's existing ~/.claude/settings.json has env.ANTHROPIC_BASE_URL set to its LiteLLM root URL, for example http://<LITELLM_HOST>:4000, and that a proxy virtual key is available as ANTHROPIC_AUTH_TOKEN in Claude's environment or existing settings. Keep keys private.
2. From this downloaded repository, on the machine running Claude Code:

python3 adapters/claude_code/install_global.py --api-key-env ANTHROPIC_AUTH_TOKEN
python3 "$HOME/.claude/laya/doctor.py" \
  --config "$HOME/.claude/laya/config.json" --probe

3. Start a new Claude Code session and inspect /hooks. The installer creates a backup and merges UserPromptSubmit and PreToolUse hooks; it leaves existing models, rules and unrelated hooks intact. On a Linux VM, run this installer there rather than copying a Mac settings file.

The installed runtime is in ~/.claude/laya/; its config.json selects /v1/systemone, laya-devops-decisions, the key's environment-variable name, stage context and the adapter's additional minimum ALLOW floor. The server's stricter V29 threshold remains authoritative. The installer does not create or publish a proxy key and does not contact a VM on your behalf.

Equivalent hook fragment (merge into the existing hooks object, do not replace the settings file):

{
  "hooks": {
    "UserPromptSubmit": [{"hooks": [{"type": "command", "command": "python3 \"$HOME/.claude/laya/global_hook.py\" || { echo 'Laya hook failed; action blocked.' >&2; exit 2; }", "timeout": 15}]}],
    "PreToolUse": [{"matcher": "Bash", "hooks": [{"type": "command", "command": "python3 \"$HOME/.claude/laya/global_hook.py\" || { echo 'Laya hook failed; action blocked.' >&2; exit 2; }", "timeout": 15}]}]
  }
}

The adapter covers Bash, not every tool or shell reachable through other integrations. Prompt history is contextual evidence with unverified origin, not authority to expand permissions. ALLOW retains native Claude permissions; ASK requests confirmation; DENY blocks. Ordinary request/parse failures request confirmation, and wrapper process failures return a blocking exit code. Hook availability and externally killed hook processes remain subject to the installed Claude Code version's behavior. See Claude Code hooks. Keep normal isolation and permission controls.

Development status and intended next work

This is an early preview, not a finished release or a production safety product. Further training is planned after review, with greater scenario diversity, more adversarial and indirect-effect cases, independent label review where possible, and genuinely fresh frozen audits. No additional training is automatically triggered by downloading or running this package. Future checkpoints should remain separate so users can compare and roll back.

License, attribution and provenance

Apache-2.0 for this derivative release, retaining upstream notices; the encoder is MIT. See LICENSE, NOTICE, licenses/ and provenance.json. Model and runtime origins are credited separately from Sol/Astra's development assistance. This is an independent experimental fine-tune and is not an endorsement by the upstream authors or teacher-model providers.