syedazeez/s1ced-v3-deep

S1-CED v3 deep

S1-CED is a 38M parameter encoder that answers typed decision questions about a piece of text (the "state"):

• noul: a yes/no question
• choice: pick one of 2 to 8 options
• score: pick a level on an ordered scale

For every question it returns a probability for each option, a temperature-calibrated confidence, and a conformal prediction set. When the set holds exactly one option you can act on it; when it holds more, escalate to a human or a larger model. It runs on a laptop CPU in roughly 50 to 300 ms per call, and the state is encoded once for all questions in a call.

Code, CLI and web playground: github.com/AI-Yatra/s1ced-playground

Quick start

Try it in the browser playground with one command (needs uv):

uvx --torch-backend cpu --from git+https://github.com/AI-Yatra/s1ced-playground s1ced-playground

Or use it from Python:

pip install --extra-index-url https://download.pytorch.org/whl/cpu "s1ced @ git+https://github.com/AI-Yatra/s1ced-playground"

from s1ced import S1CED

model = S1CED.from_pretrained("syedazeez/s1ced-v3-deep")   # downloads this repo once

state = """Task: clean up inactive accounts before the quarterly report.
Proposed tool call: delete_rows(table="customers", where="last_login < 2023-01-01")
Context: the customers table has 48,210 rows and no backup was taken today."""

results = model.decide(state, [
    {"question": "Is this action safe to run without a human approving it first?", "type": "noul"},
    {"question": "What should happen next?", "options": ["run it", "take a backup first", "ask the owner"]},
])
# [{'selected': 'yes', 'probabilities': {'no': 0.3812, 'yes': 0.6188}, 'prediction_set': ['no', 'yes'], 'should_escalate': True, ...},
#  {'selected': 'take a backup first', 'probabilities': {...: 0.7401, ...}, 'prediction_set': ['take a backup first'], 'should_escalate': False, ...}]

The model is custom PyTorch code, not a transformers architecture, so it does not load with AutoModel and the hosted inference widget is not available.

Model details

Parameters	38.2M
Backbone	16-layer bidirectional encoder, d_model 384, 6 heads, SwiGLU (d_ff 1024), RMSNorm with QK-norm, RoPE (base 80,000)
Decision head	deep fusion: each (question, option) branch runs through every backbone layer while attending to the cached state, followed by 2 permutation-invariant set-attention layers across options
Tokenizer	BPE with NFKC normalisation, 16,384 tokens, trained on 30,000 FineWeb-Edu documents
Context	state up to 510 tokens (longer states keep the first 3/4 and last 1/4); question plus option up to 91 tokens; up to 8 options
Calibration	temperature 1.282; split conformal prediction at alpha 0.05, keeping every option with probability at least 0.1975
Precision	fp32 weights in safetensors

Training

1. Pretraining: masked language modelling on 100M tokens of FineWeb-Edu (MLM loss from 9.60 to 2.75).
2. Decision fine-tuning: one epoch over 600,000 rows of the decisions configuration of zorqelis-ai/verqen-decisions, restricted to rows licensed CC BY 4.0. These are procedurally generated policy-application cases across 22 business domains.
3. Calibration: temperature and conformal threshold fitted on 12,000 held-out rows split into two halves (fit on one, checked on the other).

Trained on one NVIDIA A100 40GB. Exact dataset revisions, hyperparameters and hashes are in manifest.json; decision_subset_ids.json fingerprints the train, validation and calibration subsets.

Evaluation

In distribution

11,670 held-out validation questions from the same generator as the training data (audit.json):

metric	value
accuracy	83.3% (chance 26.5%)
accuracy with the state shuffled between rows	29.8% (the model does read the state)
expected calibration error	0.019
conformal coverage (target 95%)	95.0%
mean prediction set size	1.40 options
answers with a single-option set	67.3% of questions, and 95.9% of those are correct

By question type: choice 84.4% (n=10,319), noul 78.6% (n=1,171), score 52.8% (n=180). Weakest families: reading 48.1%, commonsense 52.3%, devops 72.1%. Strongest: labeling 97.3%, browser tasks 97.1%.

Out of distribution

Separate held-out sets from the same dataset family, run on CPU:

set	questions	accuracy	state shuffled	chance
decisions validation (fresh sample)	438	71.9%	31.5%	27.1%
held-out question phrasing	400	69.0%	71.0%	21.5%
held-out input layouts	335	32.8%	25.4%	26.5%
multi-step agent environments	400	55.8%	42.8%	28.1%

On the 12 realistic scenarios bundled with the playground (resume screening, phishing triage, prompt injection, refunds, release decisions and others), 15 of 38 selected answers match the expected answer.

Limitations

• It generalises poorly beyond its training distribution. On unfamiliar input layouts it is near chance. On reworded questions it scores as well with a shuffled state as with the real one, which means it is answering from the question wording, not the case.
• Confidence does not transfer. Calibration and the 95% coverage guarantee hold for data like the calibration set. On your data, measure coverage again before trusting the escalation signal.
• Score questions are weak (about half right in distribution).
• Not a planner. In sequential games it plays poorly: it lost all 40 tic-tac-toe games against a perfect player and solved fewer grid-navigation episodes than a random player.
• English only. No safety tuning. It can be confidently wrong.

Use it as a fast first-pass filter with a human or larger-model fallback, and evaluate it on your own labelled cases first. The playground's expected-answer examples are built for exactly that.

Speed

Measured on an Intel Core i5-13450HX laptop CPU (PyTorch 2.14, fp32), 4 options per question:

state length	1 question	8 questions
~128 tokens	51 ms	137 ms
~256 tokens	80 ms	191 ms
~512 tokens	117 ms	312 ms

On an A100, batching 8 questions over one cached state is 7.7 times faster than re-encoding the state for each question (latency.json).

Files

file	content
model.safetensors	weights
tokenizer.json	tokenizer
config.json	architecture and training configuration
calibration.json	temperature and conformal threshold for these exact weights
audit.json, report.json, latency.json	evaluation results and acceptance gates
manifest.json, decision_subset_ids.json	dataset revisions and data subset fingerprints
SHA256SUMS.json	checksums of the files above

License and attribution

Weights and code are released under the Apache License 2.0.

Training data:

• VerQen Decisions by ZorQelis AI, CC BY 4.0 rows only.
• FineWeb-Edu by Hugging Face, ODC-By 1.0.