Add the own-data readers (readers/: schema items, two reading models, re-scorer, refusal); cards describe them

README.md

@@ -1,6 +1,9 @@
 ---
 license: apache-2.0
-base_model: Qwen/Qwen2.5-0.5B
+base_model:
+  - Qwen/Qwen2.5-0.5B
+  - FacebookAI/xlm-roberta-base
+  - FacebookAI/xlm-roberta-large
 library_name: peft
 tags:
   - text-to-sql
@@ -14,9 +17,10 @@ tags:
 This repository contains the public, manifest-pinned runtime weights for
 [Prereasoner](https://github.com/ManiDoraisamy/prereasoner-data), an interpretable
 table-question-answering system. Learned components produce semantic evidence: named coordinates
-for columns and question words, and Schema.org property and class scores. A deterministic typed AST
-planner builds every query and owns joins, calculation verification, selection, rendering, and
-execution. No model in this bundle writes SQL, and the bundle is not a standalone text-to-SQL model.
+for columns and question words, Schema.org property and class scores, and the own-data readers'
+readings of a question (which tables, columns, values, groups, orders and calculations it asks for).
+A deterministic typed AST builder makes every query from those readings and owns joins, calculation
+verification, rendering, and execution. No model in this bundle writes SQL, and the bundle is not a standalone text-to-SQL model.
 
 ## Install
 
@@ -43,21 +47,27 @@ the complete bundle, and only then installs it.
 | `anchor_assignment.npz` | Calibrated named-dimension thresholds |
 | `primitives.npz` | Learned primitive-composition head |
 | `schema_property_head.pt` | Calibrated Schema.org named-property evidence head |
+| `readers/stage1/` | Own-data schema-item model (XLM-R base): scores each table's and column's use |
+| `readers/stage2_a/`, `readers/stage2_b/` | Own-data reading models (XLM-R large), one with the question's values beside their columns and one without |
+| `readers/rescorer/` | Re-scorer (XLM-R base): orders the readings' queries that ran |
+| `readers/refusal.json` | Logistic refusal over the re-scorer's confidence, fitted on held-out calibration databases |
 
 The source repository contains the small ontology, calibration, taxonomy, and manifest artifacts.
-The base Qwen model is downloaded separately from its publisher. `python -m engine.fetch_weights`
+The base Qwen model is downloaded separately from its publisher. The readers' directories hold their
+complete fine-tuned weights (encoders in bfloat16, heads in float32) and tokenizer files. `python -m engine.fetch_weights`
 downloads only the files the source manifest lists. Prereasoner loads no SQL-generating model; the
 retired SQL proposer adapter (`sql_proposer/`) was removed from this repository on 2026-10-02 and
 remains in its history.
 
 ## Model Boundary
 
-The encoder adapter emits named Schema.org property probabilities, calibrated class proposals,
-embeddings for structural intent and ranking, and calculation operand signals. A released class may
-propose a coarse resolver family, but no score can authorize a table, join, calculation, or answer.
-Prereasoner's deterministic search builds every query from these signals, and the engine serves the
-best-ranked query that runs on the data. An operator may enable an external Gemini fallback for
-questions the search cannot answer; it is not part of this bundle, and its answers are labelled.
+The encoder adapter emits named Schema.org property probabilities, calibrated class proposals, and
+embeddings for column typing and the structural intent of composition and world questions. A released
+class may propose a coarse resolver family, but no score can authorize a table, join, calculation, or
+answer. Own-data questions are read by separate XLM-R readers (`engine/data/readers/`); a deterministic
+builder makes each of their readings a typed query, and no model writes SQL. The optional Gemini model
+serves only the chat orchestrator, starter questions and reference generation, and is not part of
+this bundle.
 Deterministic code applies ontology mapping, exact source grounding, typed constraints, abstention
 rules, and execution checks.
 
@@ -95,6 +105,9 @@ servable. Unsupported and under-calibrated coordinates abstain.
 ## License
 
 The Prereasoner weight bundle is released under Apache-2.0. The base model and source datasets retain
-their own licenses and terms. The retired `sql_proposer/` adapter in earlier revisions was trained on
+their own licenses and terms. The readers in `readers/` are fine-tuned from XLM-RoBERTa (MIT) on Spider's
+train and dev questions (Yu et al., EMNLP 2018; CC BY-SA 4.0), BIRD train (Li et al., NeurIPS 2023;
+CC BY-SA 4.0) and Spider-Syn (Gan et al., ACL 2021; MIT); keep those attributions when redistributing
+them. The Spider test set appears in no training step. The retired `sql_proposer/` adapter in earlier revisions was trained on
 Spider TRAIN (Yu et al., EMNLP 2018; CC BY-SA 4.0); keep that attribution when redistributing it. See
 [`THIRD_PARTY.md`](https://github.com/ManiDoraisamy/prereasoner-data/blob/main/THIRD_PARTY.md).

qwen_lora/README.md

@@ -13,7 +13,8 @@ This directory of the
 [Prereasoner runtime bundle](https://huggingface.co/prereasoner/prereasoner-weights) holds the LoRA
 adapter that turns `Qwen/Qwen2.5-0.5B` into an encoder whose representation feeds named semantic
 readouts. Prereasoner uses the base without its decoder and loads no SQL-generating model: a
-deterministic search builds every query. A retired SQL proposer adapter (`sql_proposer/`) from an
+deterministic builder makes every own-data query a typed AST from the readings of separate XLM-R
+readers. A retired SQL proposer adapter (`sql_proposer/`) from an
 earlier design was removed from this repository on 2026-10-02 and remains in its history.
 
 Install and validate the complete compatible bundle through the source repository:

readers/refusal.json

@@ -0,0 +1,26 @@
+{
+ "coef": [
+  2.517458605608872,
+  0.429020566395794,
+  0.2848816003601845,
+  0.07420142722472789,
+  -0.11174800968699541
+ ],
+ "features": [
+  "rescorer_probability",
+  "score",
+  "margin",
+  "rank",
+  "empty"
+ ],
+ "intercept": -0.7305985111800368,
+ "matched": {
+  "answered": 881,
+  "baseline_wrong": 294,
+  "questions": 1053,
+  "right": 587,
+  "wrong": 294
+ },
+ "target": null,
+ "threshold": 0.2511
+}

readers/rescorer/config.json

@@ -0,0 +1,25 @@
+{
+  "architectures": [
+    "XLMRobertaForMaskedLM"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "eos_token_id": 2,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "xlm-roberta",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "output_past": true,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "transformers_version": "4.17.0.dev0",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "vocab_size": 250002
+}

readers/rescorer/model.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1bda010f1ed74e34f6f85e0546f0efd8799ec48d5a9e7aacc541369debaa526b
+size 556142991

readers/rescorer/sentencepiece.bpe.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cfc8146abe2a0488e9e2a0c56de7952f7c11ab059eca145a0a727afce0db2865
+size 5069051

readers/rescorer/tokenizer_config.json

@@ -0,0 +1 @@
+{"model_max_length": 512}

readers/stage1/config.json

@@ -0,0 +1,25 @@
+{
+  "architectures": [
+    "XLMRobertaForMaskedLM"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "eos_token_id": 2,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "xlm-roberta",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "output_past": true,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "transformers_version": "4.17.0.dev0",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "vocab_size": 250002
+}

readers/stage1/model.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3f84c5cd9424f008116befbcd2d1a08c9508a2583be05139694edea7a78ade7e
+size 556161927

readers/stage1/sentencepiece.bpe.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cfc8146abe2a0488e9e2a0c56de7952f7c11ab059eca145a0a727afce0db2865
+size 5069051

readers/stage1/tokenizer_config.json

@@ -0,0 +1 @@
+{"model_max_length": 512}

readers/stage2_a/config.json

@@ -0,0 +1,25 @@
+{
+  "architectures": [
+    "XLMRobertaForMaskedLM"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "eos_token_id": 2,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 1024,
+  "initializer_range": 0.02,
+  "intermediate_size": 4096,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "xlm-roberta",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 24,
+  "output_past": true,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "transformers_version": "4.17.0.dev0",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "vocab_size": 250002
+}

readers/stage2_a/model.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1c6065296b9c61dffab87c7d4f22979918a7b1322fed3e9945c09a40a1a25fa1
+size 1124924775

readers/stage2_a/reader.json

@@ -0,0 +1 @@
+{"anchored": true}

readers/stage2_a/sentencepiece.bpe.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cfc8146abe2a0488e9e2a0c56de7952f7c11ab059eca145a0a727afce0db2865
+size 5069051

readers/stage2_a/tokenizer_config.json

@@ -0,0 +1 @@
+{"model_max_length": 512}

readers/stage2_b/config.json

@@ -0,0 +1,25 @@
+{
+  "architectures": [
+    "XLMRobertaForMaskedLM"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "eos_token_id": 2,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 1024,
+  "initializer_range": 0.02,
+  "intermediate_size": 4096,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "xlm-roberta",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 24,
+  "output_past": true,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "transformers_version": "4.17.0.dev0",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "vocab_size": 250002
+}

readers/stage2_b/model.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:052ea4af552951d5589b04ae9be52a38c2167e68e024ca44e861f1663dd438d2
+size 1124924775

readers/stage2_b/reader.json

@@ -0,0 +1 @@
+{"anchored": false}

readers/stage2_b/sentencepiece.bpe.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cfc8146abe2a0488e9e2a0c56de7952f7c11ab059eca145a0a727afce0db2865
+size 5069051

readers/stage2_b/tokenizer_config.json

@@ -0,0 +1 @@
+{"model_max_length": 512}