AdityaPS/SpaceLLM_Multi_turn

SpaceLLM Multi-Turn QA

A LoRA adapter for openai/gpt-oss-20b, fine-tuned for question answering across multi-turn dialogue over space-agency-derived text. This repository contains adapter weights only; the base model must be loaded separately.

• Developed by: Aditya Pratap Singh, Shivam Kumar, Dr. Saurabh Srivastava.
• Model type: PEFT LoRA adapter (CAUSAL_LM)
• Language: English
• Base model: openai/gpt-oss-20b, revision 6cee5e81ee83917806bbde320786a8fb61efebee
• Companion model: SpaceLLM Single-Turn QA — a separate adapter trained on a different data format and evaluated on a different test set. The two adapters' scores are not a head-to-head comparison.

Please note

• This adapter uses no retrieval. There is no RAG pipeline, no index, and nothing looked up at inference time.
• LoRA targets attention projections only — q_proj, k_proj, v_proj, o_proj — not lm_head.
• Training and evaluation references were generated by a teacher model and have not been independently fact-checked. Reported scores measure similarity to those references, not verified factual accuracy.
• This is a distinct artifact from the older AdityaPS/SpaceLLM_v1 checkpoint, which used a different configuration (lm_head-only, rank 32) and different data/evaluation. If SpaceLLM_v1 remains public, treat it as a separate, earlier experiment — its card and weights should not be conflated with this one.

---

Uses

Intended:

• Research on parameter-efficient domain adaptation for multi-turn, space-agency-style QA.
• Serving as a multi-turn baseline in comparative dialogue/QA evaluations, including context-carryover studies.

Out of scope:

• Mission-critical operations, spacecraft control, orbital mechanics execution, or any safety-critical or autonomous decision system.
• Any use where an unverified or incorrect answer could cause harm.
• Conversations longer or more open-ended than the training data's three-level chains (see Training data) — behavior there is untested.

Expert verification is recommended before relying on any specific technical claim this model produces, particularly claims that depend on context carried over from earlier turns.

---

How to get started

The example loads the pinned base revision and this adapter, applies the Harmony chat template to the conversation so far, forces the assistant final channel, and extracts only that channel's text — the same approach used in evaluation. Relying on skip_special_tokens=True alone is avoided because it can mix analysis-channel and final-channel text.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, Mxfp4Config
from peft import PeftModel

BASE_MODEL = "openai/gpt-oss-20b"
BASE_REVISION = "6cee5e81ee83917806bbde320786a8fb61efebee"
ADAPTER_REPO = "AdityaPS/SpaceLLM-MultiTurn"

tokenizer = AutoTokenizer.from_pretrained(ADAPTER_REPO)
base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    revision=BASE_REVISION,
    torch_dtype=torch.bfloat16,
    quantization_config=Mxfp4Config(dequantize=True),
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, ADAPTER_REPO).eval()

# Conversation so far: earlier turns, ending with the user's latest question.
messages = [
    {"role": "user", "content": "What is the James Webb Space Telescope?"},
    {"role": "assistant", "content": "It is a large infrared space telescope launched in December 2021."},
    {"role": "user", "content": "Where is it positioned?"},
]

prompt = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
prompt += "<|channel|>final<|message|>"  # force the final channel

inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=512,  # inference-example setting; evaluation used 2048 (see below)
        do_sample=False,
        repetition_penalty=1.05,
        no_repeat_ngram_size=6,
    )

generation = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=False)
for stop_token in ("<|return|>", "<|end|>"):
    generation = generation.split(stop_token)[0]

print("Answer:", generation.strip())

Note: this example uses max_new_tokens=512 for a quick, practical demo. This differs from the 2,048-token limit used during evaluation (see Evaluation) — treat this as an inference-example setting, not the evaluated protocol.

---

Training details

Training data

Data was built offline, with no retrieval stage used anywhere in training or inference:

1. Space-agency text was scraped and stored as JSON.
2. Text was normalized and split into chunks.
3. Each chunk was passed directly to an Ollama mistral:7b teacher model.
4. The teacher was prompted to generate three three-level dialogue chains per chunk, with each chain expanded into basic, intermediate, and advanced turn-level examples, with answers requested to be grounded in the supplied chunk.
5. The generation pipeline did not independently verify that answers were entailed by their source chunk — some answers may be unsupported or incorrect.

Split	Usable examples
Train	19,527
Validation	2,295
Test (evaluated)	1,161

These are the only counts supported for this artifact. A previously circulated figure of "3,428 documents / 16,831 training samples" does not match this adapter's training/evaluation split counts or file hashes and should not be cited for this repository.

The evaluated test set contains 1,161 rows but only 1,152 distinct sample_id values (nine duplicated IDs). 1,161 is retained as the denominator for all reported evaluation numbers below, consistent with what was actually scored.

Training hyperparameters

Setting	Value
LoRA target modules	q_proj, k_proj, v_proj, o_proj
LoRA rank ($r$) / alpha ($\alpha$)	16 / 32
LoRA dropout	0.05
Task type	CAUSAL_LM
Max sequence length	2,048 tokens
Base weights	MXFP4, dequantized at load time
Compute precision	BF16

The base model is the published MXFP4 gpt-oss-20b checkpoint, loaded with MXFP4 dequantization and run in BF16 compute — this is a loading/compute configuration, not a separately distributed "BF16 version" of the model.

---

Evaluation

Test set: 1,161 multi-turn records (1,152 distinct sample IDs; see Training data). Test file SHA-256: af77ece4ea802197455182e9846a3d2326a1c0ec362fe51e4da08cfa434aa5b5 Adapter config SHA-256: 0c94a653b93d9974e09ed75c5d85fa1722b05c8e0f7c9c465dd2d54c38f75f1b

Protocol: Base and adapter used identical tokenized prompts. All 1,161 records produced a final answer; none were skipped. Generation was deterministic (do_sample=False) with max_seq_len=2048, max_new_tokens=2048, repetition_penalty=1.05, and no_repeat_ngram_size=6, applied to both base and adapter. Only text from the Harmony final channel was scored; analysis-channel text was excluded.

Metrics: BERTScore computed with roberta-large; token F1 computed on lowercased whitespace tokens; exact match computed after stripping surrounding whitespace. A missing answer would score zero on all metrics while remaining in the denominator — no answers were missing for this test set.

References were produced by the same teacher-based generation process used for training data and have not received independent factuality verification.

Results

Model	BERTScore P	BERTScore R	BERTScore F1	Token F1	Exact match	Hit 2,048-token cap
Base (gpt-oss-20b)	0.788093	0.865580	0.824194	0.104781	0 / 1,161	23 / 1,161
+ SpaceLLM Multi-Turn Adapter	0.909230	0.891980	0.900206	0.362109	3 / 1,161	5 / 1,161

BERTScore F1 change: an absolute gain of 0.076012. This is not a "6% improvement," and it is not evidence of improved factual accuracy or domain understanding — it reflects greater semantic/stylistic similarity to teacher-generated references under this specific protocol.

Reading the weaker metrics honestly

• Exact match is very low (3/1,161, ≈0.26%) for both models, which is expected for free-form generative QA — a correct answer rarely matches a reference string verbatim in wording, order, or length. This metric is reported for completeness, not as a proxy for correctness rate.
• Token F1 (0.362) is modest in absolute terms, even though it more than tripled over the base model's 0.105. Token overlap penalizes any answer that is phrased differently from the reference, including answers that may be factually equivalent. Treat the relative change as a signal of closer stylistic alignment, not the absolute value as an accuracy score.
• BERTScore is the most informative similarity signal here, but it has the same ceiling as its references: since references are teacher-generated and unverified, a higher BERTScore partly reflects the adapter reproducing the teacher's phrasing habits — including any errors or unsupported claims the teacher made — rather than confirmed improvement in factual correctness.

---

Generation stability

An earlier unguarded diagnostic run at the same 2,048-token maximum, without repetition controls, produced outputs that fell into exact phrase loops. The final evaluation protocol added repetition_penalty=1.05 and 6-token n-gram blocking (no_repeat_ngram_size=6) to both base and adapter generations to address this.

Model	Unguarded diagnostic (cap events)	Guarded final run (cap events)	Reduction
Base	97	23	76.3%
Adapter	20	5	75.0%

This is reported as an engineering diagnostic, not a controlled ablation: prompt hashes changed across rebuilt evaluation images between the unguarded and guarded runs, so the reduction cannot be attributed to the decoding settings alone. Additionally, the guarded settings removed the conspicuous exact-repetition loops but did not eliminate semantic degeneration generally — a small number of guarded outputs still reached the token cap while producing drifting numerical lists or otherwise unrelated text.

---

Bias, risks, and limitations

• The supported conclusion is narrow: these results show improved similarity to synthetic, teacher-generated references under this specific evaluation protocol. They do not establish improved factual accuracy, reasoning ability, or general domain understanding.
• Exact match (3/1,161) and token F1 (0.362) are low in absolute terms. This is a known property of free-form generative QA evaluated against a single reference string, not necessarily a sign the model is "wrong" most of the time — but it also means these numbers cannot be read as an accuracy rate.
• No independent factuality check exists. References were generated by the same mistral:7b teacher process used for training data; higher similarity scores may partly reflect the adapter learning to imitate the teacher's style, length, and any of its errors, rather than confirmed improvements in correctness.
• Training dialogues are synthetic and capped at three levels. Behavior in longer, more open-ended, or off-script conversations is untested.
• The evaluated test set contains nine duplicated sample_id values (1,161 rows, 1,152 distinct IDs); 1,161 is used consistently as the denominator throughout this card.
• This adapter has not been validated for operational, mission-critical, or safety-critical use of any kind, including spacecraft control, mission planning, or autonomous decision-making.
• There is no retrieval and no source citation — answers cannot be traced back to a specific source document by the model itself.
• no_repeat_ngram_size=6 can suppress legitimate repeated technical phrases (e.g., a mission name or unit repeated for clarity), which may slightly affect fluency on some answers.
• Generation stability numbers above are diagnostic, not a controlled ablation (see Generation stability).
• Single-turn and multi-turn adapters were trained on different formats and evaluated on different test files; do not compare their scores directly.
• Knowledge is fixed at training time and may be outdated.
• English only.
• General limitations and terms of the base model openai/gpt-oss-20b also apply.

---

Reproducibility artifacts

• Test set SHA-256: af77ece4ea802197455182e9846a3d2326a1c0ec362fe51e4da08cfa434aa5b5
• Adapter config SHA-256: 0c94a653b93d9974e09ed75c5d85fa1722b05c8e0f7c9c465dd2d54c38f75f1b

---

Files

adapter_model.safetensors, adapter_config.json, tokenizer files, chat_template.jinja, README.md.

---

License

Apache-2.0, matching the base model (openai/gpt-oss-20b). Note: this covers the adapter weights and code; it does not by itself establish licensing terms for the scraped source text or the teacher-generated training data used to produce this adapter.