mstrasser's picture
Jeff v1.4 release (2026-11-05)
03486d7 verified
|
Raw History Blame Contribute Delete
11.1 kB
metadata
license: cc-by-nc-4.0
base_model: mstrasser/jeff-base
base_model_relation: adapter
library_name: peft
pipeline_tag: text-classification
language:
  - en
tags:
  - jeff
  - lora
  - decision-model
  - calibration
  - knowledge

jeff-adapter-knowledge-qa

Multiple-choice general knowledge. Selects an answer to a multiple-choice question about science, common sense or general knowledge.

A LoRA adapter for jeff-base v1.4, Jeff's small open decision model (a fine-tune of Qwen3.5-0.8B). You give it a situation and a question with named options; it returns a calibrated probability for every option from one forward pass, with no generated text to parse.

When to use it

  • You have a multiple-choice question with a defined set of answer options.

When not to use it

  • The answer depends on private documents or recent events; supply the relevant facts or use a retrieval adapter.
  • You plan commercial use; the adapter is licensed CC BY-NC 4.0 (non-commercial use only), because its training data includes benchmarks licensed for non-commercial use only.

What it decides

What a call sends:

Field Required What it holds
question (input, text) yes A question or statement to complete, as plain text; the options are the possible answers (A, B, ...).

jeff.knowledge.knowledge-qa.blank (classify, choice)

Instructions: Which option correctly fills the blank (_) in the sentence?

This function takes its own fields:

Field Required What it holds
sentence (input, text) yes A sentence with one blank written as _, as plain text; the options are the two candidate fillers.

Labels:

  • The caller writes the text of each option; the keys are A, B.

jeff.knowledge.knowledge-qa.question (classify, choice)

Instructions: Choose the option that correctly answers the question or completes the statement.

Labels:

  • The caller gives the labels (at most 8), each a key and its text.

jeff.knowledge.knowledge-qa.continuation (classify, choice)

Instructions: Choose the ending that most plausibly continues the description of the activity.

This function takes its own fields:

Field Required What it holds
activity (context) yes The activity being described, in a few words (for example "Breakdancing").
text (input, fields) yes The start of a description of the activity, to be continued by one of the options.

Labels:

  • The caller writes the text of each option; the keys are A, B, C, D.

jeff.knowledge.knowledge-qa.passage (classify, choice)

Instructions: Read the passage and choose the option that correctly answers the question or completes the statement about it.

This function takes its own fields:

Field Required What it holds
passage (context) yes The passage to read.
question (input, fields) yes The question about the passage, or a statement to complete.

Labels:

  • The caller writes the text of each option; the keys are A, B, C, D.

jeff.knowledge.knowledge-qa.passage_text (classify, choice)

Instructions: Read the passage and choose the option that correctly answers the question or completes the statement about it.

This function takes its own fields:

Field Required What it holds
passage_and_question (input, text) yes The passage followed by its question, as one plain text.

Labels:

  • The caller writes the text of each option; the keys are A, B, C, D.

jeff.knowledge.knowledge-qa.social (classify, choice)

Instructions: Read the situation and choose the option that best answers the question about the people in it.

This function takes its own fields:

Field Required What it holds
situation (context) yes A short everyday situation involving one or more people.
question (input, fields) yes The question about the people in the situation.

Labels:

  • The caller writes the text of each option; the keys are A, B, C.

jeff.knowledge.knowledge-qa.solution (classify, choice)

Instructions: Choose the solution that best achieves the goal.

This function takes its own fields:

Field Required What it holds
goal (input, text) yes A practical goal, as plain text; the options are two candidate solutions.

Labels:

  • The caller writes the text of each option; the keys are A, B.

How it was measured

Base: Jeff v1.4; adapter trained 2026-10-25.

Test Rows Accuracy
held-out rows of public benchmarks: WinoGrande, ARC, OpenBookQA, QASC, CommonsenseQA, SciQ, HellaSwag, RACE, MMLU (auxiliary), Social IQa and PIQA (public) 1,889 78.9%

Accuracy is the share of test rows whose top answer was right. Kinds: public, a public benchmark; real, held-out real documents; generator, an in-house test from the same generator as the training data (it measures less than a public or real test does); mixed, several kinds; unknown, not recorded.

When it is confident (from the calibration set): the threshold is the confidence from which an answer counts as confident, precision the share of confident answers that were right, coverage the share of answers that were confident.

Function Threshold Precision Coverage Calibration rows
blank 0.515 95.8% 100.0% 165
question 0.498 90.0% 87.9% 876
continuation 0.731 90.0% 27.8% 180
passage 0.724 90.1% 67.3% 150
passage_text 0.508 90.0% 66.7% 195
social 0.819 90.0% 51.9% 135
solution 0.888 90.2% 21.8% 188

Caveats:

  • Part of the held-out rows were already in the base model's own pretraining data, which may make the score higher than on questions it has never seen.

How to call it

Install the Python SDK and the engine (see the install guide; until the packages are on PyPI, install them from github.com/firelex/jeff).

The example below is "Warfarin and antibiotics". Some antibiotics strengthen warfarin's effect and raise the risk of bleeding, so the answer is option B.

On your machine (Jeff.local downloads this adapter and the base from Hugging Face the first time):

import jeff

client = jeff.Jeff.local(["knowledge-qa"])

decide = client.function("jeff.knowledge.knowledge-qa.question", labels={
    "A": "Antibiotics neutralise warfarin in the stomach, so the dose stops working.",
    "B": "Some antibiotics can strengthen warfarin's effect and raise the risk of bleeding.",
    "C": "Antibiotics thicken the blood, so the patient is more likely to form clots.",
    "D": "Warfarin stops antibiotics from being absorbed, so the infection may not clear.",
})

decision = decide("A patient taking warfarin is prescribed a course of antibiotics. Why should the patient's INR (a blood clotting test) be checked more often during the course?")

print(decision.value, decision.confidence)

Through the Jeff API (an API key from jeffhub.ai):

import os

import jeff

with jeff.Jeff.remote("https://api.jeffhub.ai", token=os.environ["JEFF_API_KEY"], timeout_s=30) as client:
    decide = client.function("jeff.knowledge.knowledge-qa.question", labels={
        "A": "Antibiotics neutralise warfarin in the stomach, so the dose stops working.",
        "B": "Some antibiotics can strengthen warfarin's effect and raise the risk of bleeding.",
        "C": "Antibiotics thicken the blood, so the patient is more likely to form clots.",
        "D": "Warfarin stops antibiotics from being absorbed, so the infection may not clear.",
    })

    decision = decide("A patient taking warfarin is prescribed a course of antibiotics. Why should the patient's INR (a blood clotting test) be checked more often during the course?")

    print(decision.value, decision.confidence)
curl -s https://api.jeffhub.ai/v1/decide \
  -H "Authorization: Bearer $JEFF_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"calls": [{"function": "jeff.knowledge.knowledge-qa.question", "input": "A patient taking warfarin is prescribed a course of antibiotics. Why should the patient'\''s INR (a blood clotting test) be checked more often during the course?", "labels": {"A": "Antibiotics neutralise warfarin in the stomach, so the dose stops working.", "B": "Some antibiotics can strengthen warfarin'\''s effect and raise the risk of bleeding.", "C": "Antibiotics thicken the blood, so the patient is more likely to form clots.", "D": "Warfarin stops antibiotics from being absorbed, so the infection may not clear."}}]}'

Recorded answer (2026-11-03): B with confidence 0.832.

To download the weights yourself:

hf download mstrasser/jeff-base --revision v1.4 --local-dir jeff-base
hf download mstrasser/jeff-adapter-knowledge-qa --revision v1.4 --local-dir adapters/knowledge-qa

Files

File What it is
adapter_model.safetensors, adapter_config.json The LoRA weights and their configuration
readout.safetensors The adapter's readout over the answer codes
decision_config.json Answer codes, temperature, prompt layout and the base it was trained on (the SHA-256 of the base's weights, which the engine checks)
manifest.yaml The adapter's manifest: its functions, labels, fields and calibration, as the SDKs read it
test.jsonl The held-out test set the numbers above come from
calibration.jsonl The calibration set the thresholds come from
LICENSE The Apache License 2.0, for the Qwen3.5-0.8B notice below

Licence

Adapter licence: CC BY-NC 4.0 (Creative Commons Attribution-NonCommercial 4.0 International: free to use and share with credit, but not for commercial purposes; https://creativecommons.org/licenses/by-nc/4.0/)

It is non-commercial because its training data includes sources licensed for non-commercial use only. The test and calibration sets in this repository are shared on the same terms.

Qwen3.5-0.8B notice: these weights were modified from Qwen3.5-0.8B by the Jeff project: jeff-base is a fine-tune of Qwen3.5-0.8B, and this adapter was trained on top of it. Qwen3.5-0.8B is Copyright 2026 Alibaba Cloud and licensed under the Apache License, Version 2.0; a copy of that licence is in LICENSE.

Jeff is an independent project, not affiliated with or endorsed by the makers of Qwen.