harpertoken commited on
Commit
cdd6d04
·
verified ·
1 Parent(s): d27dadf

Upload clue.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. clue.md +52 -0
clue.md ADDED
@@ -0,0 +1,52 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ language: en
4
+ library_name: transformers
5
+ pipeline_tag: feature-extraction
6
+ base_model: distilbert-base-uncased
7
+ datasets:
8
+ - squad
9
+ tags:
10
+ - distilbert
11
+ - extractive-qa
12
+ - question-answering
13
+ - squad
14
+ - fine-tuned
15
+ ---
16
+
17
+ # clue
18
+
19
+ A short continued-fine-tuning run of [`quiz`](https://huggingface.co/harpertoken/quiz), itself a DistilBERT encoder adapted for extractive question answering on SQuAD. The architecture and tokenizer are identical; only the weights differ. Where `quiz` reflects a full training pass, this checkpoint reflects roughly a thousand SQuAD examples seen once, which makes it a useful small-scale reference point and a poor substitute for a properly trained model.
20
+
21
+ Training used a learning rate of 2e-5 at batch size one for a single epoch, in float32. The published weights are `model.safetensors`. The `config.json` previously carried a key `tie_weights_`, which no version of Transformers reads; it has been removed, and nothing else in the config was altered.
22
+
23
+ ## Usage
24
+
25
+ Transformers 5 removed the `question-answering` pipeline, so load the model directly:
26
+
27
+ ```python
28
+ import torch
29
+ from transformers import AutoModelForQuestionAnswering, AutoTokenizer
30
+
31
+ tok = AutoTokenizer.from_pretrained("harpertoken/clue")
32
+ model = AutoModelForQuestionAnswering.from_pretrained("harpertoken/clue")
33
+
34
+ question = "Who wrote Hamlet?"
35
+ context = "Hamlet is a tragedy written by William Shakespeare around 1600."
36
+ inputs = tok(question, context, return_tensors="pt", truncation=True, max_length=512)
37
+
38
+ with torch.inference_mode():
39
+ out = model(**inputs)
40
+ start, end = int(out.start_logits.argmax()), int(out.end_logits.argmax())
41
+ print(tok.decode(inputs.input_ids[0][start : end + 1]))
42
+ ```
43
+
44
+ On the three questions used to check `quiz` — the capital of France, the author of Hamlet, and the year the Eiffel Tower was completed — this checkpoint returns `paris`, `william shakespeare` and `1889`, the same answers. A thousand examples has not visibly degraded it, which is itself a reason to doubt that the fine-tuning taught much.
45
+
46
+ ## Limitations
47
+
48
+ A thousand examples is a demonstration of the fine-tuning mechanics rather than a trained model, and the documentation this replaced claimed SQuAD exact-match and F1 figures that were never produced by an evaluation. Treat this as a low-fidelity copy of `quiz`. It is English-only, inherits the same uncased tokenisation and SQuAD domain bias described in the `quiz` card, and shares its 512-token limit. Compare the two directly before assuming the fine-tuning helped.
49
+
50
+ ## Attribution
51
+
52
+ DistilBERT follows Sanh et al. (2019); SQuAD follows Rajpurkar et al. (2016).