| --- |
| base_model: bert-base-uncased |
| datasets: |
| - rajpurkar/squad |
| language: en |
| library_name: sklearn |
| license: apache-2.0 |
| pipeline_tag: question-answering |
| tags: |
| - question-answering |
| - bert |
| - squad |
| - feature-extraction |
| - logistic-regression |
| - baseline |
| model-index: |
| - name: bert-squad-qa-feature-based |
| results: |
| - task: |
| type: question-answering |
| dataset: |
| name: SQuAD v1.1 |
| type: rajpurkar/squad |
| split: validation |
| metrics: |
| - type: exact_match |
| value: 14.35 |
| - type: f1 |
| value: 24.57 |
| --- |
| |
| # bert-squad-qa-feature-based |
|
|
| **Baseline** for extractive question answering on SQuAD v1.1: the **frozen, original** `bert-base-uncased` |
| is used only as a feature extractor, and two scikit-learn Logistic Regression classifiers predict the |
| answer span on top of its features. No BERT weight is trained. |
|
|
| The fully fine-tuned counterpart is published separately at |
| [Giobbva/bert-squad-qa-full-finetuning](https://huggingface.co/Giobbva/bert-squad-qa-full-finetuning). |
|
|
| | Model | Exact Match | F1 | |
| |---|---|---| |
| | **This model (frozen BERT + Logistic Regression)** | **14.35** | **24.57** | |
| | Full fine-tuning ([bert-squad-qa-full-finetuning](https://huggingface.co/Giobbva/bert-squad-qa-full-finetuning)) | 69.04 | 79.53 | |
|
|
| Evaluated on the full SQuAD v1.1 validation set (10,570 questions). |
|
|
| ## Files in this repository |
| Two `StandardScaler` + `LogisticRegression` pipelines, stored as `.joblib` files: |
| - `model1_start_classifier.joblib`: scores each context word as the answer start |
| - `model1_end_classifier.joblib`: scores each context word as the answer end |
|
|
| This repository contains **no** transformer weights: load `bert-base-uncased` from its own repository. |
|
|
| ## Features |
| - Input: `last_hidden_state` (768-d) of the frozen `bert-base-uncased` at the **first sub-token of each context word** |
| - Question and context are encoded together (`question [SEP] context`), max length 384, stride 128 |
|
|
| ## Training |
| - 15,000 SQuAD v1.1 training examples; per window, the start word, the end word |
| and 10 randomly sampled negative context words |
| - `LogisticRegression(max_iter=300)` after `StandardScaler`, seed 42 |
| - Span decoding: best `start + end` score with `end >= start` and at most 30 words |
|
|
| ## Usage |
| ```python |
| import joblib |
| import torch |
| from huggingface_hub import hf_hub_download |
| from transformers import AutoModel, AutoTokenizer |
| |
| start_classifier = joblib.load(hf_hub_download("Giobbva/bert-squad-qa-feature-based", "model1_start_classifier.joblib")) |
| end_classifier = joblib.load(hf_hub_download("Giobbva/bert-squad-qa-feature-based", "model1_end_classifier.joblib")) |
| tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased") |
| bert = AutoModel.from_pretrained("bert-base-uncased").eval() |
| |
| question, context = "Where is the Eiffel Tower?", "The Eiffel Tower is in Paris." |
| encoded = tokenizer(question, context, truncation="only_second", max_length=384, return_tensors="pt") |
| with torch.no_grad(): |
| hidden = bert(**encoded).last_hidden_state[0].numpy() |
| |
| # first sub-token of each context word |
| word_ids, sequence_ids = encoded.word_ids(), encoded.sequence_ids() |
| positions = [i for i, w in enumerate(word_ids) if sequence_ids[i] == 1 and word_ids[i - 1] != w] |
| start_scores = start_classifier.decision_function(hidden[positions]) |
| end_scores = end_classifier.decision_function(hidden[positions]) |
| |
| s = int(start_scores.argmax()) |
| e = s + int(end_scores[s:s + 30].argmax()) |
| start_char = encoded.word_to_chars(word_ids[positions[s]], sequence_index=1).start |
| end_char = encoded.word_to_chars(word_ids[positions[e]], sequence_index=1).end |
| print(context[start_char:end_char]) |
| ``` |
|
|
